Making the Unix Case: Saving Whackabilly

Table of Contents

• Note
• Background
• Action Agenda
• The Plan
• The Budget
• Impact Summary

DECK: Author Paul Murphy's previous Linuxworld article on making the Unix decision suggested that Unix is usually a smarter business choice than Windows. The current article, first in a series, looks at what it takes to implement that knowledge in a small business.

Note

The scenario being explored is that of a new systems manager brought into a small company "to bring systems costs under control". The company, its staff and products, are fabrications but the situation presented, and the remedies offered, reflect the author's recent experience with real-world clients that got over enthused about the prospects for ecommerce applications. Whackabilly is imaginary, but the conditions, decisions, and outcomes described are broadly based on real events.

Case Background

Whackabilly Toy Inc. makes the Ventables series of training aids and the Whackabilly line of deformable plastic dolls along with the Windows software that goes with them. The products are made out of a dense foam material incorporating up to about 30% sand for weight and feel and have one or more presure sensors embedded near the center. The toys are typically about eight inches high and the training aids usually nearly life size. All products ship in Styrofoam kits complete with instruction manual, CD, and mouse cable splitters for both serial and PS2 type connectors but the whackabillies also include a foam rubber mallet.

When the dolls are deformed the software grabs the amount, direction, and initial duration of the deformation to trigger an appropriate action. For the toys this is a sound and graphics display set according to user preferences; for the training aids it is usually a measure of the source, direction, and strength of the presure applied. We ship a default set of these with each product but also provide an exchange service for users who make their own and wish to share them with others. These, needless to say, are often hilariously obscene and far more creative than the cartoonish violence and sound effects our lawyers let us ship.

Our biggest sellers, by volume, resemble public figures but the greatest revenue growth is now coming from serious training aids like our Heimlich model.

The manufacturing operation is reasonably simple. The sensors and ASIC [Application Specific Integrated Circuit] are custom built for us in Singapore and shipped to an extruder in Detroit which produces the dolls. Kit assembly, packaging, and shipping are co-located with the executive offices and sales center in Omaha.

The company currently employs about 109 people of whom 35 people are in sales and 26 are systems staff. Of these 26, only two work on the software shipped as part of the product while 14 are fully committed to the ecommerce effort and 2 or 3 work on it part time.

At present all computer work is done using Windows based gear. The previous director of information systems precipitated his own demise by requesting nearly a million dollars to upgrade everything to Windows 2000 on new Compaq gear and the president's words to me early on in the relationship were that "Bob was out of control and wanted to throw money, good money, away on gear we don't need." Nevertheless he agreed to give me $400,000 for new capital investment and to front another $400,000 from next year's IT budget to support changes needed now.

Action Agenda

In the longer term the strategy will be to turn this place around; to transform the IT role from that of an expensive after thought to being a key part of the business. I'm taking two small steps in that direction right up front:

1. working with the VP Sales to evaluate using Palm Pilots, together with some new web services software, to replace 22 IBM thinkpads now checked out by sales people when they leave the office on business; and,
2. working with the VP Finance ostensibly to develop the cost/benefit case for spinning out our current web based retail operation as a separate company but really to build rapport and get her to understand that systems costs are not measured by expenditures but by foregone oppportunities - just as the cost of a cold is measured by time off work and not by the cost of the drugs needed to control it.

In the short term, however, there are urgent problems that need to be addressed:

1. the most difficult of all the transition jobs starts with meeting with each member of the IT staff and explaining the changes to come. Most of them know that the company can't possibly sustain present IT staffing levels without an enormous, and highly improbable, boost in sales but that doesn't relieve me of responsibility for making the cuts. As usual, this won't be made easier by the fact that some are genuinely likeable but incompetent, some are afraid to talk honestly, some are competent but angry, and a few have user or executive relationships to be wary of. No matter what I feel, however, at least half of the 26 have got to go quite soon to get staffing down to anything remotely reasonable and, ideally, another two or three will go next year;
2. the e-commerce effort, something which threatens to eat the company and now requires two rackmounts of NT servers, has to be moved to a single Unix machine to eliminate the need for 24 x 7 staffing and cut that workload from nine people plus a manager to part time work for two. Not only are costs out of control, but volume is low and the sales people are unhappy because this thing is really a web store when what they wanted was automated fulfillment with customers placing, and tracking, their own orders;
3. that same Unix machine will get a write only laser disk sub-system to squirrel away copies of all outgoing, and incoming, e-communications. Right now the company has no defense against someone who claims they emailed us something, but didn't; no defense against someone who claims to have received an Email no one here sent; and no defense against workplace harassment or other claims arising from email abuse. Put some disk control procedures in place and we'll have the best of all defences: unimpeachable records;
4. we're turning down orders (!) because the product software doesn't run on Linux or Mac OS; and,
5. there's lots of user unhappiness ostensibly about being stuck with two and a half year old Windows 98 machines because of all the money going into the e-commerce venture but really, of course, about the failure by the systems group to meet expectations. To meet expectations the company needs to review and upgrade its design software, clean up the executive information system, and try to bring more functionality to the sales group while getting upgrade and support costs under control.

The Plan

The technology plan is easy:

1. we're going to swap all production services onto two Sun 850s;
2. we're going to transition users to smart displays initially with software running on the existing NT servers but gradually shifting them to purely Unix software;
3. we're going to require Linux on company supplied home machines used to access office documents; and,
4. we're going to go looking for new and better software for the design, sales, and reporting functions.

• five people will not be attending because they've been let go the night before;
• ten more have to go by Christmass;
• all 15 will get three months severance and we'll carry their medical coverage until March 31st of next year;
• we will be hiring two new Unix sysadmins with considerable Solaris and RDBMS experience as quickly as we can find qualified applicants;
• the two person product group is temporarily being expanded by one, preferably by transferring an existing staffer, to add Linux and Mac OS X compatibility to the product;
• the average salary will rise, by the late next year, from around $38,000 now to around $50,000 and the people who earn the right to stay can expect more responsibility, more training, and more scope for personal initiative.

At about the same time the VP Finance will send a personal email to all employees outside IT informing them of these changes. She will also be telling them that people whose desktop PC gets replaced by a smart display will have the option of taking the PC, minus licensed software, home as a gift from the company. Since most are reasonable quality Compaq 500s and 550s they'll make great Linux machines and we'll be hosting a series of Linux clinics for employees timed to coincide with shutdown of the machines for office use.

Attendees will load Linux on their own machines and get a tutorial introduction to the system and key tools such as the OpenOffice.org suite, Pipeline based VPNs, and the advantages of remote X access.

The Linux installation and training offer is far more important than it looks. Physical control of the systems has to be centralized for cost, security, and performance reasons but functional control has be pushed out to the users. That's the reverse of the present situation in which users have physical control of desktop computers but little or no say in how they are used. By trying to get as many home users as possible to switch to, and learn, Linux I'm hoping to apply the maxim that "knowledge is power" to help shift systems control toward the user side. If I can create a pool of knowledgeable users, they will not only be more effective in working with IT staff on identifying and developing new services but their independent knowledge and information access will also allow them to function as a check against any power grabs by IT staff.

The Budget

There are two months left in this year's 1.2 million IT staffing budget but I've committed to reducing that by $400,000 next year. As a result my change budget is going to get spent on:

1. staffing
   

   My goal is to be down to 12 full time staff by February 1st of next year at a buy-out cost of about $190,000 including the extended medical coverage, giving the remaining staff an immediate raise to about 44K on February first and then raising that again to about 50K in August. That should meet the commitment to spend no more than 800K, fully burdened, on salaries and related costs with a handful of change left over for emergencies.

   
   

   Initially two people will be assigned to each server team and be responsible for the full range of operations from making sure the UPS has working batteries through to user desktops but that will increase to three as applications come off the Citrix systems. In the longer run the three product support people will get the additional responsibility of running any truly necessary Windows stuff on their test systems. Two will initially be assigned to new project development and I'll need one to play vice-principal looking after security, discipline, vendor meetings, and paperwork for a total complement of 12.



2. Servers

   I'm ordering two new Sun Ultra 10 workstations with 21" screens, and two 850s each with 4 CPUs, 8GB of RAM, and two sets of 6 x 36GB drives on condition Sun provide free iPlanet licensing, an immediate loaner 450 or comparable until delivery and acceptance of the 850s, and fully test and configure all three on arrival here -including setting up the iplanet email/scheduling services and the HP Sure Store 400 Optical disk writer ($12K). Since I'm prepared to pay retail (about $175K all in) and the dot bomb hype is well behind us, I have every confidence Sun will agree.

   
   

   I'm going to convert two of the existing web servers to run Windows 2000 Advanced with Terminal Server by stripping everything I can out of the other two and shutting those down. That'll give me four 550Mhz Xeons in each one along with 8GB of RAM and 260GB of accesible disk. Licensing will be an expensive throw away at about:
   

   Upgrade from NT server to Win2K Advanced Server with 25 CALS	$1,9191
   Upgrade from NT backOffice to Win2K Servers with 50 CALS	$3,895
   Upgrade to 80 CALS	$18,560
   Terminal Service with 80 CALS	$10,476
   Upgrade 80 Microsoft Office 98 licenses for TS	$19,120
   Total Microsoft Licensing	$53,074
   Add Citrix Metaframe license for 70 concurrent users	$22,500
   Total Licensing	$76,640
   1Pricing is from the Microsoft and Citrix sites as of Nov. 6/01.

   
   but will enable the users to keep on using the software they know as they get used to the smart display environment. As we bring in replacements for the accounting and other software they use now, I'll be able to phase this stuff out instead of endlessly paying to upgrade or replace it.



3. Smart displays
   

   I'm ordering three new Mac G4 workstations, one a cheap model for testing and development at about 3K but two with dual CPUs and various Adobe tools for use in packaging and advertising materials design at about $9,000 each -mainly because of the fancy 22" LCD displays and the Adobe licenses. Almost everyone else will get an NCD NC900 with a standard 21" CRT at $1,900 each for a total of 165,300 for 87 units.



4. Networking
   

   At present there are firewalls, routers, internal switches, and various other layers of gear that keep a couple of people busy. All of that is going away, as is the current 10MBS service connection to a local carrier. Worldcom will be putting in two burstable T1s to each of our two service nets at a combined annual cost that's a little more than we're paying now. We'll get less nominal bandwidth but higher actual performance and much better stability. Those will be hooked up via Pipeline 220s at about $2K each to provide basic firewall protection where it belongs - in the router. Internally all the other stuff is going away in favor of a 16 port switch stack that feeds unmanaged hubs in user areas at an estimated cost, mostly for re-wiring and new power connections in user offices, of about $11,400.

   
   

   Externally Worldcom will also be providing company paid home user connectivity for those sales and excutive personnel who currently have, or are otherwise authorized to get, company paid home PCs. These will become Linux machines connected to the internet via Ascend pipelines, again with built in firewalls but, more importantly, with built in VPN capabilities that let traffic to and from the office system be fully protected without imposing burdens on other traffic.



5. Software Conversion
   

   The existing web services software is eating an enormous amount of resources. Not only does it need two rackmounts of little servers to run but there's a 24 x 7 babysitting service and an additional staff of three day shift programmers who develop for it. The project manager, who's definitely on my short list for next week, rather shamefacedly admitted that they'd been brought down by both Code Red and nimdA but it's actually worse than that - they had to re-install NT on the four servers used for user file exchanges in early September and promptly got nailed by both of them again.

   
   

   The people in sales tell me that most of this stuff does fundamentally the wrong things. The user file exchange works well but what they want for the other stuff looks a lot more like a CRM/Supply chain system from Peoplesoft or Oracle than the combo webstore and batch order service that seems to have evolved here. The application does not, however, warrant the real heavyweight stuff because there's no ERP backend to link to - and no need for one either. We're just too simple to justify the overhead involved.

   
   

   The argument for assembly of a custom solution is very strong but I want to be running on the new software January 1st so I'm hedging my bets. We're getting four vendors of packaged solutions in to show their products to our sales staff; running an internal design competition for a better custom solution; and, asking two companies, one using Unify Vision and the other specializing in PHP/Postgressql applications, to provide custom designs and matching quotes.

Impact Summary

From a short term perspective these actions seem to be about using server consolidation to get a handle on costs and related issues like system failure and information security. It is true that, outside of product testing, we'll be going from 14 NT servers to 2 Unix servers and eliminating 107 desktop computers while dropping half the IT staff and one third of our staffing costs, but those savings are not the point. As discussed in my book The Unix Guide to Defenestration, the Unix transition is not an end in itself but a means to an end. The goal isn't to swap one set of hardware and software for another but to swap one culture for another.

At present users have desktop machines over which they have no control. They think they do and refer to these machines as "theirs" but, in reality, these can't be used for anything more productive than playing Solitaire if the servers go down - and the servers go down far too often. When we're done with the Unix transition the users will have smart displays on their desktop and these too will turn into scrap if the servers go down - but these servers won't do that. Everything on them is redundant, we will have separate teams running the two machines and independent power and network connections for them. Sometimes one will fail, sometimes the other will fail, but the probability of both failing in the same interval is vanishingly close to zero. Direct user control isn't any stronger than it was with Windows servers, but the pretense will be gone and reliability improved.

In the past, systems people were encouraged to mind their own business and user ideas for systems services had to pass through various management filters to be considered. In the future systems people will focus on working directly with users and new ideas can be evaluated and acted on within hours.

A few systems people will, of course, try to maintain their previous power relationships with users but many of those users will have adopted Linux at home and have developed very good understandings of what can, or cannot, reasonably be done. IT staffers who try this, whether new hires or veterans who managed to hide those attitudes from me, will find it impossible to bluff knowledgeable users. As a result real power over systems decisions will migrate to the users, counter balancing the centralizing tendency of the Unix/Smart Display architecture.

Will everything in this example be rosy by March of 2002? No, there will probably be outstanding issues on web services and executive reporting, but costs will be under control, the relationship between users and systems staff will have changed, and control will be where it belongs: in user hands.