% fortune -ae paul murphy

Working at 99% CPU utilization

The key metric used in data processing management is system utilization - and that same metric has been adopted to push the idea that consolidation through virtualization makes sense.

The key factor driving the evolution of system utilization as a management metric was the high cost of equipment - first in the 1920s when data processing was getting started, and again in the 1960s when the transition from electro-mechanical to digital equipment got under way.

What made it possible, however was something else: the fact that batch processing of after the fact data was the natural norm for the gear meant that user management expected printed reports to arrive on their desks on a predictable schedule wholly divorced from the processes generating the data.

Thus what drove data processing management to use its gear 24 x 7 was money - but what made it possible to do this was user management's acceptance of the time gaps implicit in an overall process consisting of distinct data collection, data processing, and information reporting phases.

Almost a hundred years later data processing still operates in much the same way - and the utilization metric is still their most fundamental internal performance measure. And, since their costs are still high too, the financial incentive to high utilization continues to provide justificatory power.

The usage process has, however, changed considerably: internally the time gaps for both the data collection and the data reporting phases have decreased to near invisibility - on line data entry is now standard, for example, and many traditional reports are now "printed" only as data transfers to query servers whose accessibility from user desktop PCs hides the after the fact nature of the data.

Externally, meanwhile, users have grown accustomed to PC response rates and now expect to get what they want, when they want it - and what that means is that efforts aimed at meeting IT management's expectations on utilization conflict directly with the mandate to meet user expectations.

For example, if a domino server shows above 90% average system CPU utilization during the organization's regular working hours, data processing management will consider that machine appropriate to the load - but the average user will be waiting more than two seconds for any server response and usually well over a minute to get full access to the typical inbox.

In contrast data processing management would consider a machine serving that same user community at an average 5% system utilization during working hours completely absurd - but users would typically get nearly instant response and full inbox access in a few seconds.

To me that 5% machine is a success - to a data processing guy it's both a failure and an obvious candidate for consolidation to a busier machine.

To see just how that works from a user perspective try (assuming you're using a personal computer of some kind to read this) starting some seriously CPU intensive job requiring full system resources to run, and then see what it feels like to use the browser and maybe one or two other of your favorite tools while it does.

The results you get depend on your CPU, your OS, and your default settings - but I guarantee you'll find it unpleasant unless you artificially restrict that resource hog to significantly less than half the system CPU and other resources - and, furthermore, that you'll find the system personally unusable if you let the resource hog have 95% or so of the total available CPU and memory resources.

And yet: every time we use utilization to justify consolidating processes from many boxes to one, these frustrations are exactly what we're talking about imposing on our users.

And that, I think, is the bottom line on consolidation through virtualization: it makes data processing sense - but it's terrible for users, for the business, and ultimately pretty much the opposite of what you should be doing.


Paul Murphy wrote and published The Unix Guide to Defenestration. Murphy is a 25-year veteran of the I.T. consulting industry, specializing in Unix and Unix-related management issues.