% fortune -ae paul murphy

The need for a mixed load benchmark

If you work at a small to mid size company and are considering a server upgrade but aren't committed to one vendor, you're likely to have an interesting selection problem with deep ramifications throughout your operation.

The problem arises because the middle range for memory, storage, and CPU upgrades within product lines are usually where you get the best bang for the buck - but making use of those additional resources almost always requires process change in your data center. Basically a mid range Dell is now about four times as powerful as one from four years ago, so if the workload you put on a mid range then has only doubled since, you're likely to buy today's low end and thus get a lesser bang for your buck than if you bought today's mid range.

The currently popular solution to this is to use virtualization to pretend that the new server you buy is really just several smaller ones. Back in the 1960s, when data processing adopted this technology, it made sense because System 360 memory management limitations combined with costs to make partitioning a smart way of separating development work from production work without paying for a second machine. Today, of course, that second machine costs a few thousand bucks, but commitment to the technology continues.

To see how absurd this really is, consider that the big excitement among the PC and data processing people using this technology today is that things have improved, over only fifty years, to the point that system resources can now be dynamically re-allocated between partitions and/or ghosted OSes (hosted OS instances) as workload conditions change - something the basic Unix scheduler has been able to do reliably since about 1972.

In reality therefore the right way to get the most out of your hardware is to run multiple applications under one OS running directly on the hardware. This minimizes complexity, minimizes overhead, and allows the scheduler to fairly allocate all available resources all of the time.

(Note, in this context, that Solaris containers extend, but do not replace, the usual Unix group id management approach and can be very useful for things like enhancing project security or ensuring near real time response on specified devices while LDOMS are simply Sun's way of meeting customer expectations on virtualization.)

However, when you want to migrate from many smaller boxes to a pair of mid range units you need to know, in advance, roughly how your applications will work on the new gear.

A critical first step is to plot, preferably at minute resolution, usage patterns for all major resources - CPU, memory, disk I/O, storage - over a period of at least a month, and preferably one including a quarter end, for each machine you're going to replace.

Every OS has some available tools for this - in the case of Solaris, for example, the the DTrace toolkit works for Solaris 10 and the SEtoolkit works well for earlier releases.

Sum those graphs and you should get a picture of the minimums your new machine will have to meet - and all of that's pretty easy except that I don't know of anything that can help you predict what interactions, if any, in the combined workload will affect the performance you actually get.

Now whether your target is Linux or Solaris there are standard things you can do - once you get your hands on the gear. On Solaris, for example, you use ZFS and separate controller/storage combinations for major applications, use the fastest network ports first (there are two 10GB ports on the T5220 and higher), use resource control to ensure that specified ports (e.g. those for Sybase or Domino) get priority attention, and so on.

Notice, however, that I said you can't do this until you get your hands on the box - and then if it turns out you guessed wrong it will be too late for all but palliative sysadmining, and that's not a good position to find yourself in.

What's needed as a kind of guessing guide is a solid multi-application benchmark with real results. Unfortunately I don't know of one: VMware's VMmark multi-application benchmark was developed specifically to address this in an x86 virtualization context, but both the limits they impose, and those whose effect they measure, are exactly those use of the scheduler gets away from.

What's needed, therefore, is for someone, presumably at Sun, to develop a true concurrent usage, multi-application, benchmark that more or less reflects what real people in real businesses can do when they look at options like replacing four older V880s or a couple of Dell racks with a single T5440.

I'm thinking, for example, of something that combines the usual SAMP web and e-communications services, a couple of different RDBMS/application combinations, some PC backup and file support, a few hundred Sun Rays, and maybe a heavy cruncher like an engineering materials management and simulation application, all in one test with realistic usage patterns over at least a one week period and targeted at workloads in the 100 - 1,000 combined user range.

But, for right now, the bottom line is that's its trial and error - in other words, risky enough to discourage many of those facing this particular "interesting problem" from even trying what's obviously a better solution - or, in other words, making something like this available could help Unix sales by reducing both real and perceived risks.


Paul Murphy wrote and published The Unix Guide to Defenestration. Murphy is a 25-year veteran of the I.T. consulting industry, specializing in Unix and Unix-related management issues.