This is a draft for my LinuxWorld.com series. Please do not copy or distribute this without the permission of Linuxworld.com.
This is the last of three articles in which Paul Murphy takes a close hard look at running Linux on the mainframe.
In this one he suggests that IBM should play by the same rules that everyone else has to adhere to.
| Not why the aliens came? | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Although not its intention, SETI@home is a floating-point benchmark. A
reader of part one in this series, who wished to remain anonymous, sent me his
results of running SETI@home on Linux on a z900 with one dedicated CPU. He asked that I not
report his actual results. However, the
SETI@home platform page offers a tabulation by host processor and operating
system. With 409 workunits completed, the average time for the System 390
is 43 hours, 42 minutes, and 7 seconds.
By way of comparison:
|
When I was first asked about Linux on the mainframe, I discovered that I'd never really thought about it at all, having, instead, taken IBM's imprimatur at face value. When I stopped to take a closer look in article one the thing that bothered me most was IBM's reliance on very dubious third party performance claims and their refusal to provide solid cost/performance data.
When I looked carefully at the hardware it seemed to be pretty much lower mid-range stuff, more power than an eight processor Xeon at 900MHZ but less than a 24 processor Sun 6800. The consolidation market the machine is nominally aimed at doesn't seem that good a fit either. As discussed in article two in this series combining hardware and software consolidation by amalgamating workloads usually makes far more sense than consolidating only the hardware by replicating numerous small machines as Linux instances on a mainframe.
Most of these conclusions are based, however, on estimated performance and estimated pricing. In the end that comes down to speculation and does not suffice as the basis for making a final judgment about the comparative price/performance positioning of mainframe Linux. To resolve the issue we need verified cost and performance data for mainframe Linux -something IBM has, so far, refused to provide.
That's what this article is about: pointing out that IBM is trying to play in the open source world without adopting open source behavioral norms -like telling us what things cost and how well they work.
The essence of the scientific method, and of the open source movement, is full disclosure, peer review, replication, and the enthusiastic incorporation of other people's ideas into one's own work.
| Does "open source" assume "commodity hardware"? |
|---|
|
IBM is making its changes to Linux source available to the open source community, so it's open, right?
Maybe, but changes being made apply only to the mainframe and IBM makes essentially all of the new VM capable mainframes being sold. Hitachi announced, in March 2000, that they would stop selling mainframes to new customers and Amdahl made a similar announcement in October of 2000. (See: this discussion.) Both continue to support existing customers and can use IBM's Linux changes, but both have trivial market share relative to IBM. We think of Linux as an open system - but that includes a hidden assumption about commodity hardware. By itself, open software isn't very useful, to make an open system using it you have to run it on something. So if the only hardware this stuff run on is intensely proprietory, where is the community value in IBM's claim to openness? |
Academic research, and most scientific progress, is fundamentally an open source phenomenon. The free and open exchange of ideas, the distinction between critizing an idea and its orginator, and the ability to adopt and expand other people's ideas, characterize Linux --and the physics behind the box it runs on.
Unfortunately, that openness is missing from IBM's attempts to sell mainframe Linux; what we see, instead, are non disclosure agreements, vague assertions about undefined value propositions, nonsensical TCO "studies," and dubious performance claims.
If IBM's mainframe, or iSeries, Linux is to break out of the IBM cloister, IBM has to adopt the openness that goes with open source: participate in standard benchmark tests, publish cost information, allow third party disclosure of results, and treat well intentioned criticism as helpful rather than antagonistic.
If the VM/Linux product is worthwhile, doing this should be easy and profitable. If ghosting is cheaper, better, faster, than alternatives then people will buy it. But refusing to tell us what it costs, where or when it is better, or how its performance compares to competitive offerings conveys the opposite message: that it probably isn't better, faster, or cheaper.
If IBM wants to make the case for ghosting all it needs to do is:
Public Benchmarking
| Silence is Golden |
|---|
|
When the San Francisco
Chronicle claimed that you can replace 750 "big Sun Microsystems servers" with a single S/390, IBM didn't
rush to inform anyone either that the "big Sun Microsystems servers" involved were five year old workstations
or that the actual conversion discussed had never been done.
When LinuxPlanet.com headlined the story of David Boyes and his 41,400 ghosts with the claim that each was "doing real work under simulated load as web servers" IBM didn't rush out a correction. When Equant claimed on Bynari's behalf that you can add 45,000 Bynari Insight users, and 2.2 Terabytes of ESS/SHART DASD to a mainframe for a total cost of only $85,000 including installation and three years of support, IBM not only didn't repudiate the thing, they sent it to me as supporting evidence. When Amy Newman reported on Serverwatch that "Performance tests conducted by Sendmail and IBM found that a single IBM eServer z900 mainframe could support more than 2 million users, making it the largest single-server e-mail system .."neither Sendmail nor IBM hurried to inform her that this number was predicted rather than achieved; that "users" meant "accounts" of which only 10% are considered active; or that the activity considered was limited to pop3 users downloading five relatively small messages per day. We've had a lot of conversations with Jeff Morris and others at Sendmail about this and they've just put up a new whitepaper on performance testing the z900 as an email server. It says:
SAMS Multi-server POP tests indicate that three instances of SAMS, front ended with Sendmail's Message Access Proxy, running multiple Linux images across four zSeries 900 CP is capable of supporting 750,000 POP users. Be aware that there were two distinct test series. In one series they used four z900 CPUs to successfully run 750,000 POP3 accounts using a process in which each account is updated with new mail files and 10% of the accounts are activated just long enough to download five messages during each half hour test interval. In the second test series, they used four CPUs to run 14,000 IMAP users and these people are assumed to stay logged in for the duration of the test. To draw their conclusions they multiplied the 75,000 actual POP3 users by 10 to get 750,000 accounts and then multiplied that by four because they used one quarter of the maximum number of CPUs in a z900. Note that this assumes that POP3 processing is CPU bound; the actual number of accounts will be less than the 75000 x 10 x 4 = 3,000,000 if processing is I/O or network bound instead. By way of comparison a basically similar 1998 test series done using a 15 CPU, 250MHZ, Sun 6000 is claimed to have shown it supporting a mix consisting of 100,000 concurrent pop3 users and 20,000 IMAP users. Similarly an audited report on using Oracle's messaging server run in mid 1999 on a Sun 6500 with 22 CPUs at 336MHZ showed that it would support about 350,000 POP3 users (i.e. 3.5 million accounts) concurrently with 10,000 IMAP users. If the software were comparable (it isn't) and the process were CPU bound (it isn't) then the logic here would suggest that it would take 1.34 (=((350000/(75000/4))+10000/(14000/4))/16) fully configured z900s to match a three year old Sun 6500 running at 336MHZ.
|
The two most important public benchmarking organizations currently working are:
IBM participates in both sets of activities and offers apparently compliant benchmark results in most categories for all major product lines except the mainframe.
Although I personally believe that the available benchmarks from TPC and SPEC would be both fair and adequate, Joann Duguid, Director of Linux on IBM eServer zSeries, explicitly told me that these benchmarks do not reflect IBM's value proposition.
The implication here is that these benchmarks are somehow inadequate to convey the unique positive value of the mainframe. If so, all IBM has to do is:
Those organizations can then judge the value of the excerise and decide whether or not the claims make sense. If they do, other vendors can either run the same benchmark or face a potential loss of market share as customers scramble to take advantage of IBM's unique value.
Publish Cost Information
Every other major vendor provides web accessible information allowing potential customers to configure gear and estimate its list price. If IBM wants to sell mainframe Linux into the general Unix community it should adopt community standards and publish list pricing - including the services and license cost information needed to complete a base cost estimate.
Take off the handcuffs
IBM should remove the veil of secrecy by opening up its Mainframe Linux demonstration site to:
People could then do things like arrange for a few thousand internet users to log on at about the same time and each run some combination of small, but widely known and understood, benchmarks like those being collected at The Linux Benchmarking Project.
Benchmark Commercial Applications
For software developers doing benchmarks often costs real money but usually pays off in increased product visibility and credible access to the markets represented by the gear on which benchmarking is done. More IBM participation in this kind of work with respect to mainframe Linux would enable us to make better judgments about the product's appropriate use and probable performance in real world applications.
Except for the lack of financial information in the results supplied by, or on behalf of, IBM, both the Lotus Domino and Sanchez Financial benchmarks referred to in article one fit this mold. However, there are many other examples where vendors like Sun, Dell, HP, and the non mainframe hardware divisions of IBM have already provided relative cost and performance information but IBM has not done so with respect to the mainframe.
IBM claims, for example, that their mainframe gear is appropriate to enterprise applications like SAP but does not participate (with respect to the mainframe) in the major Oracle, PeopleSoft, or SAP benchmarking tests. Doing so would provide the comparative cost and performance information needed by people deciding between their VM/Linux offerings and everyone else's Unix products.
IBM is trying to add open source to VM on the zSeries without adopting the philosophies and business practices that go with it. Speaking personally, I didn't have a problem with this until I looked closely and discovered that IBM itself is extremely careful about the wording of its performance and other claims for mainframe Linux --but underpins its sales efforts with third party reports of unrepeatable experiments, undocumented assertions, and seemingly absurd TCO "studies."
There's a lot more at stake here than a few mainframe sales. There are several hundred thousand functioning VMS, MPE, Tru64, and HP-UX shops which are being forced to consider alternatives -if they make an IBM decision based on unsupported opinion, nonsense performance claims, or phony TCO "studies" they may very well end up having to ask themselves, as a class, whether or not they might be entitled to compensation from IBM.
The bottom line is simple: absurd performance claims - replace 750 Sun machines with one mainframe; inane costing -add 45,000 Bynari user licenses and 2.2TeraBytes of high speed buffered disk for $85,000 including three years of operation; and innuendo based marketing - that shouts "41,400 instances" or "2 million email users" but hides the equivelent of "provided they don't actually do anything" somewhere in the fine print; coupled with the secrecy, handwaving, and general clubiness of a value proposition outsiders don't have the right or ability to understand has led me to disbelieve IBM's claims on the entire Linux on mainframe idea.
Before I looked closely it seemed like a good idea; right now my attitude is "show me". Show me why it makes sense to spend five million bucks on a zSeries with all of 16 CPUs that doesn't even have disk and tape resources let alone software when I can, for the same money, put in 80 fully loaded four way Linux servers and skip the joys of hiring, and paying for, a small army of VM experts to run the hardware.