% fortune -ae paul murphy

DTrace and the Linux bunker mentality

Since I don't usually get many chances to work with DTrace I can't claim to have any serious hands on experience with it - but I do know that it's insanely great because it works well, it's easy to use, and it doesn't impose significant loads or risks of its own.

DTrace is an open source Sun product that made its debut with Solaris 10 but has since been ported to MacOS X and various other BSD derivatives. IBM is reported busily copying it into AIX (along with containers, zones, and whole bunch of other things they ridiculed when Sun introduced them) and only Linux is left without significant corporate or community support for a DTrace port.

Instead the Linux kernel group has been working on a thing called SystemTap. Here's a bit of what DTrace developer Brian Cantrel had to say about that in his blog:

The interest in DTrace on Linux is heating up again -- this time in an inferno on the Linux 2008 Kernel Summit discussion list. Under discussion is SystemTap, the Linux-born DTrace-knockoff, with people like Ted Ts'o explaining why they find SystemTap generally unusable ("Do you really expect system administrators to use this tool?") and in stark contrast to DTrace ("it just works").

While the comparison is clearly flattering, I find it a bit disappointing that no one in the discussion seems to realize that DTrace "just works" not merely by implementation, but also by design. Over and over again, we made architectural and technical design decisions that would yield an instrumentation framework that would be not just safe, powerful and flexible, but also usable. The subtle bit here is that many of those decisions were not at the surface of the system (where the discussion on the Linux list seems to be currently mired), but in its guts. To phrase it more concretely, innovations like CTF, DOF and provider-specified stability may seem like mind-numbing, arcane implementation detail (and okay, they probably are that too), but they are the foundation upon which the usability of DTrace is built. If you don't solve the problems that they solve, you won't have a system anywhere near as usable as DTrace.

So does SystemTap appreciate either the importance of these problems or the scope of their solutions? Almost certainly not -- for if they did, they would come to the same conclusion that technologists at Apple, QNX, and the FreeBSD project have come to: the only way to have a system at parity with DTrace is to port DTrace.

So why isn't the Linux kernel group doing what the BSD people did: seeing a better idea and grabbing it with both hands?

DTrace is open source, the ideas behind it are public, the MacOS X port is a success - and, minor limitations aside, exceedingly useful. So why are the Linux people fighting to protect an obviously bad idea? Because many of them see Sun as the enemy? because they're still reacting to SCO by pretending that Linux isn't Unix and they're a copy-nothing shop? because they don't have the IBM's cheerful certainty that the customer won't know where the ideas came from? because they didn't invent it? or, as I think, all of the above?

There's no catch here: if SCO eventually wins its case (as I'm sure it will) the losers will be IBM and patsies like Novel, not Linux -and taking a few good ideas from the Solaris side of the Unix house to incorporate in the Linux side is well within open source norms, not seriously impeded by licensing, and good for both.

So why not? my guess (and that's all it is) is that some of the power players are suffering severe not invented here syndrome, leveraging the anti-Sun feeling in the x86 Unix community to inflict their opinions on others, and generally displaying a lack of technical integrity in trying to re-invent a square wheel alternative to an industry standard for no good reason other than personal antipathies.


Paul Murphy wrote and published The Unix Guide to Defenestration. Murphy is a 25-year veteran of the I.T. consulting industry, specializing in Unix and Unix-related management issues.