% fortune -ae paul murphy

The sysadmin job: what's changed in ten years

About eight years ago I spent nearly ten months working as a hands on sysadmin and DBA in an HP-UX shop with Sybase.

In this kind of job I generally use a simple csh alias to make both task specific and daily summary notes on what I've done:

alias addnt 'echo \!* " "`date` >>$HOME/job_notes.list '

Some samples from October of 1999:

08:51 16:54 ERP re backup failed -K again - reset permissions on transfer files Fri Oct 1 16:54:19
08:06 17:47 misc re ERP init load, backup, and reboot tests on reserve k580 -aok- osa re export stayed late for bev Mon Oct 4 17:47:10
08:18 11:41 ERP re working on setup of restricted menus Tue Oct 5 11:41:27
08:10 17:19 ERP re bitch and glicth commitee and various menu debugs Wed Oct 6 16:32:52
08:23 17:19 ERP test reset all user accounts, emptied databases, reloaded from last week, set scripts and tested Thu Oct 7 17:19:03
19:10 19:30 ERP run dbcfix for pymt and general then restart all via delayed script Fri Oct 8 19:25:21
08:28 17:05 ERP re misc problems and db recover Tue Oct 12 17:05:37
08:14 16:30 misc re hp performanc emeeting for 2 hours james at lunch - cleared batch issues on ERP Wed Oct 13 16:30:59
08:21 16:46 ERP re rev of user lists and securities tested on Primary K580 -works- Thu Oct 14 16:46:06
08:11 16:25 misc re mostly ERP babysitting worked with james re mjok, back tonight at 9:30 to expand prqt and pymt Fri Oct 15 16:25:33
03:23 10:46 General panic re production file transfer and update failure with Sysco -K again, removed line from passwd but left account - Sun Oct 16 10:46:59
08:10 17:06 ERP re misc panics claims of batch issues -none real- multiple pc connection problems Mon Oct 18 17:06:51
08:18 16:46 ERP re misc panics -rebuild of 580 find files from 26 crash on ARQ multiple pc connection problems Tue Oct 19 16:46:51
08:10 16:51 ERP re misc panics and NT re-install needed for client to work Wed Oct 20 16:51:51
08:32 17:07 ERP re misc meetings on storage cleanup and documentation Thu Oct 21 17:07:49
08:24 15:23 ERP re documentation and meeting with hp re nclass issues plus full restore Fri Oct 22 15:23:47
08:14 16:35 ERP re misc user panics and restarts look at psc tables re missing jobs on reload -K again!- Mon Oct 25 16:35:33
08:20 17:16 ERP re performance code and tests on n plus prqt panics on backup k580 Tue Oct 26 17:18:09
08:16 16:42 ERP re worked with liz on reinstall of NT reporting and re did whole thing..plus usual chris issues Thu Oct 28 16:42:08
08:3: 16:23 ERP re misc ERPs perf meeting with hp mystery of perf problem deepens .... Fri Oct 29 16:23:23
11:46 12:30 ERP re rebuild prqt and general review Sat Oct 30 12:34:21 MST 1999

So after talking to one former colleague last week I phoned another - and guess what his work week looks like? They have a project going to upgrade their ERP/SCM software to the vendor's latest release (production is two full releases behind the vendor right now) and so here's his list (somewhat paraphrased):

- The Windows people are having problems with the new client and no one really knows why
- Performance on the p550 is nowhere near expectation, spending time documenting that
- Doing a lot of user DB resets and restores as they learn to work with the new screens and reports;
- Writing process documentation and modifying old scripting for the new software;
- Setting and testing autoupdate file perms on P550 (still at 5.2 on 620); doesn't work right with AD Server.

Bottom line: since they laid him off about eight years ago they've had to pay him a lot more money, but the job itself doesn't seem to have changed much.

But it should have: technology has changed tremendously over the period and his organization has spent tens of millions keeping up. So why? why isn't he working in a gleaming data center occasionally pushing a button here or there to rebalance workloads or recover user files? My guess is that the daily realities of the sysgrunt job aren't determined by the technology, they're determined by the relationships between IT and the users.

To bring the job forward with the technology the business either isolates the data center from users (something you see fairly often) or integrates sysgrunt operations with user operations -something you see rarely. Do anything else, and the sysgrunt job doesn't change: the business just spends money peddling really hard, and getting nowhere. At my friend's company user complaints now are almost exactly what they were then; to them everything IT sucks: IT is slow, it's arrogant, it's uninformed, and it is failing to keep up with the times - and what really hurts him is that he thinks they're right, about all of it.


Paul Murphy wrote and published The Unix Guide to Defenestration. Murphy is a 25-year veteran of the I.T. consulting industry, specializing in Unix and Unix-related management issues.