Note: This is a draft of an article for Linuxworld.com (IDG). Please do not copy or release this without their permission. Comments addressed to me are, of course, solicited.
About a month ago the SANS Institute, in co-operation with the FBI, released its list of "The Twenty Most Critical Internet Security Vulnerabilities (Updated) ~ The Experts' Consensus" for 2002.
The information provided was picked up and relayed to the public by many news sites and major newspapers at least across the United States and Canada. Although the SANS Institute notes, further down in the top20 page, that this is actually two Top Ten lists, even sophisticated publications such as Computerworld, which referred to the list as the "top 20" throughout its front page treatment of the story, didn't make that distinction clear to readers.
In addition, you have to dig fairly deeply into the announcement to see that the top 10 Windows list is limited to a few current variants of major Microsoft Windows brand server operating systems while the Unix list includes applications, desktops, and bugs going back at least as far as 1990. More subtly, the title coupled with the silent omission of all information about the relative costs and risks represented by the listed vulnerabilities invites readers to impute a rational basis, such as cost or risk, for the rankings shown.
The result was, I think, misleading in that many readers and editors would have seen this as an FBI certification of the relative equality of security problems between systems running Microsoft Windows and those running Unix.
Here's how, for example, Business Week headlined its report:
FBI names most wanted security flawsThe bureau and a prestigious computer-security research group announce the 20 most serious security flaws affecting both Windows and Unix systems.
This, of course, is obvious nonsense, but a majority of the popular press picking up the story reported it without warning readers or viewers either that there were two distinct top ten lists or that the two lists do not cover comparable product sets.
Since I couldn't tell from the top20 page whether the list compilers set out to create a false impression of parity or merely stumbled into doing so I decided to ask the people at the institute for help:
After reviewing your listings carefully I'm unable to determine:
1- the basis on which vulnerabilities were included in the list; or,
2- the basis for the rankings.
Could you please help me understand the answers to these two questions?
I have, however, concluded that your basis for classifying a specific
vulnerability as pertaining to either Windows or Unix consisted of
the rule: if it involves at least one product not sold by Microsoft, classify
it as a Unix issue. Please help me understand what the rule was, if this
is incorrect.
In addition I would have thought that a list of "internet vulnerabilities"
might have included:
1- Microsoft desktop vulnerabilities; and,
2- Cisco and other router vulnerabilities.
Could you please help me understand why these were omitted?
And got a prompt reply:
From: [suppressed]
Date: Mon, 14 Oct 2002 09:53:55 EDT
Subject: Re: Top 20 lists
To: murph@winface.com
MIME-Version: 1.0
Thanks for the note, Paul.
The system we use is to empower the people who have to clean up after the
attacks and the people who staff red teams, to rate the ones that they feel
are most critical-- the ones that lead to significant numbers of successful
compromises.
The Cisco ones are planned for group two (early 2003). And the desktop ones
will probably be there, too.
What is your role?
Alan
which seemed to indicate that the choices reflect the distribution of successful attacks, so I asked for the numbers:
Thanks for the quick response.
However, I wonder if you could be a little more precise in your answers since
your comment about empowering people seems more of a restatement of the
ascription to expertise in the your top20 page than an answer to my questions.
Of course, I may just be missing the obvious because I have no information on
the number of successful system compromises related to each vulnerability on
your list. Since you must have it, would it be possible for you to share
it with me?
and that question too got a quick response:
Paul,
Tell me more about your role in the community?
Alan
I've heard the macroLenat "Just who do you think you are ... ?" speech since high school (you're surprised, right?) so I fired back a request that he explain how my role would affect the answer and decided to explore the issue by looking at the clues in their top20 page to see if I could determine:
Two things that may have a bearing on these decision processes stand out from the page:
Instead both refer to systems, something which makes no practical difference on the Windows side but allows weaknesses in third party applications running on Unix to be presented as Unix System vulnerabilities; and,
This type of system recovery work, rare in Unix but common in the Microsoft Windows environment, reloads previously resolved vulnerabilities and is a key reason Unix web servers around the world continue to be bombarded by new instances of both Code Red and Nimda.
The SANS document combines the two top ten lists with recognition and remediation information about each vulnerability cited. That information uses the CVE - Common Vulnerability and Exposures - nomenclature developed by Mitre and refers to the detail maintained by the ICAT team at the National Institute of Standards and Technology.
As a result a naive reader could reasonably assume that the CVE information forms the basis for both the rankings and the Windows/Unix distinctions.
In total the SANS Institute's top20 page cites 89 different ICAT listings in support of its Microsoft top 10 list and 147 unique listings with respect to the Unix top ten. Among these, 16 (17%) of the Microsoft citations and 75 (51%) of the Unix citations have 1999 prefixes while two Microsoft citations, and eight Unix citations, are used to document more than one vulnerability.
To see how the detailed CVE (or CAN, for "Candidate CVE(?)") listings illuminate the top ten lists, consider just the two highest ranked vulnerabilities for each environment:
From the Windows list:
and from the Unix list:
Notice that these seem quite parallel, to a non technical reader it would appear that the order is reversed for the two environments but both have roughly comparable web server and remote access vulnerabilities. Unfortunately this implicit conclusion is completely wrong.
The SANS top20 page cites only 7 of the 17 current (year 2002) CVEs listed for IIS in the ICAT database in support of its decision to award this product set top billing on that list.
On the Unix side, however, SANS cites nine year 2002 ICAT vulnerabilities against Apache. There are currently 21 listed, of which eight were added on or after 09/05/02. Of the four missing ones, two, 2002-661 and 654, have the words "Apache 2.0 through 2.0.39 on Windows, OS2, and Netware" in the first sentence but I could see nothing to suggest why the other two were omitted.
The nine are:
| CAN/CVE | Nature of issue? | Applies to? |
|---|---|---|
| CAN-2002-0061 | Apache for Win32 before 1.3.24, and 2.0.x before 2.0.34-beta | Windows Only |
| CVE-2002-0082 | mod_ssl pre 2.8.7-1.3.23, and Apache-SSL before 1.3.22+1.46 | Cross Platform |
| CAN-2002-0257 | Cross-site scripting vulnerability in auction.pl | Third party Application |
| CAN-2002-0392 | Apache 1.3 through 1.3.24, and Apache 2.0 through 2.0.36 | Windows Only |
| CAN-2002-0513 | PHP administration script in popper_mod 1.2.1 | Third party Application |
| CAN-2002-0655 | OpenSSL pre 9.7beta2 | Cross Platform |
| CAN-2002-0656 | OpenSSL pre 9.7beta2 | Cross Platform |
| CAN-2002-0657 | OpenSSL pre 9.7beta2 | Cross Platform |
| CAN-2002-0682 | Cross-site scripting vulnerability in Apache Tomcat 4.0.3 | Cross Platform |
Only two of these unambiguously apply to Apache running on Unix, two apply uniquely to Apache on Windows; the three rather dubiously applicable OpenSSL CVEs are repeated in support of the SANS decision to list OpenSSL on Unix as the number three Unix vulnerability, and two represent problems with third party (and cross platform) applications that have little to do with either Unix or Apache.
I don't see anything here that remotely justifies considering Apache the number two source of Unix vulnerability but, once I looked at a few other rankings the general pattern seemed clear: the vulnerabilities cited against Microsoft products generally under represent the ITAC database, sometimes by considerable margins, while those cited in support of Unix vulnerabilities often seem to have little or nothing to do with Unix. Instead, problems affecting applications, problems that have long since been remediated, problems affecting network gear, and problems affecting cross platform libraries, are all cited in support of the Unix system vulnerability list.
The number four Unix vulnerability, for example, is given as SNMP- something about which the SANS Institute's top20 page says:
SNMP is not unique to Unix; it is extensively used on Windows, in networking equipment, printers and embedded devices. But the majority of SNMP-related attacks seen thus far have occurred on Unix systems with poor SNMP configurations.
The first statement above is certainly true and supported by the ICAT data, but the other is neither supported by the evidence provided nor pertinent to the question. At first glance it looks as if the writer is trying to explain why SNMP problems are classified as Unix system problems, but a second look will show that it fails to give the kind of factual information needed. Instead a matter of opinion is supported by a statement of opinion and the real question: "how important are SNMP attacks relative to other attacks on Unix?" is silently elided.
I cannot tell from the document whether this kind of wording reflects sharp practice or just fuzzy thinking, but the same kind of thing occurs in other places - starting with use of the word "system" in the list titles; something which I'm guessing was done as cover in case anyone questioned counting Unix application vulnerabilities as Unix vulnerabilities.
Consider, for example, the word "remote" in both the second ranked Microsoft systems vulnerability: "Microsoft Data Access Components (MDAC) -- Remote Data Services" and in the label used for the number one Unix vulnerability: "Remote Procedure Calls (RPC)." I think that this repeated use of the key word "remote" makes the two sound fairly parallel; but they're not.
In the Unix world "RPC" has a reasonably precise and widely understood meaning that has been consistent over time. Thus a 1994 RPC vulnerability in Ultrix 4.3A (CVE-1999-0170) can be cited to support the idea that RPCs represent the number one source of Unix vulnerabilities.
In contrast, Microsoft's tendency to rename things between product announcements means that "Microsoft Data Access Components (MDAC) -- Remote Data Services" has no real meaning beyond its reference to one specific, and short lived, product release. Here's the SANS description for what they see as the second highest ranked Microsoft internet vulnerability:
The Remote Data Services (RDS) component in older versions of Microsoft Data Access Components (MDAC) has a program flaw
...affecting...
Most Microsoft Windows NT 4.0 systems running IIS 3.0 or 4.0, Remote Data Services 1.5, or Visual Studio 6.0
SANS cites one CVE, from 1999, in support of this artifact's claim to second place. The ITAC database, in contrast, has rather a lot of current CVEs relating to Microsoft's own RPC implementations for NT, Windows 2000, and XP as well as RPC-like products and APIs --including 45 just for ActiveX.
In sharp contrast to the lack of text supporting the status accorded MDAC, the top20 page lists 26 CVEs (65% of them from 1999) in support of the idea that RPCs represent the number one Unix vulnerability.
It's not obvious that RPC represents the number one Unix vulnerability - if you check the targets list at incidents.org RPC attacks are number six on the list for the year (dropping to number 7 on the front page current stats); well behind attacks on other common functions like FTP and SMTP.
The more of these RPC related CVEs I read, the odder this choice for first place seemed; then I noticed that almost all of the cited material referred explicitly to Sun - not least by naming the protocols "SunRPC" - while the SANS Institute top 20 page uses "RPC" and the only mention of Sun outside a URL context contains a logical error suggesting a bit of last minute dictation slipping past an editor:
In addition to this inherent transmission insecurity, critical flaws have been found in many versions of FTP server software, both those provided by operating system vendors (Sun, HP-UX, etc) and those developed by the open source community (WU-FTPD, ProFTPD, etc)
As a result I started to think that having a finger pointing at Sun but not Microsoft might have been a primary selection criterion that someone later attempted to disguise by sanitizing the top 20 page to remove the direct references to Sun that occur in the source documents. If so, you have to ask why anyone would bother; or, more cynically, what they might have wanted to hide.
Consider CVE CAN-2002-0391. It is cited by SANS as showing the Unix vulnerability to RPC exploits and refers directly to Sun as the source of the offending SunRPC code, but doesn't say that the vulnerability is peculiar to Unix:
Integer overflow in xdr_array function in RPC servers for operating systems that use libc, glibc, or other code based on SunRPC, allows remote attackers to execute arbitrary code by passing a large number of arguments to xdr_array through RPC services such as rpc.cmsd and dmispd.
There is more detail in the matching CERT advisory. This makes it clear that SunRPC is found in a lot of operating systems:
Because SunRPC-derived XDR libraries are used by a variety of vendors in a variety of applications, this defect may lead to a number of differing security problems. Exploiting this vulnerability will lead to denial of service, execution of arbitrary code, or the disclosure of sensitive information.
In fact, Netmanage wrote an RPC/XDR dynamic link library for Windows 3.0 and RPC capabilities have been common in Microsoft operating environments released since, but this CVE (CAN-2002-1141), illustrating CERT's "variety of vendors", isn't mentioned in the SANS document. It says:
An input validation error in the Sun Microsystems RPC library Services for Unix 3.0 Interix SD, as implemented on Microsoft Windows NT4, 2000, and XP, allows remote attackers to cause a denial of service via malformed fragmented RPC client packets, aka "Denial of service by sending an invalid RPC request."
There may, or may not, be a pattern here in which CVEs that point at Sun are selected while those pointing at Microsoft are skipped, but even if this pattern exists it would not explain either why these particular vulnerabilities were selected or how the ranking was done.
I asked Mr. Paller to review the draft for this article but received no reply. However, when
the editor for Linuxworld asked for comment he got this response -but without a copy to me:
I've never seen it done better: not only is this gracefull and even elequent, but it still manages to put me firmly in my place. Now if he would only answer the question... |
In an effort to understand the latter I considered, and rejected, seven alternative hypotheses:
The ICAT group at NIST maintains a top ten list ranked according to "the number of requests made for a particular vulnerability in ICAT." Six of the ICAT top ten for 2002 relate to issues affecting Microsoft products; two refer to OpenSSL, one to apache, and one to CDE.
Clearly, therefore, technical interest had little or nothing to do with the SANS ranking.
The Windows list seems limited to current major market server OS variants while CDE desktops feature front and center on the Unix side.
The Windows list is fairly current with the exception of item two which is technically limited to an NT variant marketed in 1999 but the Unix list ranges all the way back 1991 and seems to consider as core Unix almost anything with a defect that could remotely be attached to, or run on, a Unix machine.
For example the first CVE listed on the top20 page in support of the RPC ranking is CVE-1999-0166. This, pertains to a CERT advisory from 1991 reporting an already patched problem with SunOS 4.1 and 4.0.3. The next two refer to the same issue on SunOS and number four reflects a closely related 1994 issue with Ultrix.
In fact, most of the issues listed for Unix systems have been resolved years ago. Number 10 has one CVE, a candidate from 1999 that isn't found by searching the on-line database while all five of the CVE's supporting number 6, trust services, are from 1999.
Clearly, therefore, neither actual currency nor concurrency with the Microsoft list governed the Unix selections or rankings.
Of five 2002 CANs listed for SunRPC, four have the words "Currently we are not aware of any exploits for this issue" attached to them on the security focus exploits database. One of these four has a rumored, but unknown exploit, and the fifth has a sketch for a possible exploit but I could find no record of a single successful attack based on any of these.
Clearly therefore, demonstrated risk had little or nothing to do with the rankings.
Here are the two lists in full with the original labels:
Top Vulnerabilities to Windows Systems
- W1 Internet Information Services (IIS)
- W2 Microsoft Data Access Components (MDAC) -- Remote Data Services
- W3 Microsoft SQL Server
- W4 NETBIOS -- Unprotected Windows Networking Shares
- W5 Anonymous Logon -- Null Sessions
- W6 LAN Manager Authentication -- Weak LM Hashing
- W7 General Windows Authentication -- Accounts with No Passwords or Weak Passwords
- W8 Internet Explorer
- W9 Remote Registry Access
- W10 Windows Scripting Host
Top Vulnerabilities to Unix Systems
- U1 Remote Procedure Calls (RPC)
- U2 Apache Web Server
- U3 Secure Shell (SSH)
- U4 Simple Network Management Protocol (SNMP)
- U5 File Transfer Protocol (FTP)
- U6 R-Services -- Trust Relationships
- U7 Line Printer Daemon (LPD)
- U8 Sendmail
- U9 BIND/DNS
- U10 General Unix Authentication -- Accounts with No Passwords or Weak Passwords
A cost or risk basis is what most people would expect: a 20 worst list should be ordered according to damage or cost. Unfortunately even casual inspection suggests that this doesn't apply here.
We don't have actual numbers for any of these things but we know that problems with sendmail, especially open relays, have cost far more to recognize and remediate than problems with secure shell, Apache, or R-services.
If you search ICAT according to the top three Unix vulnerabilities you start out with a nice declining series: 68, 67, and 49 hits respectively; but just as you get that "ah hah!" feeling number four gets 58 hits and blows that explanation away. Furthermore, neither list matches the incident reports at incident.org.
Again, this might have been logical since third parties, the people who have their bandwidth reduced by things like Code Red but have no Microsoft products on site, are unarguably the "innocent victims" here. This list, however, appears to equate weaknesses in IIS that affect millions of third parties to Unix RPC buffer vulnerabilities that have yet to victimize their first third party.
So if it wasn't one of the above, what basis did they use first for the selection and then for the ranking? I still don't know, but I have two hypotheses:
Personally I'm more inclined to believe in stupidity than malice but, either way, this "top 20" announcement seems to me to have been an extremely misleading and quite possibly disingenuous piece of work that the Institute and the FBI should promptly correct.