Issue TOC Print full article Print this page E-Mail this URL Discuss this article Flame the author     In this article Introduction Wants & Needs Foundstone Enterprise and FoundScan Engine 2.6 Qualys QualysGuard Intranet Scanner Harris Corp. STAT Scanner Professional Edition 5 eEye Digital Security Retina Network Security Scanner Vigilante.com SecureScan NX 2.6.50 SAINT 4.3 nCircle Network Security IP360 Vulnerability Management System 5.3 Other Products Reviewed How We Tested Web Links Report Card

We modeled our vulnerability-assessment tests on real-world conditions. Our approach was straightforward: We deployed 27 devices of different types--Windows, Linux, BSD, NetWare, Solaris, firewalls, routers and switches--with varying levels of patches and ran each scanning solution against this environment to identify known vulnerabilities. We then compared the results, measured the time each scanner took to complete the scans, and noted the state of the target systems after the scanner completed its job.

Although the task of testing 11 VA scanners against a static environment and comparing the results may seem simple, we found the exercise far from easy. Each product offers a different set of features, has different configuration methods and covers various applications and OSs to varying degrees. But what really plagued us was the comparison method: How do you evaluate hundreds of vulnerabilities--sometimes close to a thousand pages of text--across 11 products?

Because many of the products we tested reported thousands of vulnerabilities, we needed a common taxonomy to compare results. We chose CVE numbers (see cve. mitre.org) because they were the lowest common denominator between products, and for the most part, the effort is comprehensive. Unfortunately, using CVE addressed only part of the problem; the real challenge lay in parsing the reports (see our list of CVE numbers tested and how the products fared at www.nwc.com/1412/1412rd5.html).

Although Foundstone Enterprise and QualysGuard have well-designed reporting utilities, others, including Tenable Lightning (Nessus), Beyond Security's Automated Scanning Server and bv-Controls for Internet Security, have reports that are difficult to read and even more difficult to manipulate and re-sort. Complicating matters, we found that report content was often dissimilar. For example, nCircle's reports were so detailed we could review the entire attack decode to see how the vulnerability worked, while Vigilante's reports didn't even include remediation information; we had to follow a link to its Web site for further details.

Although all the products display a common vulnerability ID number (such as CVE or CERT) somewhere, they don't always list the place upfront. In fact, we noted several occasions where the vendor rolled several vulnerabilities into one heading and failed to list all the CVE numbers it represented. We wound up with a best attempt at digesting and comparing thousands of pages of reports. We ran all the scanners an exhausting number of times and spent weeks rebooting and resetting our systems and test bed. However, it is possible that a scanner may have flown under our radar as it knocked a service offline, inaccurately detected something it claimed to detect, or functioned irregularly in our environment. None of these factors would have radically change our results--and they are situations most organizations will face--but there is a margin of error. After all, even we need four hours of sleep once in a while.