Picking a clear winner was difficult, because there really isn't a one-size-fits-all scenario here. Still, for sheer reporting power, and because we had the fewest complaints about it, we gave our Editor's Choice award to netForensics. That said, while these products may be based on similar models, some of their design goals are distinctly different. For example, GuardedNet's neuSecure has a Web-based interface designed for the hands-on operator. The real-time console displays top alerts based on source, destination and event type, and the submenus provide querying tools for further analysis. NeuSecure even has a built-in trouble-ticket system, which falls between network-operations needs and security-investigation needs.

What Do Readers Think? Check out our e-poll results on intrusion detection software.

On the other hand, products like netForensics are limited in real-time monitoring but provide a dizzying array of reporting tools for dicing and slicing data. Intellitactics takes yet another approach: Its Network Security Manager (NSM) has unique visualization and asset classification tools to let the less-technical operators distinguish between events.

If you're looking for an excellent data-mining and trending tool, netForensics is good. To empower the less-technical operators, NSM is a clear choice. NeuSecure could be a great tool for smaller MSSPs (managed security service providers). Finally, if you're already an IBM Tivoli or Enterasys Dragon customer, building on your current deployment may be more cost-effective, depending on your needs.

Our ideal solution would take the base of the netForensics product, add the visualization elements of NSM, use the more useful components of neuSecure, and combine it with e-Security's Windows-based rules creation engine.

Seas of Despair

Our goal was simple: Deploy the SIM solutions, integrate an assortment of live security devices, spend some time customizing the products and evaluate their usefulness to a security practitioner. The implementation of these ideas, however, was not simple.

Our journey began when the first products arrived, which set in motion the building out of our testing environment. This was no small endeavor, as the testing required us to set aside dozens of systems and devices. The device list of our Neohapsis partner lab, in Chicago, is fairly substantial: firewalls from Cisco Systems, NetScreen Technologies and Nokia; IDSs from Cisco, Enterasys and Internet Security Systems (ISS); not to mention routers, switches, dozens of operating systems and applications ranging from Apache to X Windows. We even commandeered a number of firewalls, IDSs and other security devices in our production network, in an effort to use as much real-world data as possible.

All the solutions we tested follow a similar design model: Security devices feed into a data collector or aggregator; aggregators feed into a database-driven back end; the consoles or interfaces attach to the database back end for monitoring and querying (see "Components of a SIM Solution").

Our first task was to assemble a plan for feeding the data output streams of our security devices into the aggregator components of the SIM solutions. This is tricky, as different devices output varying types of data using an assortment of transport mechanisms. For example, Check Point Software Technologies firewalls are typically configured to output their log information using OPSEC or SNMP. By comparison, Cisco Secure IDS devices default to using the proprietary Cisco POP, but they can also be configured to use SNMP as their transport mechanism. Although SNMP and syslog are frequently the lowest common denominators for interdevice communication, they are also the least desirable, from a security perspective, because they rely on UDP (User Datagram Protocol). UDP is a lot easier to spoof than TCP and is also considered less reliable. We didn't spend much time worrying about this problem, but hard-core security administrators will notice it.

As if OPSEC, SNMP, syslog and proprietary protocols weren't enough options, we also had to deal with custom agents. Some vendors bundle agents for harvesting data on devices and sending it back to aggregation points. Although these custom agents frequently offer a more secure solution than their UDP-based counterparts, they require the additional overhead of installation on a per-device basis. This can be a burden for larger environments. Placing an Intellitactics data-gathering agent on a few IDS devices is one thing, but deploying it on a few hundred is quite another. In the end, we opted to use syslog and custom agents for most of our data-transport needs. While we look forward to the day secure syslog comes to fruition, the state of secure log data transport is still a bit away from where it should be.

The second step on our task list was to install the SIM solutions. Each solution comprises various software components, each of which has a different installation process. Often, this includes database back ends, such as IBM DB2, MySQL and Oracle; various middle-tier aggregation components that run on multiple platforms, and front-end consoles and interfaces that are Web-, Java-, Microsoft Windows- or X Windows-based. A SIM solution can easily be spread across four or more systems, each requiring a separate installation.

The lesson we learned here is that paying for professional services to handle the installations is a must. During our installation marathon we were forced to deal with JDBC and ODBC driver configurations, shell scripts, service packs, hot fixes, database table and permission settings, and an assortment of other inglorious tasks. Sure, we learned a ton while spending days getting a single component functioning. We also discovered all sorts of odd compatibility nuances, like that Oracle 9.0.1 runs only on Red Hat Linux 7.1, not on SuSE Linux, the distribution Oracle's documentation specifies.

But unless you're suicidal or have a ton of time to spend interpreting misleading documentation, the time and frustration saved by getting help are probably well worth the money spent. These products are young and raw, and with the exception of netForensics, their documentation is pretty horrid.

The last leg of our deployment process involved tying all the pieces together: getting the agents to communicate with the aggregators, the aggregators to properly feed the back ends, and the back ends to accept and store the endpoints' data correctly. This phase also presented us with a number of unanticipated challenges. While one might expect this process to entail simple configuration changes at the SIM console and the endpoint device, rarely was this the case. Instead, we were often sent down dark paths of undocumented procedures, calling tech support and performing file tweaks.

For example, to get our PIX 535 to report events to the IBM Tivoli Enterprise Console (TEC), we had Tivoli support walk us through the steps:

>> Install a PIX component from the Risk Manager CD.

>> Edit a few configuration files.

>> Append a PIX format file to another file, and execute some command-line scripts to regenerate a new file on the aggregation machine.

>> Copy that file back to a different configuration directory.

Wait. There's more. Our next steps were as follows:

>> Install on Windows NT a syslog server downloaded from Cisco.

>> Stop and restart a few services.

>> Install a PIX event handler to tie together the syslog server and the TEC console.

>> Configure the PIX to start sending log info to the aggregation point.

A polished product would have an installation wizard to perform most, if not all, of these tasks. Unfortunately, polished these products are not. In this scenario, we had to touch every part of the solution: the endpoint device, the data-aggregation components, and the back-end system.

Not all the device integration efforts were as difficult as our Tivoli Risk Manager undertakings; getting the PIX 535 to talk to netForensics, for example, was a snap. But many weren't far off, either. If we took one thing from this experience, it was the knowledge that a pilot would have let us make more accurate time calculations. If your organization goes down the SIM path, make sure you supply ample time to pilot the solution properly. You need to calculate for time estimates on building the aggregation systems and back-end systems, then perform estimates on how many devices you'll have to reconfigure and whether you'll need to install any agent software.

In short, launch a pilot. You'll be kicking yourself if you don't.

The pain and overhead of the installation process, however, did not outweigh the benefits of having the systems deployed. Once they're installed, having all event and log data streaming into an integrated system for analysis is incredibly valuable. We enjoyed the real-time aspects of neuSecure and NSM, as their visual aids helped us determine which events we wanted to spend time investigating. The return in time, energy and efficiency is obvious after just a few weeks of use. However, these systems have a way to go and are obviously targeted at the large enterprise customer -- with prices to match.