We also added a twist to the typical Network Computing RFP by testing the performance of the 10,000-user mail systems proposed for MediaMakers. These tests took place in our Real-World Labs® at Syracuse University.

What Do Readers Think? Check out our e-poll on large-scale e-mail systems!

All six vendors responded to both of our RFPs. Critical Path, iPlanet, Mirapoint, Novell, Rockliffe and Stalker Software submitted bids ranging from $26,000 to $428,000 for the MediaMakers bid and $474,000 to $4.4 million for the NetMagik request. We commend each of these companies for the amount of time and resources it committed to this project. With the exception of Mirapoint, which submitted a turnkey hardware- and software-based solution, these vendors are software shops. We insisted they specify the hardware as well as the software for the bids, obtain and ship the hardware, send representatives to the Syracuse labs for setup and tuning, and subject the solutions to two days of prodding and probing. (See MediaMakers' and NetMagik's requirements for an overview of the requirements for each mail system.)

Unlike highly functional (and complex) messaging servers--Lotus Notes, Microsoft Exchange and Novell GroupWise--the LSMS (large-scale messaging system) is only for e-mail and relies on standards-based transport and access protocols throughout. These systems are designed for massively scalable throughput, high reliability and delegated administration.

To get an invitation to participate in the RFP, a vendor needed to have an established presence in the ISP arena as well as a corporate offering. Software.com, one of the leaders in the ISP arena, doesn't have a corporate offering, so that company wasn't invited to participate.

Based on our testing, each of these 10,000-user systems generated enough throughput to handle MediaMakers' e-mail requirement, with plenty of room for growth. Even the slowest system in the test, Novell Internet Messaging System (NIMS), processed nearly 136,000 messages during the five hours our benchmark ran. This was slow only by comparison. If we had stretched the five hours to 24, the system would have processed 653,846 messages. Keep in mind that we tested an IMAP mail system, not a less compute-intensive POP system.

Uptime All the Time?

The MediaMakers RFP required a 99.9 percent uptime guarantee. Three-nine uptime permits about 10 minutes of downtime a week, or nearly nine hours a year. In our estimation, requiring three-nine uptime doesn't automatically relegate the buyer to costly clustering/high-availability solutions. Three-nine uptime can be met by stocking hardware spares, implementing RAID-based storage and having an established, written procedure in place for swapping spare hardware for failed hardware.

Some vendors--namely, iPlanet, Novell and Rockliffe--offered cluster-based solutions that wouldn't require operator intervention if a server failed. Critical Path specified a hot-standby solution with a second mail server; it was connected via NFS (Network File System) to the storage array, ready to step in via operator intervention if the first server failed. Mirapoint and Stalker Software specified a cold-spare solution that would also require operator intervention in the event of a failure. We believe that any of these solutions would meet the three-nine requirement of the RFP.

The NetMagik RFP required a 99.99 percent uptime guarantee. Four-nine uptime permits one minute of downtime per week, or 52 minutes a year. The NetMagik uptime requirement could be met only by implementing a clustering/high-availability solution. All our vendors specified a multitiered, multiserver solution to achieve four-nine uptime as well as scalability.

And the Winner Is...

We scrutinized each bid to make sure it met the uptime and functional requirements specified in the RFP. Functionality, manageability, support policies and costs, price, and performance were central factors in determining which vendors made the shortlist as well as in determining the winner. Use these proposals as a guide to create your own shortlist: Examining each vendor's solution will give you insight into how to implement a large-scale e-mail solution of your own.

In the end, both MediaMakers' and NetMagik's bids went to Stalker Software's e-mail solution, which featured CommuniGate Pro. CommuniGate Pro wasn't the fastest mail server in the roundup (Critical Path's InScribe Messaging Server beat CommuniGate Pro's overall speed), but it put up the highest price/

performance numbers as well as the lowest error rate in the group. CommuniGate Pro worked flawlessly on the least expensive hardware during the performance testing and nailed the functionality requirements.

Rockliffe made the shortlist for MediaMakers, and Rockliffe and Critical Path made the shortlist for the NetMagik RFP. Assuming that Microsoft Windows 2000 proves its mettle during the next year, we think Rockliffe's entry will be worth a very close look even in the ISP community. Based on performance alone, Critical Path's solution should be on the shortlist--but only if the company brings its software and support costs in line with the other vendors in this RFP.

Each of these vendors would be given further consideration in a purchasing situation. You'll find summaries and evaluations of these six vendors' proposals below.