We broke costs out into monthly recurring charges and one-time groupings. There were differences in what providers consider standard offerings, but we tried to normalize the menu, using specifics stipulated in the RFI as our baseline. So, for example, performance baselines and capacity planning were included by iNOC and PerformanceIT, but quoted as separate MRCs by HCL, while NetProactive required a separate bid to provide these services. Because TacDoh's in-house networking department provides, and our RFI asked for, capacity planning and historical baselines, we included the separate cost in HCL's total MRC (see the full RFI and responses).
We also cared about guarantee of services. Every branch was to have 99.9 percent uptime; every warehouse, 99.99 percent uptime. Transactions from the branches were to be completed within 5 seconds, with no more than 2 seconds using network time. We knew that the care and feeding of the PoS (point of sale) systems, back-end application and database servers might be beyond the scope of the MSPs, but in the real world the network group is the first line of defense--and the arbitrators between IT factions--so we had to ask. We were surprised again: All but NetProactive stepped up and took on the SLAs (service-level agreements) without hesitation.
The RFI also spelled out some unusual problems--NTF (No Trouble Found), act of God and erroneous WAN provider disconnect--to get a sense of how each vendor handled situations where the root cause was outside the official responsibility of the network-management group but nonetheless a common and time-consuming chore. During the process of checking out a case of WAN trouble, service providers frequently clear the problem, thus masking, not resolving, the issue. It may appear to the service provider as though nothing was wrong even though the problem is likely to recur, requiring tenacious interincident tracking. Acts of God--say, power outages due to lightning striking power transformers--can cause dispatches, which, while technically not the fault of the network-management group, is clearly its mess to clean up.
Finally, WAN circuits are occasionally disconnected by mistake. It's up to the NSM group to jump up and down to get the service restored. We considered it vital to pin down exactly how the MSPs would react in these cases--especially considering that TacDoh might not have any network-savvy people left on staff once the outsourcing went into effect.
Our final two evaluation categories, operations and reporting, are much more important in real life than in the land of TacDoh. Operations is our catch-all term for a wide range of service differentiators, from strict change control to the cafeteria and recreation available to the NOC staff. The common denominator: Acing these criteria indicates stable, reliable and mature operations, lending credence to the idea of trusting them with the deep-fry kingdom.
Reporting, of course, is the conduit by which network and application health and activity are communicated. Each MSP cited complete and comparable reporting engines, with what appeared to be some differences. We say appeared to be because we didn't test them. Nor did we visit the network operations center, taste the lunchroom special or play any late-night ping-pong. So while we recognize the importance of these categories, we minimized their overall weight.
We broke reporting scores into two basic areas: The breadth and depth of the reporting, including how flexible it was, and report publication. As mentioned above, all the vendors promised link, availability and even application statistics on a store-by-store basis. (Read more on how we graded.)
Our Editor's Choice is PerformanceIT, which provided excellent cost, service management and reporting. Its RFI response addressed all of TacDoh's needs. A close second, iNOC addressed TacDoh's requirements, but was a bit less aggressive in terms of cost savings and didn't give quite the same level of service assurance as PerformanceIT. HCL did well overall, providing granular detail about its operations and reporting, but it didn't compete as aggressively on the cost side as did iNOC and PerformanceIT. Also, HCL ignored TacDoh's special requests (see "Specific Requirements"). NetProactive Services offered what appeared to be above-average reporting and good service-level and operation assurances, but at a price that didn't compete.
The pricing we received was based on TacDoh's specific requirements, and none of the vendors was aware of rivals' responses. In real life we could have pitted them against one another and perhaps gotten them to shave off a few bucks.
REPORTS
Analyize In-Line NAC strategies and products.
ANALYTICS Plan and design your enterprise blade server deployments
InformationWeek U.S. IT Salary Survey 2008
Salaries for business technology professionals are falling. Here's what you need to know in order to make good hiring decisions and personal career choices. Download Today
REPORTS
ANALYTICS
Follow 
