Confidence Isn't Enough

The importance of checking on testing procedures came to light during my recent talk with a vendor regarding Network Computing's testing methodologies for a review. As I often do, I turned the tables and asked the representative about his company's own testing methodologies, mainly because this information can sometimes help us determine what to test in the Network Computing labs. This time I was in for a surprise. The representative revealed that his company doesn't have any formal testing methodologies in place; the developers do a few ad-hoc tests and then send the software over the fence.

Red lights immediately started flashing. Wait a minute, people pay good money for this software yet the vendor doesn't invest the resources to make sure it works properly? And this particular vendor's software is quite complex and enterprise-oriented. The company has great confidence in its developers, but is that enough?

I wouldn't be writing this column if I thought so. If you invest heavily in any software or hardware, it pays to investigate your supplier's testing processes and methodologies. In the long term, you could avoid problems with security, stability and management.

Beyond the Bugs

You're probably already asking your software vendors how you should report and deal with bugs, and how hard it is to implement solutions or patches. These and other questions about service and support are important because all software has bugs, no product is unbreakable, and you'll want to upgrade when your vendors release new versions of their products. You need a good methodology to deal with problems that certainly will arise.

However, let's step back and rethink the scope of these questions. You're asking vendors about how to fix their broken software. Great. Now how about asking them how they limit the number of programming errors in the first place? Remember to question them about their QA and performance testing procedures.

You may also want to ask vendors how their testing staffs are structured. Are they part of the product development teams, or are they separate? Is testing built into product development, or is it done afterward, if at all? Is software quality part of the development process? What tools and methodologies do the vendors use to evaluate their products' quality?

Getting the answers to these questions will save you time in the long run and will help the vendors perceive the importance of ensuring quality. However, even if a vendor's testing procedures meet high standards, don't use this as an excuse to avoid testing new products in your own environment before rolling them out to the masses.

Mike Lee mlee@nwc.com