· Frame relay throughput and latency

· ATM throughput

· T1 interface bit error rates

· Voice quality and demonstrated bandwidth optimization

· Trunk failover delay

We asked the vendors to create a small network across two switches, using a single DS-3 trunk for all except the failover test, which involved two DS-3s (see "WAN Switch Test Configuration" on page 62). We supplied varying loads of ATM and frame relay data traffic across multiple interfaces, several DS-0s of voice traffic and a test pattern over a T1 interface. All traffic measurements were taken concurrently.

ATM traffic descriptor tuning and vendor proprietary configurations can make a dramatic network performance difference, so we ran all tests (except trunk failover) six times--four times with common virtual circuit definitions used for all vendors (two runs with no added delay and two times with 300 ms of delay across the DS-3 trunk). Then we asked vendors to tune their switches in any way they wanted, introducing new hardware-software combinations and adjusting ATM para meters to better fit the traffic patterns observed in previous tests.

Frame Relay Test Measurements Frame relay was tested using two independent data paths, across separate Cisco Systems routers with T1 interfaces to the vendor's switch. Frame relay PVC parameters were set at maximum port speed; access rate, committed information rate (CIR) and committed burst all equal to 1.536 Mbps, with burst excess set to 0. Traffic was generated and analyzed using the Wandel and Goltermann DA-30 Internetwork Analyzer's RTBench Application. RTBench generates IP packets of various sizes, increasing the injection rate in 10 percent increments, up to our target maximum (1.5 Mbps, the T1 line rate). RTBench reports throughput at the maximum lossless level--the highest point at which 100 percent of submitted packets are received.

Latency calculations were also made at the maximum lossless throughput level. Note: All vendors passed 100 percent of 768-byte packe ts, making 768 bytes a good point for comparing delay.

ATM Test Measurements ATM throughput measurements were taken using the W&G DA-30 FDDI analyzer with the RTBench Application, sending traffic across a Cisco 7000 OC-3c interface. The target offered load maximum was set at 20 Mbps, which was the amount of traffic remaining on the DS-3 trunk when other traffic was present.

Circuit Quality Measurements To stress-test T1 quality, we created a single virtual circuit that passed across the DS-3 trunk and seven T1 interfaces per switch. On each end of the circuit we placed a TTC FireBerd 6000 Communications Analyzer. We used a DDS-5 test pattern and measured average bit error rates across the link.

Voice Quality and Demonstrated Bandwidth Optimization We subjectively analyzed voice quality across a Nortel Meridian PBX with two T1 connections, one on either side of the network being tested. We placed three concurrent calls over the link and listened for choppiness, echo and any other quality degradation. We also aske d vendors to introduce voice optimizations (compression, echo cancellation and silence suppression, for example) and to demonstrate bandwidth optimization via their network management stations. These tests were conducted under normal conditions, and with an additional 300 ms of delay across the DS3, simulating a high-latency link such as a satellite connection.

Trunk Failover Delay For the final performance test, vendors added a second DS-3 trunk, wired directly between the switches. With all traffic still running, we disabled the primary DS-3 and watched for our test equipment to report a resumption of end-to-end traffic.

Performance Highlights Overall, GDC turned in the least variation in performance, though it showed the highest frame relay latency and slowest trunk failover time. Nortel's voice performance was outstanding. Nortel's data performance was also good, but ATM averages were dragged down by two bad runs. IBM Corp. turned in average-to- mediocre scores.

Frame R elay Lossless Throughput GDC's frame relay throughput remained flat relative to our baseline, with low loss across all packet sizes (see "Frame Relay Throughput" on page 71). Nortel showed loss at 64-byte packet sizes, but did well otherwise. IBM turned in lackluster figures, as low as 15 percent off the baseline at 64-byte packets.

Frame Relay Latency Both IBM and Nortel had less delay than our baseline frame switch. GDC's latency was 2 ms to 4 ms higher than the baseline with larger packets (see "Frame Relay Latency" on page 64).

ATM Lossless Throughput GDC turned in the best scores in ATM throughput. IBM and Nortel scores were acceptable (see "ATM Throughput" on page 67). Nortel's average throughput dropped off at larger packet sizes because of a 100 percent packet loss condition that occurred in the last two runs of the test (ironically, these were the runs following vendor optimizations).

T1 Circuit Quality Tests All vendors turned in equivalent bit error ra tes--normally in the 10-9 or better range, with occasional dips into relatively high error rate territory--10-6 and 10-7.

Voice Quality Tests Our subjective assessment of Nortel's voice quality pegged it as outstanding, particularly given the very low bandwidth that it was consuming. Under high-delay conditions, we detected a slight echo over the IBM switches, but could not determine if this was due to mismatched time slots between the PBX and switch. GDC did not demonstrate echo cancellation, but otherwise quality was acceptable.

Trunk Failover Delay Both IBM and Nortel's trunk failover capabilities kicked in promptly, and end nodes resumed conversations within 30 seconds of a trunk failure. GDC took a bit longer--about 90 seconds, though with proper tuning it would be possible to reduce this to 70 seconds. The delay is attributable to the network's use of soft PVCs over switched virtual circuits, which require a little extra signaling time.