It's A Question Of Latency


				HOT PICKS • U.S. IT Salary Survey 2008 • Rolling Review: SOA Appliances • XKL Brings New Life To Dark Fiber

IMMERSE YOURSELF:

It's A Question Of Latency

Q : We recently added a frame relay connection between our headquarters and a remote site to allow people on the Ethernet remote LAN to transfer files to and from the corporate LAN. Our service provider set up routers to route our NetWare traffic across the frame relay "cloud." Since each site had its own "home" servers, we didn't think we needed a large amount of bandwidth between sites. We started with a committed information rate (CIR) of 56 Kbps at the remote site with a 1-Mbps CIR at the corporate LAN to accommodate future remote sites. We upped our remote site to 256 Kbps to lessen the time required to transfer larger files. We failed to see a four-fold reduction in file transfer time. Our file transfers are only averaging about 8 KB per second. We're using the maximum size Ethernet packets possible. We hesitate to buy a higher CIR and are reluctant to install a more expensive alternative. Help!

Scott : Say, Bill, haven't we seen this before?

Bill : Can you spell L-A-T-E-N-C-Y?

Scott : Ah, yes, the old latencyproblem!

Bill : This is an infrastructure of multiple 16-Mbps Token-Rings (see figure, next page). In the above scenario, we have several possible points of latency, or delay. To name but a few, the workstation may not be turning its requests for more file data around fast enough, the workstation's LAN may be congested, there are switching and distance propagation delays in the frame relay "cloud" in question, the CIR is too low, there are additional routers after the far side of the frame relay link, the remote Token-Ring infrastructure is congested or the remote file server is not quickly responding to the read requests.

Scott : We start by traveling to the remote site and hooking up the ol' protocol analyzer. We immediately see that the LAN is lightly loaded and that the workstation is capable of producing rapid read requests to the remote server, taking less than one millisecond to request more data after processing a piece of a file.

Bill : So things look good at the remote site. If the least common denominator in the link is 256 Kbps, then 32 KB per second would be our maximum theoretical throughput. Our client noted that the end-to-end file transfer rate was only 8 KB per second.

Scott : We confirmed that the remote server is returning 1,494-byte packets, near the Ethernet maximum of 1,518 bytes (including CRC). Of the 1,494 bytes, 1,436 bytes are file data after discarding the protocol headers, so we have a high percentage of data to overhead. However, our analysis shows that the 1,494-byte packet is not received from the remote server until 180 milliseconds after the station's read request!

Bill : Part of that latency is due to the fact that at 256 Kbps, 45.5 ms is required to transfer 1,492 bytes of data over the WAN portion of the link. (For comparison sake, a 10-Mbps Ethernet can deliver the same amount of data in 1.2 ms.) Since our 180 ms latency was very consistent over several runs, we didn't suspect remote LAN congestion or server congestion. So, even if we increased the WAN link speed, the best we could hope for is 120 ms of latency.

Scott : Since every read/request pair is throttled by latency, why not send a "stream" of packets for each read request?

Bill : Exactly! So we need to get rid of the IPX "ping-pong" protocol, where there's a request packet for every reply packet, and implement the NetWare packet burst protocol, where multiple packets are returned for each read request. This is similar in concept to the "sliding window" protocol built into TCP.

Scott : By using the burst protocol, not only can we reduce the effect of latency, we also reduce the number of pesky little request packets by a factor of 10 or more!

Bill : The remote server should have no problem returning a "burst" of packets to keep the 256-Kbps pipe full. Since the intervening routers shouldn't have a problem handling data at this rate either (especially since the routers are dedicated local-to-frame relay routers), the only remaining latency is when the workstation's 64-byte request packet travels across the frame relay to the server.

Scott : We loaded the latest NetWare Virtual Loadable Modules into our workstation and made sure our remote server was equipped for packet burst.

Bill : We ran the file transfer as before, captured the data, reanalyzed...

Scott : And voila! The gains were spectacular. Our file transfer throughput increased to 28 KB per second or 224 Kbps, a 350 percent gain over the original 56 Kbps, as indicated in the figure. During a continuous packet burst (with no intervening workstation request packet), our large packets (1,484 bytes this time, slightly less than the old number of 1,492 bytes, due to a small amount of additional packet burst protocol overhead) were being delivered to the workstation at 254 Kbps, or near our theoretical 256 Kbps maximum.

Bill : Thus by switching to the NetWare packet burst protocol, the IPX throughput should approach that of the least common denominator link speed between stations.

Bill and Scott are principals of Pine Mountain Group, and spend their time troubleshooting large networks, training end-users in protocol analysis, and developing tools to allow users to make better use of their protocol analyzers. They can be reached at otw@pmg.com.