NFS Woes Creating Work Nightmare

Bill: Ah, yes, the old "low file transfer throughput on the local LAN" problem--no bridges, no switches, no routers, no WANs, no throughput!

Scott: But why? NFS is another one of those "multiblock" protocols, like NetWare packet burst and TCP sliding window, where a client can send one request and receive multiple data packets in return from a server. We normally expect to see fairly high throughput with NFS, especially on a local LAN.

Bill: Since NFS is a client/server application built using Sun's Remote Procedure Call (RPC), NFS will access a file by sending an RPC request to a server.

Scott: Our customer's NFS is sending its RPC requests via User Datagram Protocol (UDP), which every i mplementation of NFS supports. Since NFS will request a read (or write) for a block of data larger than the underlying network's maximum transmission unit (MTU), the resultant data is returned in several IP fragments (see figure "NFS Using UDP/IP"). Since UDP is a connectionless protocol, there must be a mechanism for time-out and recovery in the event of a lost fragment.

Bill: All it takes is one lost IP fragment during a read to cause NFS to reread the entire block.

Scott: Some newer versions of NFS can use TCP for error-prone environments. Then, TCP will retransmit a lost segment, relieving NFS of the burden and reducing the number of packets retransmitted.

Bill: We first performed a baseline of the current throughput. It never hurts to do a little baselining. Then after resolving the problem, the network analyst doesn't have to rely on an end user saying, "Uh, yeah, like, I think it's faster."

Scott: And there's some concrete data to prove to management that the investment in that little ol' protocol analyzer really paid off.

Bill: Our baseline showed that the NFS throughput was approximately 80 KB per second. Trace analysis revealed that the workstation's NFS read block size was set to 1,024 bytes, as indicated by the client's read requests. The data was subsequently returned by the server in a single NFS reply packet.

Scott: This pattern repeated itself with no indication of lost packets. The response time for each read request was a bit slow at 10 milliseconds, but that didn't account for the poor throughput. Since NFS can request a much larger block size, we set the read block size to 8 KB and reanalyzed.

Bill: This time, the data was returned in an NFS reply packet plus five IP fragments, making for a more optimal NFS scenario.

Scott: But our expert system indicated several NFS retransmissions, which we found was the client asking for an identical block of data a second or third time in a row.

Bill: Since no packets were lost or had CRC errors, we knew that the NFS client was unable to process properly the 8-KB "burst" of packets being returned by the server for every read request. The NFS retry time-out was close to one second, meaning that every time a fragment was "lost" by the workstation, one second plus the time to transfer another 8 KB of data was added to the file transfer time, making for several seconds of delay.

Scott: The average throughput actually worsened to under 10 KB per second.

Bill: So we cut the block size in half to 4 KB and reanalyzed. We saw that the workstation was still dropping packets.

Scott: We knew there had to be a happy medium, so we set the NFS read block size to 2 KB and once again, reanalyzed.

Bill: Now there were no NFS retransmissions by the workstation; the throughput increased to 160 KB per second.

Scott: So bigger block size is not always better?

Bill: In this case, yes, due to poor performance of the client NFS. We have seen a wide variance in performance of different vendor implementations of Windows-based NFS running on the same platform such as same processor, network adapter and drivers. Normally, a Windows-based NFS implementation on a PC should be able to handle an 8-KB block size with no problem.

Scott: So, our customer is looking for an upgraded or new Windows-based NFS implementation, which will be tested and baselined before deployment!

Bill and Scott are principals of Pine Mountain Group, and spend their time troubleshooting and optimizing large networks, training end-users in protocol analysis, and developing tools to allow users to make better use of their protocol analyzers.

Portions of the actual trace files that were discussed in this column are available in the Network Computing On-the-Wire CompuServe forum (GO OTW). These files can be loaded into your own analyzer. This month's file is NFS.ZIP and contains NFS trace files for various protocol analyzers. These files are copyrighted by Pine Mountain Group, and may not be used for any commercial purposes. Additional files may be made available from time-to-time, at our discretion. Please e-mail us at otw@pmg.com. Or, e-mail Bill or e-mail Scott directly.