BILL ALDERSON AND J. SCOTT HAUGDAHL

On The Wire

The 'Lock Next' Monster

Scott : Talk about a network full of saturated fat!

Bill : Totally. Our client was thinking that given the number of users, routers, routes to servers and underlying network technology (predominately 4-Mbps and 16-Mbps token rings), perhaps the network had to be upgraded or rearchitected. But the client wisely decided to perform forensic network analysis before taking such drastic and costly action.

Scott : So we start out by analyzing packets captured during a file open/load/close sequence between a DOS/Windows workstation and the server (one of the most basic analysis exercises). We discover that workstations were generating less than optimal packet exchanges to and from the file server operating IBM LAN Server 3.0.

Bill : LAN Server uses the Server Message Block (SMB) protocol on top of NetBIOS and LLC2 to carry out its basic file operations. We observed that acknowledgments were occurring at both the LLC and NetBIOS Layers for every frame. We check the version of LAN Support Program (LSP) in the w orkstation only to discover that it's an old release and...

Scott : We then decide to upgrade to the latest version rather than trying to optimize the old stuff. Once again, we perform the same file open/load/close test with the latest version of LAN Support Program with encouraging results: The newer version performs much better "out of the box," as the figures indicate. The kilobytes of bandwidth consumed to perform this file transfer drop a bit, but far more significant is the large drop in the number of frames.

Bill : Right. Not only that, but dropping the frame count directly equates to router relief as the router's CPU doesn't have to process as many frames by looking up their headers, making routing (or bridging) decisions and so on. But this simple optimization doesn't satisfy us so...

Scott : We look at the packet exchanges again and realize that we can do even better!

Bill : Once again we study the packet exchanges and realize that LSP and LAN Server are not taking full advantage of the protocol's capabilities.

Scott : One of these capabilities of LLC is a "sliding window" or "burst" capability, much like NetWare's burst protocol. By setting the workstation and server parameters properly, we were able to reduce the number of acknowledgments by another 40 percent. This reduced the kilobytes a tad more, but reduced the frame counts significantly.

Bill : The net result is that we reduced the number of frames required for this particular application from more than 4,000 frames to less than 1,000 frames.

Scott : Monitoring frame counts across routers for several hours after this optimization revealed an average overall reduction of better than 2:1.

Bill : So we got the frame counts down crossing routers, but it still didn't solve the login problem!

Scott : We shift our analysis from observing a single workstation/server pair to one of observing mu ltiple simultaneous logons. The login process took nine seconds to complete with one user logging in, and when we tried three users simultaneously it increased to more than 20 seconds. Since it shouldn't take double the time for multiple stations, we knew something was amiss.

Bill : Sure enough, the analysis revealed the locking of several common batch files (located on the server) during the login process. Nothing unusual there.

Scott : Nope, except that the batch files are opened with a sharable attribute request, which in LAN Server, locks the file from other users until it is finished. The problem was compounded by a master server-based batch file that in turn calls other batch files, which effectively blocks access to all batch files until the calling batch file finishes!

Bill : Wow. This creates a random serial access of the server-based batch files, meaning that that thousands of users wind up waiting to grab the "master" batch file!

Scott : This is a classic category of problems that we have coined the "Lock Next" Monster.

Bill : Ol' Nessie is spotted again.

Scott : Only this time it's in a network instead of a remote lake in northern Scotland.

Bill : We can get rid of the monster by moving the batch files back to the user's login directory where it belongs and where it was originally located by LAN Server.

Scott : Doing so resulted in the unbelievable, totally amazing, absolutely stunning reduction in login time from 20 minutes down to...

Bill : 20 seconds!!!

Scott : And this sets a new record for our best time reduction optimization to date.

Bill : Not to mention a significant 50 percent reduction in saturated fat by lowering our cholesterol (frame) count. The routers' CPU peak utilization also dropped to 67 percent.

Scott : Hey, Bill, I heard a rumor recently that the last living person cl aiming to have seen the Loch Ness Monster recently died, and confessed on his death bed that the sighting was a hoax.

Bill : Yeah, but the Lock Next Monster has, through forensic network analysis, been proven to be alive and well, lurking on a network near you.

Bill and Scott are principals of the Pine Mountain Group and spend their timetrouble-shooting large networks, training end users in protocol analysis and developing tools to allow users to make better use of their protocol analyzers. They can be reached at otw@pmg.com.