Bill: Another critical component was a second server t hat received an order from the mainframe involved in the verification process.

Scott: The server's responsibility was to ensure that certain files requested by the user actually exist, before informing the mainframe that "all's well" so that the mainframe could respond to the user.

Bill: We had another path to check, from the mainframe to the server.

Scott: That server in turn checks a file server for the existence of the aforementioned files.

Bill: Whew. This was the third, and final, path in the verification process.

Scott: As it turned out, this third path involved a very busy portion of the main corporate network.

Bill: Because this was where the action was, we set out to analyze what transpired in this final step of the verification process.

Scott: Many of the file verifications were taking about 9,800 milliseconds (ms), or 9.8 seconds, to complete, as evidenced by the delta time between the request and delayed response packet.

Bill: We could almost set our stopwatch by these delays, as each one was very close to 9,800 ms every time. Occasionally, the delay was exactly twice or three times that.

Scott: Because the analyzer was on the same segment as the verification server, we proceeded to move our analyzer to the file server segment to make sure the packets were always reaching that segment.

Bill: It turns out that they were, and an immediate response from the server was seen as wellę

Scott: ...which meant that the response packets were not always making it back to the verification server's segment.

Bill: Checking the queue stats of the busiest router between the two servers brought us closer to the answer because we could see that packets were being dropped by the router.

Scott: We could always see the reply packet at the source segment connected to the router, but occasionally it never made it to the destination segment because the router was dropping the packets internally.

Bill: But why was there nearly a 10-second delay for the higher-layer protocols to recover?

Scott: Ah, the final piece of the puzzle. This application was using the SMB protocol running over NetBEUI (NetBIOS over LLC).

Bill: The recovery, as we could see in our analysis, was SMB resending the reply packet, since it was never acked. But why wait 10 seconds before resending?

Scott: It goes back to a legacy network where the customer set all servers from that point forward, to have a 10,000 ms (or 10 second) SMB retry timer, up from the default of 1,000 ms (or 1 second).

Bill: Changing the retry time to a more respectable 2,500 ms was the first change, but this was only a Band-Aid until the real problem could be fixed with the router, which involved many complex issues and multiple solutions.

Scott: We should note that the problem wasn't directly related to lack of router CPU resources or main router memory, but rather the way in which the router handled packets through cache memory.

Bill: Rather than optimize or upgrade the e xisting router, the "cache" that our customer chose to spend was to purchase another router to help off-load the busy one.

Scott: We can't advocate this solution in all situations, but in the heat of the moment, it was probably a reasonable thing to do.

Bill: Hey! Don't forget to send us your scariest networking horror stories.

Scott: We'll print our favorites again this year in our annual "Halloween Horror Stories" column in the October 1 issue.

Bill and Scott can be reached at otw@pmg.com. Portions of trace files from selected columns are available via Pine Mountain Group's Home Page (http://www.pmg.com).