ON THE WIRE

A Broadcast Storm Becomes A Thunderstorm

by Bill Alderson and J. Scott Haugdahl

Recently, during one of our training sessions, we encountered a major broadcast storm while analyzing a network in the customer's training center. As this was our first course with this customer, their senior personnel were in attendance to evaluate the course. All those responsible for router configuration were watching as we observed the network momentarily melt down at regular intervals.

As is often the case during a training course, real problems are the best training exercises, which in this instance, happened to be an IPX-related broadcast storm. Worse, this real problem also affected our customer's production network.

Scott: Being a Midwesterner, this problem reminded me of those broadcasts we receive on TV during the summer: "We interrupt this program to bring you a severe weather update. Thunderstorms have been observed in the area. We urge you to..."

Bill: But this time, the severe weather was observed on our customer's network! We believed there was a problem when we observed thousands of broadcasts in a short period of time. Looking closer, we found they were NetWare IPX broadcasts coming from routers and servers. Novell IPX servers are by default also routers, in that they are involved in the use of Routing Information Protocol (RIP) and use Service Advertising Protocol (SAP) processes even if only attached to one network segment.

Scott: More specifically, we noticed both IPX RIP broadc asts and IPX SAP packets coming from all the routers and servers.

Bill: Each packet was found to contain bad news about a service (SAP) or network (RIP) that was no longer reachable.

Scott: Wow, that is bad news!

Bill: The packets that appeared to be the beginning of the storm were from routers announcing that they had lost contact with hundreds of networks and SAP services.

Scott: The result of just one lost network or service generates a packet from the router that is broadcast to all stations. That packet is heard by other routers and servers, thereby informing them of the loss. They, too, announce to all stations with a broadcast packet that they can no longer access the network/service.

Bill: Imagine if you have two routers and five servers--since only one unreachable network/service is sent in an IPX SAP packet, each lost network/service will now be broadcast seven times.

Scott: Now imagine if you have 500 networks/services lost. Seven times 500 is 3,500 broadcasts as fast as the routers and servers can send them.

Bill: Talk about turning your basic cirrus cloud into a cumulonimbus.

Scott: OK, now that we identified the source of the broadcast storms, why were they occurring? We thought perhaps some interval analysis on how often these storms occurred would give us a clue.

Bill: So we picked one router to filter on and looked for the first packet in each broadcast storm and found that it was always the same network/service that was lost during the prior storm. By comparing the timing of the start of the storms, we found that there was indeed a consistent timing of their occurrence (see the Lost Networks/Services figure). The router was advertising that the server "CORPORATE1" was no longer reachable.

Scott: Interestingly, the router was broadcasting these lost networks close to every five minutes, as indicated by the frame delta times.

Bill: Then we filtered on the router advertising the same service and looked at good SAPs. In the Regular SAP figure above, we noted that the services were being advertised over two minutes, then were absent for 180 seconds, or three minutes.

Scott: From this analysis we noticed the pattern that led us to an idea. Perhaps the router had been set to supply RIP and SAP across a WAN link every five minutes.

Bill: And, we did indeed find this to be true, but only on one side of the WAN link, not at the router at our end. Therefore the sending router would update RIP and SAP every five minutes (a router option often used to save WAN bandwidth), while the receiving router was configured to expect updated RIP and SAP every minute.

Scott: So if updates weren't seen in four minutes, the router assumed the networks/services were dead, and then let everyone know the bad news before purging the information from its tables, thus causing the broadcast storm.

Bill: Right. When the receiving router was set to five-minute RIP and SAP updates, the broadcast storms went away, and SAP broadcasts were then observed on a regular 60-second basis as indicated by the packets following the ellipses in the figure.

Scott: The lesson learned was that increasing the RIP/SAP interval on a WAN router to reduce the number of broadcasts across the link may instead lead to a very undesirable side effect, namely triggering a broadcast thunderstorm.

Bill and Scott are principals of Pine Mountain Group. They can be reached at otw@pmg.com.

September 15, 1995