A Trip Down The ATM LANE by Kirk Kirkpatrick If you listened to the sales pitches of the ATM vendors at Interop in Las Vegas, you might believe that ATM is here and that everyone is using it to solve their bandwidth problems. However, when you ask these same vendors for examples of corporate networks using ATM, they always seem to point to a NASA wind tunnel site, a university multimedia lab, a Wall Street trading floor or some other equally out-of-this-world customer with either an unlimited budget or little accountability for the performance of the network. The reality is that while many companies are playing with early ATM products, few networks are larger than a dozen nodes, and almost all were built using first-generation ATM gear, which the vendors no longer actively push (make sure you don't buy one of these). The key question that must be answered before network managers can move their networks to ATM is: Have the vendors lear ned from these early architectures and adequately addressed the issues of scalability, reliability, manageability and cost in their second-generation offerings? Last year, Andersen Consulting, a large professional services organization, decided to relocate its worldwide technology competency group to a new campus in suburban Chicago. One purpose of the Andersen Consulting Technology Park (ACTP) is to provide a place where we can showcase and build our technology expertise. Partially to satisfy our users' bandwidth requirements and partially to see what we could learn from the experience, we decided to build the new network with ATM. This 300,000-square-foot facility is home to 1,600 users (rapidly growing to 2,000), two data centers, 19 network closets and 14 major technology testing and demonstration centers. The network has 35 VLANs and 2,500 LAN connections. Servers run the gamut. There are 150 Lotus Notes and NetWare servers. There are 185 NT, OS/2, Unix, VAX, AS/400 and other hosts and servers. We selected Bay Networks as our partner and deployed an ATM backbone solution using its first-generation ATM gear. In September 1995, we installed three Bay System 5000AH switches in our main computer room, connected via multimode fiber to Bay Ethercells (ATM-to-Ethernet edge devices) in each wiring closet. We used a combination of ATM Forum LAN Emulation 1.0 (LANE) and port-switching shared Ethernet cards in the System 5000 hubs to virtualize the network at layer 2. Note that in this issue's cover story on Ethernet-to-ATM edge switches, we state that Bay is not shipping a LANE 1.0-compliant edge switch. At the time of this testing, however, Bay was pushing its 5000AH product, which was compliant. Bay is working on a next-generation switch. This enabled us to use Ethernet segmentation to increase the bandwidth available to the average user as well as employ network management software to assign any LAN port anywhere in the facility to any one of 35 vi rtual LANs (VLANs). Below we detail some lessons learned while building ATM networks. Know what you need. You will need to procure four components to build an ATM LANE network: ATM switches, ATM-to-LAN edge devices, an ATM router and ATM NICs. Also, you will need management software and should be aware that one of your components will also have to run the LANE services--LAN Emulation Service (LES), Broadcast Unknown Server (BUS) and LAN Emulation Clients (LECS). In our case, we used the Bay 5000AH switch, Bay Ethercells and the Bay 5000AH ATM router card (the latter was not initially implemented). No ATM NICs were implemented because of their lack of support for NetWare servers at implementation time. LANE services ran on a Unix-based switch controller card. Understand the standards. Keeping track of rapidly evolving ATM standards is a frustrating but necessary task. Many standards are incomplete and require vendor-specific extensions to deliver required network function and resiliency. It is critical that network managers contemplating early ATM deployments understand the features and limitations of key standards so that they can monitor vendor compliance and recognize and evaluate proprietary extensions. While we found that a basic understanding of several ATM standards, including UNI, Signaling, PNNI Phase 0 and Congestion Control was useful, a detailed understanding of the inner workings of LANE 1.0 was critical to our evaluation and to the design and troubleshooting of our network. Stick with a single vendor. While many ATM standards such as UNI 3.1 have reached a level where, although they continue to evolve (for example, UNI 4.0), they are usable in their present form. Others either have not emerged from the standards bakery (MPOA) or have emerged only partially baked (LANE and PNNI), requiring further cooking in a vendor's proprietary oven (or in the heat of your network) before they can be consumed. Time-to-market pressures wi ll continue to drive prestandard implementations and proprietary extensions. As a result, interoperability will be problematic. The early adopter is advised to stick with a single vendor for most components. Standards don't mean interoperability. Most standards leave some of the implementation details up to the vendor. Just because two vendors claim compliance with a standard does not mean that their products will interoperate. We found the opposite to be true. As a result, the only ATM component that we did not buy from the switch vendor was the ATM NIC. Even here we limited our selection process to two or three vendors who had formal alliances with our switching vendor. While LANE 1.0 offers a higher degree of interoperability than we've seen previously, there are still many snags. If a multivendor solution is on your horizon, be very careful. Make both vendors demonstrate product interoperability before you buy. And ask for evidence that both vendors are working toward the same standards in the same time frame. You don't want one concentrating on LANE 2.0 while the other one is focused on MPOA. Also, don't be fooled by the testing going on at the University of New Hampshire and elsewhere. These tests cover only basic functionality and only the versions of the products that happened to be available at that time. Force the vendors to show you interoperability using your specific configuration. Standards aren't always the best way to go. Some of the first-generation ATM Forum standards aren't really useable. LANE 1.0, for example, does not provide redundancy or distribution of the LECS, LES and BUS processes. This means that if the piece of hardware running these services (which allow the end nodes or more likely the edge devices to register with the switch and join the correct VLANs) fails or becomes isolated from the network, the entire network will be unavailable. This standards omission will be corrected in LANE 2.0 when that standard i s available. But, for now, each vendor has a proprietary method of providing some level of redundancy. We implemented LANE 1.0 with Bay Networks' standby LANE services functionality. This gave us a primary set of LANE services running on one switch while the other two switches ran LANE in standby mode. During a failure of the primary LANE server, one of the secondary LANE servers takes over and rebuilds all of the virtual connections. In our network this takes three to 7 minutes. Without this proprietary extension, the network would stay down until the primary LANE server was fixed and brought back online. Network managers who want to deploy multivendor solutions may be forced to choose between vendor extensions that provide required levels of functionality and standards-based solutions that do not. Know the definition of interoperability. When two vendors claim that their ATM products are interoperable, make sure you know what they mean. For your network to work, your ATM NIC and LAN edge device must communicate with the switch using ATM signaling protocols and then use LANE protocols to register and join a VLAN. In addition, it will also need to use a proprietary (standards developers have ignored this area) VLAN configuration protocol to communicate with management software, such Bay Networks Optivity LAN Architect. Make certain that the components you select are both UNI and LANE interoperable and that you understand the limits of this interoperability. In many cases, the vendor's ATM LANE management software may not even interoperate with its own legacy hub and router management applications (Bay's Optivity fortunately does). Know the data plane and control plane. In legacy LAN networks, we referred to scalability and performance purely in terms of bandwidth. In ATM networks, scalability and performance must be understood at two levels: data throughput (bandwidth) and control (switching). The data plane refers to the virtual connection s (VCs) that provide bandwidth between end nodes. The control plane refers to the processes that are required to establish, maintain, tear down and recover the virtual connections used in the data plane to transfer data from one end node to another. As advertised, ATM allows network managers to significantly scale the amount of bandwidth available in the data plane. In our case, we went from an aggregate backbone throughput of approximately 500 Mbps (50 Ethernet segments) to nearly 2 Gbps (10 10-Mbps switching ports in each of 19 closets over OC-3 ATM connections). While bandwidth scaled magnificently, we found that most scalability problems with first-generation ATM architectures are in the control plane. The ability of switches, NICs, edge devices and ATM routers to handle the set up, tear down and recovery of virtual connections limits the scalability of these networks today. This is almost entirely a software and architectural problem. The switching software that we implemented has the capability of supporting approximately 3,000 virtual connections. While it is possible to build a network with many times this number of virtual connections, in a failure situation (such as a power outage) it may take the switches' control software many minutes--or even hours--to rebuild the lost connections. In some of our testing, some switches were unable rebuild connections if networks became too large. We have found that the number of VCs that a switching system can rebuild in three minutes to five minutes is the key measure of the scalability of that system. Second-generation hardware like Bay Network's 5000BH should offer big performance boosts in this department. Ask vendors what their call setup times are, how many calls per second the switch can make and what kind of CPU it sports. Call setup is very CPU intensive, and a faster CPU will yield better performance in recovery situations. Performance can range anywhere from 10 ms to 100 ms or more. Second-generati on switches should also have room for PNNI and ABR support, when they are finalized. LANE makes it worse. While 3,000 virtual connections might sound like a lot, it really isn't an embarrassment of riches, because LANE and internal switching management devices can generate a lot of overhead. For example, every LANE client requires a minimum of five administrative VCs just to register itself with LANE and provide multicast emulation capabilities. An additional VC is required for each end node with which the client wants to communicate. In the first-generation Bay Networks environment, VCs were managed by a central controller. In second-generation environments, VC scalability is increased by distributing this function to each switch. Here's an example of how to calculate VCs. 100 users on 2 different VLANs (50 per VLAN) 1 LAN printer per VLAN 1 server on each VLAN 1 router providing cross-VLAN and WAN connectivity Total LECs = 100 Users + 2 printers + 2 servers + 2 router interfaces = 106 Total data VCs per user = 1 server VC + 1 printer VC + 1 router VC = 3 Total VCs = 106 LECs * (5 LANE overhead VCs + 3 data VCs) = 848 Scalability and the edge device. One of the key determinants of the scalability of a LANE network will be the number of LECs per edge device. Some devices require one LEC per LAN port. That means a 12-port device would require 12 LECs and their associated overhead VCs. Most second-generation edge devices use an extra field in the ATM cell header (the so-called Selector Byte) to multiplex multiple ports on a device into a single LEC. Some vendors plan to take this even further and share LECs across an entire box. ATM NICs: Are they ready? One of the most interesting and useful things possible in a LANE network is the virtual server. By putting an ATM NIC into a NetWare, Unix or NT file server, it should be possible to place one file server on many (or even all) VLANs through a si ngle high-speed interface. Unfortunately, we have found that the performance of most currently available NICs limits the usefulness of this capability. The best throughput (under realistic network conditions) that we know on a 155-Mbps interface is about 75 Mbps. Most interface cards seem to support approximately 1,000 VCs, which in practical terms probably means that less than 400 users could be connected concurrently to an ATM server. When connecting this many users to a server, you probably would want to have the NIC support multiple virtual LANs. We found that most cards can only support two to six VLANs. NICs appear to be the weakest link in the ATM product chain. When a NIC is really a switch. When evaluating ATM architectures, be sure to concentrate a sufficient amount of effort on the NICs. In Ethernet and Token-Ring networks, NICs are fairly simple devices whose functionality is well defined in standards differentiated by largely by costs--and sometimes performance. The ATM NIC, in contrast, is much more complex with many key implementation details left up to the vendors. ASIC-based NICs generally will be the fastest, but we found that NICs based on general-purpose processors to be cheaper and more widely available. Cache size and the efficiency of the ATM driver software seem to be the key determinants of performance. Do I still need a router? The short answer is yes. But this is not your father's router. This router can use a single physical interface to route between logically defined VLANs (that is, sub nets). Basically the virtual router is a box with one or more physical ATM interfaces supporting multiple VLAN logical interfaces--and running on top of this is the same old Bay or Cisco routing code that we run today. Since first-generation ATM routers reuse (unchanged) routing code, we expect most of the issues to revolve around scalability and performance. How many VCs can a single interface support? And how many logical VLANs can each in terface support? Indeed, this "one arm bandit" introduces interesting performance issues. If a single OC-3 interface is used to route all traffic on a virtual network, it is conceivable that the router could be easily overrun in a 2-Gbps environment. Fortunately, additional bandwidth can be made available by simply load balancing traffic across multiple OC-3 links. Despite these issues and others not covered here, we are bullish on our Bay Networks' ATM network. Our ATM implementation continues to progress smoothly. Our initial ATM implementation has been a success. The users are happy with the performance of the physical network and we have learned a lot of valuable lessons about how to design and implement large switched ATM networks--important lessons we will be able to leverage in helping our clients migrate to ATM-based networks in the future. Despite the recent popularity of switched Ethernet, Fast Ethernet and switched Fast Ethernet at ATM's expense, I believe that none of these technologies have the capability of scaling elegantly to support large corporate campuses--none provide the ability to virtualize the network like LANE and all have inherent protocol and bandwidth limitations that will prevent them from seamlessly scaling. Sure, ATM is going through some difficult birth pangs, but the good news is the baby has been born. And while all babies are inherently ugly, wrinkly, whiny things, they eventually grow into healthy strong people. Kirk Kirkpatrick is a senior manager with Andersen Consulting's Network Solutions practice. With people like Diane Primeaux, Kevin Goss, Joe O'Donnell, Skip Dolan Jesse Luna, David Wenger and others doing most of the work, he built a 2000+ node network using switched ethernet and an ATM LANE backbone. He can be reached at kirk.kirkpatrick@ac.com. Print This Page E-mail this URL