Victory over my Sys Admin

Warning

Our building has 2 commercial DSL drops via the same company on opposite sides of the office. One has been up providing Internet access for years, plugged into a Cisco Catalyst 2950. The other new one was to be plugged into a Catalyst 3750. For some reason, plugging in the 2nd DSL modem (CSU/DSU) would drop Internet access on the other side. The lines each tested well to our provider.

After a lucky hour of troubleshooting, I determined that the second modem could provide Internet access at the same time that the other "network" was up. As soon as you plugged that 3750 switch into the modem (that was only connected to the ISP), Internet access would drop on the other side. It seemed like phone wires were crossed.

I had a sneaking suspicion (read guess) that it had to do with the Switches Spanning Tree Protocol (STP), a layer 2 algorithm that protects you from loops in the network by logically creating a "spanning tree". After describing the symptoms and my guess to our Systems Administrator, he was sure that wasn't that problem. Sir, there's no way, with my understanding of STP, that this could be the problem. Maybe something similar, but STP just blocks single ports. After observing that the single port with the modem on the 2950 was orange (in blocking mode), he still didn't believe me. In his defense, we were troubleshooting a lot and I've only mentioned the relevant findings. Only after my network engineer had no idea did I contact the technical people at the ISP. He'd seen the problem before as described in a few sentances. "I don't know what causes it, but we've seen it". Only then would my sys admin modify the STP settings. Viola, it worked perfectly.

If you didn't follow what was happening (I don't blame you), the two Cisco switches were forming a spanning tree (that happens when you uplink switches) across the DSL gateways and through the ISPs cloud. We were supposed to be able to think of the two circuits as totally separate. Weirder yet was why that spanning tree blocked the port plugged into the modem on the old Cisco Switch. The new one was fine. The ISP said 2 new switches wouldn't have that problem.

This post wins the "Most boring" award. What I'd like to convey for Industrial readers is that growing networks require a specialized skill set. IT can be your friend and your ally. It's important to break that historical tension between plant production and IT.