In the Court of the King In Yellow

Or, The Crawling Chaos

Names have been changed to preserve anonymity. I assure you, however, that as fantastical and maddening as the following tale may seem, it is entirely true.

One of our clients is Leng Telecom. For the last three or four weeks, they've been unable to download Necrotelecomnnicon updates from our system. So they open a ticket. Everything looks good on our end, but we start troubleshooting anyway. About a week later, we've done an exhaustive analysis of our system. The verdict: Not our problem.

Our Necrotelecomnnicon servers are good. Our client interface is good. Our network is good. We even set up a test network, and downloaded their Necrotelecomnnicon updates, using their account, from our system to our test network. Everything works. From the inner depths of our network all the way out to our border routers, everything checks out. Our servers are good. Our network devices are good. The Necrotelecomnnicon download is good.

Naturally, this leads us to one conclusion: It's their problem. So we spend a second week trying to find someone at LT that can actually troubleshoot their network. It's like pulling teeth. Nobody we talk to seems to understand or care. Nobody is a netadmin or knows how to get us in touch with a netadmin.

Meanwhile, their management continues to complain that the problem isn't solved yet, and to insist that we arrive at a timely resolution of their ticket, which has already been open for two weeks, which is at least one week too long, in their opinion. So we go into another week of trying to explain to them that it's their network, and trying to track down an LT netadmin we can talk to about troubleshooting. We finally get in touch with someone who appears to be a netadmin. But he doesn't seem to be able to engage in any actual troubleshooting activities. And he won't let us troubleshoot his network for him, for obvious security reasons. By the end of the third week, our progress consists entirely of this guy telling us there's a rumor that one of the network devices at LT might be having some kind of problem.

Meanwhile, LT management is getting quite angry that their ticket has been open with us for three weeks now, and they still can't download Necrotelecomnnicon updates. So we go into a fourth week of trying to solve this problem. As of today, at the end of the fourth week, here's how things stand: One of their border routers seems to be having some issues, and needs to be power cycled. But they won't do this, because it's a front-end device, and taking it down would disrupt service to LT's paying customers.

Our Leng engineering liason pointed out to them that Necrotelecomnnicon downloads are a back-end service, and in a good network design the same device wouldn't handle both FE and BE traffic (and would also be suitably redundant in any case). Today, their netadmin admitted the wisdom of this design principle, and petitioned his superiors to either bounce the router anyway, or improve their network design. It's not clear which, because our engineering liason is Lengese and has a somewhat unorthodox grasp of the English language.

Meanwhile, says their netadmin, could we please double-check our system, to make sure the problem isn't on our side?