Another year has gone by, and another VCF has happened. This was the largest VCF in history, occupying the entire 100,000 square foot Discovery Hall at the Schaumburg convention center in Schaumberg, Illinois. It featured an unprecidented number of exhibits, guests, and talks, and ran from September 11th through the 13th. In this post, I’d like to talk a look at the IP network, and the technologies that supported this massive show.
At its core, the IP network for VCF MidWest is a very anachronistic component of the rest of the show. The network is comprised almost exclusively of modern equipment, which is cross-connected to the retro network. This is done for two reasons:
- We want the network to be relatively servicable, and so this is easier to do on modern hardware.
- Other people depend on the network doing modern things, which is easier for it to do with modern hardware. Phrased differently, we don’t want to pass data from payment terminals across the vintage network, even though that should already be encrypted.
The Hardware
The modern network provides internet access, LAN connectivity, and various services that we deployed on the fly. It is comprised of these major parts:
Mikrotik CCR2004
This is a very capable router that sits at the heart of the IP network. It provides NAT translation services, IP firewall, BGP route reflection, and DHCP/DNS, just to name a few of its duties. I really like the quality of the Mikrotik hardware and software, and through the use of the Terraform Provider, the entire network is defined as code, and can be rapidly configured and de-configured.
HP 2350 Series Switches
All of my core network switches are HP2350 series 1U switches. These provide a good mix of being cheap, quiet, and relatively featureful while still being PoE enabled. I’m developing a Terraform provider for them, which should improve their utility substantially.
Ruckus ICX7150-C12P Switches
I have a large collection of Ruckus ICX7150-C12P switches, and while experiencing the design choices that make up FastIron would make anyone reconsider buying CommScope products, these switches are amazing. In a small steel box you get PoE, managed features, and its fanless. Four of these were deployed to support wireless equipment that was attached to ShadyTel provided telephone poles.
Mikrotik mANTBox ax 15s
This year we tried a new approach for wifi. While I have in the past deployed Aruba IAP-225 series access points, the sheer scale of the venue this year, coupled with some of my own health issues, demanded a solution that did not involve a staggering number of radios. Mikrotik’s sector radios were a nice upgrade to my fleet, and supporting 802.11AX will go a long way for other events I support.
These sector radios have a 120 degree pattern, and so we secured 3 of them to a telephone pole using a custom bracket. At the show, four telephone poles were supplied with radios to provide a good coverage of the floor. These poles were located in pods D2, D6, B2, and B6. I had some concerns about there being RF nulls beneath the towers, but this thankfully did not occur.
Mikrotik hAP Lite
This is a very small low power router, but it serves as the VPN router that connects the on-site network to a collection of different off-site networks which host other support services. Namely ShadyTel’s outbound-calling capability, which depends on hardware at an off-site location to provide POTS lines back to all the equipment on the show floor.
Overhead Fiber
This was the first year that I ran fiber optic cables overhead. While my preferred fiber is armored and rated to be on the floor, this would mean a lot of tape to keep it in place and a lot of time to put it down and take it up. To avoid this, I designed tension clamps that could be 3d printed to secure the fiber along with all of the other overhead cables already going up on the telephone poles. This allowed us to have the throughput of fiber while not having to make a mess of the floor.
Stay tuned as I will put the tension clamp components up on Printables for others who might want to use overhead fiber.
Of Many WANs
To actually get internet on to this network, we need a WAN or Wide Area Network connection out to the internet at large. We do this through a variety of sources, some of which worked and most of which didn’t. Each of these connections was added with the same route priority so that they would form an ECMP egress group using a standard 5-tuple hash to glue outbound traffic flows to specific WANs. I’ll start with the one that predictably failed within minutes of being turned on.
The Hotel’s Own Network
I had a Miktrok Groove AC configured to join the hotel’s network, which failed predictably. The Hotel’s network uses a captive portal, which while I was able to get the MAC address of the radio added, the bandwidth available was functionally unusable. This connection was dropped early on to avoid debugging problems that were not worth the time or effort.
You may wonder why we don’t just plug in to an Ethernet line and redistribute that. For those who have not run conferences before, you may be surprised to learn that the Ethernet capabilities of most convention centers are practically non-existant, and where they do exist, they’re almost always managed by an MSP called Encore. Encore appears to have last updated their pricing guides some time around 2005 as evidenced by the 5 figure quote I received for getting a 30Mbps line. If you work for Encore and want to discuss turning your internet product into something someone might actually buy, rather than just laughing at the ineptitude of an aging conference services company, please feel free to reach out.
AT&T LTE
I have a business AT&T LTE service which I make use of whenever I’m not at home, and which works extremely well as a single endpoint service. My only real complaint with the CPE I have is that its external antenna connectors are TS9, which is a pretty bad slip-on RF connector. I would really prefer something better, but the AT&T connection had some problem that I still have not been able to identify. This connection was relegated to fall-back usage, and while it did work well for that, it was painful to use.
T-Mobile 5G UWB
The bulk of our connectivity at the show comes out of a T-Mobile 5G UWB connection. I use external 2x2 MIMO antennas from Waveform which allow a great connection even from inside the building. While this works well enough, it is only a cellular service, and can saturate fairly quickly. It is rock-solid though, and is usually the first connection we bring up since it is so stable.
I do believe something in T-Mobile’s core network is wildly misconfigured though, as we need to drop the MTU on the show floor down to 1200 to reliably get traffic out. There’s also the issue where the CPE won’t respond to ARP, which makes detecting when its crashed rather tricky.
Starlink
This year I added Starlink as an additional WAN to the show network, and I was extremely satisfied. While the process of getting the antenna outside and setup was a bit of a pain, once it was setup it worked flawlessly providing a stable connection over the 3 day run. This is by far the most expensive connection since I had to buy the terminal equipment outright, but it was still not terrible.
My only real complaint with starlink is that users are expected to administrate the system using an app despire the terminal equipment being sizable linux hardware, and a web interface would be well within its capabilities to host.
Physical Plant
This year VCFMW was a very large show, occupying the entire 100,000 square foot Discovery Hall in Schaumberg. This kind of scale is on the limits of what Ethernet can comfortably do, so we migrated to an all-optical core network. It had been my intention to run a 10Gbit/s everywhere to the switching equipment, but an as-yet-unidentified issue prevented the 10G link from coming up on two switches on the floor. I suspect this has to do with optical loss stemming from ~80m of fiber being coiled up on spools and Fiber Store’s 10G optics probably being marginal at these distances in the first place, but have yet to debug this issue fully.
At the base of four telephone poles were Ruckus ICX7150-C12P switches to terminate the fiber and supply PoE power to the Mikrotik mANTBox sector radios. These are integrated access points with a sector antenna designed for WISP operations. Rest assured, we had the transmit power turned down to an acceptable level on the floor, which is required to make roaming work in an enclosed space.
I am generally fairly happy with CAPsMAN, and it is the system that I use in my home, however in a convention center environment it was a swing and a miss this year. Problems included all the radios choosing the same channel, weird issues with datapathing, and some DNS issues I’m still not certain I have pinned down that did not show up on the wired network. This will be the focus of a lot of off season work.
On Prem Hosting
Finally I want to talk some about on-premise hosting and how I manage the network at large. Given the WAN issues I have talked about above, it should come as no surprise that all the critical services have to run on-prem in compute I bring with me. This also includes the network control plane. As I talked about in Software Defined Networking, I’m quite fond of controlling everything with Terraform where possible. To this end, the entire show network was configured using Terraform in the network core, and I’m working on providers for Ruckus switches and the HPE switches I use on the network as well. This will allow the entire network to be controlled using a unified network configuration system, and make mistakes much less likely.
This covers the network itself, but what about IP applications? Those are also hosted on-site as well, as Nomad managed containerized workloads. Nomad makes a great single-node scheduler, and it allows for rapid deployment and rapid-removal of applications. For example, at the show we deployed a speedtest application on the floor to help us troubbleshoot where slowdowns were happening. This was only possible because of a foundational layer that made applications cheap to deploy.
Looking Forward
Next year promises to be just as good of a show, and an exciting new season of ShadyTel Hijinks. If you go to VCF and have ideas of new demos or any thoughts on what I have posted here, please feel free to send me an email.