The laughing stock of Twitter a couple years back after Devin Akin did his extensive post of how much of a failure it is. (http://divdyn.com/technical-analysis-blackbox-hd3-wi-fi-solution/)
I had the pleasure of hanging out with Devin on the first week of my employment at my former workplace and it was during one of these days that a colleague and I took Devin over to our new hospital to show him this monstrosity of a design. The pictures in his post are from that location.
In case you haven’t clicked on that link yet, let me explain what HD3 (High Density, Design, Demand?) is in some layman’s terms. HD3’s whole purpose is to put SSIDs on different sections of the 5Ghz band. They divide it into low, medium, and high band sections. Their argument is that the higher frequencies are “more robust” and “faster”, so they take all your SSIDs and put voice and medical SSIDs on the high band, Guest on the (DFS) mid band and 2.4, and standard employee SSID on the low band. How do they do this? They use an RF filter called a Tri-plexor to limit the RF frequencies that can be heard by the AP. 3 profiles are created for each “band”. Then, they remove a 2×2 ceiling tile and inside of an overpriced antenna box that they sell you, they place 3 APs and 3 Tri-plexors. Each AP has its antennas plugged up to the respective Tri-plexor and then each Tri-plexor is connected to the antenna. Yes, they are connect 3 APs to one antenna. If they wasn’t interesting enough, think about how they are getting 3 bands out of the 5ghz frequency range. They use the Tri-plexors to limit the 26 5Ghz channels into 3 groups of 4 channels. 36-48, 100-116 (5 but at this point does it matter?), and 149-161. The other channels are ignored and cannot be used in the system. If you are reading that correctly and know anything about the limited channel problem of 2.4GHz, you are coming to the conclusion that they are bringing that problem over to the 5GHz band. Because, wifi is easy, right? How hard is it to not run wires?
Here is my colleague trying to draw it on a whiteboard…
So in the simplest terms possible, HD3 is 3 wireless systems that are installed over top of each other. These 3 systems only have 4 channels to work with and CCI and ACI go through the roof. The design that was installed at our location had an average power output of 14dBm for all access points. The antenna boxes were installed in locations that wouldn’t make sense except in a coverage only model, and introduced failures throughout the building with low power clients. Most AP boxes were in hallways and open areas.
This all sounds so wonderful! Because with the filtering, the APs see this:
But unfortunately, then the clients see this:
To quote the sales guy during a meeting: “When the network has no clients, the system is flawless!” (Yes. Really.)
Here are some of the tickets that I was given in the first 2 weeks of my employment, which was 2 months after the grand opening (problems began at day 1):
- Wireless VOIP phones (iPhones) were constantly dropping in all areas of the hospital. Wireless ceased to function in any stairwell or elevator, and coverage to those areas was offered to us for only an additional umpteen thousands of dollars! What an offer!
- Registration (Wireless on wheels, or FLO-carts) would drop in any room they went in. They would also roam while they were sitting still. Nurses very quickly refused to take them into rooms and had to register patients on paper, and then transfer that information to the carts that were sitting in the hallways.
- Pharmacist PDAs that were used to scan and dispense medications would not work in the rooms.
So what do you do with that? You rip it out.
After 6 months of arguing over protocols and wireless fundamentals, 3 on-site visits and “tunings”, unending tickets and troubleshooting, and numerous chief level meetings trying to keep their gear in the hospital, it was approved for me to rip and replace the entire system.
My first step in ripping the system out was to figure out what could remain. With 3 CAT6 cables run to every box, there was a definite desire from management to recoup some of the money that was used on the copper. In the hospital, we had a total of 5 SSIDs. 2 guest networks, a medical device network, our main connectivity network, and a VOIP network. Since the VOIP network was the only one on the high band, I was able to successfully move all the SSIDs to the mid and low filters. This freed up the cable that was run to the “high band” AP without having any downtime.
Next, a new design was created using the AP215 from Aruba. We had already moved to Aruba as a wireless vendor at 4-5 remote sites, and with the Juniper exit from the wireless industry, we decided to continue that migration. At the time, the AP325 had just come out but the code had not reached GA(general availability) level so I was uncomfortable moving to such a new model. We also discussed the AP225, however, my high capacity design would negate any need to put a high client AP in such a dense deployment. The AP215 had been out around 6 months and we felt comfortable with standardizing on that model. This is also the model that we had designed for at the main campus. The design was completed using the standard steps:
- Walk the building and take measurements of RF attenuation for each change in wall and door type. Since the hospital was brand new, this didn’t take long because everything was the same. There were no new additions or changes to the building. I took a baseline measurement from a standard wall and door, and measured some of the oddball materials that were scattered throughout. Elevators were also measured as well as floor-floor signal.
- Using Ekahau, I was able to take those measurements and create a model of the building that represented the environment to the best of my ability.
- APs were added at 10mw from the top floor working my way down. Primary and secondary coverage were planned at -67 and -70dBm levels. Elevator and Stairwell coverage was included (and ended up only costing about 2k. Imagine that.). Channel planning was also completed. I’ll go into that during another post.
- Once the design was complete, low voltage maps were pulled from Ekahau and the new placements were compared to the current layout of the HD3 system. Cables that were able to be relocated were marked accordingly and the cabling contractor was able to begin moving the existing cables and running the new cables. They also hung all the new APs since we were not using any cables that were in use. This process took around 3 months. By installing all new cabling we were able to find DOAs, test, and preconfigure Aeroscout location tracking before go-live.
- During that time I was able to fully configure the new controller design and spin up 2 sites prior to the go-live. This ensured that when the hospital went live there would be no troubles. Phone connectivity, medical devices, and other equipment was brought on location to the test bed site and tested for connectivity issues that would cause problems at the hospital. All tests passed although we did determine that the 8010 Spectralink phones that were in use would have to be swapped out. The phones were older 1×1:1 devices that would not support 2 spatial streams. Because of the Aruba profile settings and how they operated, we would have had to limit the APs to 1×1:1 in order to get the phone to connect. I described this to management as “putting wooden wheels on a Ferrari” and they decided it would be best to upgrade.
- The new controller system was “baked” for a month before go-live. The two testing sites were live and working well. On the night of go-live, at 12am during a maintenance window, I was able to turn all the Aruba radios on, and turn all the Juniper radios off. It was a glorious event for an entire hospital system that had never had a wireless network that worked efficiently. All clients successfully roamed to the new Aruba system and only a few devices needed to actually have hands put on them. The conclusion to all the efforts made was one of the most boring 6 hour blocks of 2016. And I was very appreciative.
After the go-live, tickets stopped immediately. All of the problems mentioned earlier in the post were eliminated and my life was boring for 2 weeks. Post installation surveys had been completed before the go-live since all access points were up and mw was reduced to 5, and sometimes 1mw. Unfortunately, some of my measurements had been a little on the high side. This made the network a little hotter than I liked but it was easily resolved and was still lightyears above the coverage that was there originally.
There were some issues that arose later in the month involving iPhones, Airwatch, and Avaya. I’ll go into some of those next time!
’til next time…