When is a client problem also an AP problem?

My employer is currently building a new home office (HO) campus. In every building except two infrastructure support buildings, we are installing Mist AP45s which are 6E capable. The two support buildings received AP43s which don’t support 6E, but are Wi-Fi 6 (802.11ax) capable and still very capable APs.

Why is that important? Well, as most if not all of you out there know by now, WPA3 is mandatory in 6 GHz. We haven’t deployed the AP45s in many places yet, so the new HO campus is an opportunity to really get our hands dirty with not just 6 GHz, but also WPA3-Enterprise on our corp WLAN and OWE on our guest WLAN to see how some of our client base would respond and operate. But Keith, didn’t you just say the support buildings had AP43s which don’t support 6 GHz operation? I did, but as I mentioned in the opening paragraph, they are 802.11ax capable and 802.11ax does support WPA3 which isn’t something we had broadly enabled yet in our environments up to this point!

Long story short, we enabled WPA3 Enterprise and OWE on our WLANs in three buildings. One building has AP45s installed and the other two buildings have the AP43s. For a week or so, there weren’t any reports of issues until one day, someone reached out stating they couldn’t connect to the OWE enabled guest WLAN in one of the two support buildings while using their Samsung Galaxy S21 running Android 13. After looking at the Mist dashboard and downloading some of the dynamic packet captures that were taken during the client’s connection failures, I reached out to the Wi-Fi Pros community to see if anyone had seen or heard of any issues like what I saw in the screenshots below (Figures 1, 2, and 3). Bryan Ward mentioned that he saw some references in Mist’s release notes, so with that knowledge, I decided to upgrade the firmware from 0.10 to the “recommended to us” version of 0.12.27147. This immediately fixed her issue and I went on my way. Thanks again Bryan if you’re reading this!

Figure 1. Notice the SSID in the association request. I didn’t catch this during the first report of problems
Figure 2. SSID in the association request has a tag length of 15, but shows as 30 “0”s
Figure 3. Association denied by the AP with reason “outside the scope of this standard”

Fast forward just a few days later and I received another report of several Android devices, both Google Pixel and Samsung Galaxy phones all running Android 14 having issues connecting to the same OWE enabled guest WLAN. This time, the reports were for both AP43 buildings. After speaking to them and doing some initial troubleshooting, there were a few things of interest that I will break out into a bulleted list below:

  • During the connection failures, I did not find any client events for the phones in the Mist dashboard. It was as if the phone never tried to connect (keep this in mind for later)
  • A screenshot (Figure 4) from one of their phones showed “Connection failed” underneath the network after the phone failed to “join”
  • Packet captures from the Mist APs while the phones attempted to connect only showed probe requests from them without any probe responses from the nearby APs. The part that was even stranger to me was that the phone was probing for a SSID value of 000000000000000000000000000000 and with a tag length value of 15 bytes (Figures 5 and 6) – This was also evident in the association request from the initial report of problems, but I didn’t notice it until after everything was fixed.
Figure 4. “Connection failed” message under the WLAN on a Samsung Galaxy
Figure 5. Many probe requests with a SSID value of 30 0s or 15 bytes of 0 – Notice there aren’t any quotes around it which is usually what you’d see in Wireshark for a directed probe request
Figure 6. Details of probe request with 30 0s and tag length of 15 bytes

After a brief discussion on the WLAN Pros Slack (thanks Karsten and Stefan!), I came to understand that this value of 30 0s was actually the UTF-8 representation of a bunch of 0 bytes (15 to be exact). The 802.11 standard (Figure 7) talks about the SSID potentially being represented in the UTF-8 format, however, the UTF-8 SSID bit would need to be set in the Extended Capabilities IE which it was not. So based on what we know, the SSID value should be interpreted as ASCII and likely as 30 consecutive “0”s.

Figure 7. 802.11-2020 standard talks about the SSID field being 0-32 octets (or bytes) and what happens if the UTF-8 subfield is set to 1
Figure 8. UTF-8 SSID bit with a value of 0 which should mean the SSID is interpreted as ASCII

My head hurts at this point… So what does this all mean? I’m still trying to understand exactly why the client behaved the way it did, but it turns out that the problem was likely attributed to a failure loop that was initially caused by the Mist AP. I could be wrong, but here’s what I’m thinking…

You see, what I initially missed is that the APs had ALSO been beaconing (Figure 9) with the SSID value of 000000000000000000000000000000 and the UTF-8 SSID bit set to 0 on the previous firmware version. Based on what the standard says and if our interpretation of the standard is true, my opinion is that the Android clients interpreted the SSID in the beacon frame as “000000000000000000000000000000” and decided to therefore directly probe for it. And because the UTF-8 SSID bit was also set to 0 in the probe request from the client, the APs also interpreted the SSID as an ASCII string of “000000000000000000000000000000” that they didn’t have available or configured so in turn ignored the clients’ probes. This resulted in the clients never attempting to associate to the network (different from the initial problem reported when the S21 actually did try to associate) which is why there weren’t any client events for these failures in the Mist dashboard! I guess my only remaining question would be what is different in the Android OS vs iOS that the Android devices couldn’t handle this scenario, but iPhones could. Unfortunately, there’s not enough time to dig into that, but definitely something that remains on my mind.

Figure 9. Single AP (8d:33:51) was upgraded and notice that the SSID value in the beacon frames is now showing as “Wildcard” as opposed to the other APs (8d:88:d1 and 8d:7d:c1) which weren’t upgraded and are using 0s as the SSID

Ultimately, the problem was resolved by upgrading the firmware of the APs to 0.12.27250 which in Mist’s defense, was mentioned in their release notes for this type of problem. From what we were told, there were some differences in the SDKs between AP model platforms that handle this situation. This problem was not present in the SDK used in the AP45s which is why there weren’t any reports of issues on those APs. So was this a client problem? Or was this an AP problem? Or can we point the blame at both?

Hopefully this was helpful. Thanks to my coworkers who were on-site to help collect and analyze what we were seeing and thanks to the community as always for spitballing with me.