ExtremeCloud IQ

 Why so many retires, but only on wifi interface

Jump to Best Answer
systemscsn's profile image
systemscsn posted 08-16-2021 15:37

Recently we got rid of the 2.4ghz band on wifi interface wifi0 and replaced it with 5ghz, leaving interface wifi1 alone (802.11ax_5g) radio profile radio_ng_11ax-5g and channels set to Auto.

 

Heres what i noticed, and its probable that this has been going on all the time, but with more devices connecting to wifi1, we now see the issue.

 

only on wifi1 am i seeing a ton of RX TX retries.

 

 

What is the reason for this?  thats an insane amount of retires, and notice how the other is totally fine.

 

any ideas?

 

thanks,

J

Ovais Qayyum's profile image
Ovais Qayyum

Hi J, 

The screenshot has the answer, on 5GHz interface you have got 56 connected clients and the radio can sense another 72 “surrounding clients” which are the clients not directly connected to this AP but may be using the same channel i.e. 161 which may also also be used by another AP. The AP in question may not be the only AP in your environment which is using channel 161. So, with 56 connected clients and 72  other clients that are probing on the same channel 161, the AP will hear a lot of chatter which has increased the channel utilization to 69%. High channel utilization means less transmit opportunity for each client, more collisions, more back-off and more re-transmit attempts. 

There are many reasons for high transmit retry % in a network:

1- Too many Clients and too less of APs/Radios.

2- RF interference caused by the neighboring APs, it could be adjacent channel or co-channel interference. An example would be an environment where APs using the same channels can hear each other loud and clear. 

3- Co-channel interference (CCI) caused by clients, CCI is the no. 1 cause of interference in a WiFi network and it can’t be completely mitigated. Only a properly planned and designed network can help lessen the impact of CCI.          

 

Regards,

Ovais

systemscsn's profile image
systemscsn

Guess i didnt know about that whole using the same channel thing - even if the device isnt on that AP.

 

So, i have 56 clients on that AP, and somewhere around are another 72 clients on a different AP, but if both AP’s are using the same channel, then both AP’s “hear” the other clients?  is that what you are saying?

 

Because we have two other AP’s in the general area, and while i don't know for sure, id hazard a guess that at least one of them is using the same channel.  I know it wouldn't be #1 as we have AP’s everywhere, probably too many.

 

im thinking if the above is true, then what?  reducing the power isnt going to help! and id bet I cant configure each of the 160 AP’s to only use a channel that another AP isnt using (all on their own single channel… probably not  good even if possible).

 

I appreciate the explanation, it helps, but it makes me even more full of questions. UGH i didnt know that about a channel shared by AP’s close enough to see each other would do that, but it does make sense.

 

Thanks,

Ovais Qayyum's profile image
Ovais Qayyum

Here is some good information on this topic from Extreme Networks, this should answer most of your questions:

https://www.extremenetworks.com/extreme-networks-blog/whats-the-biggest-cause-of-co-channel-interference/

Here is another article on how a 5GHz network should be planned, though it references Extreme Identifi wireless solution but the fact that RF design does not depend on the physical hardware solution makes it valid and valuable:

https://extremeportal.force.com/ExtrArticleDetail?an=000079876

 

I would recommend you take a look at the current channel plan to verify if you have APs that are close to each other and have been assigned the same channel. You can use Network 360 (if a floorplan is configured and AP are mapped) to check the AP locations, channel and power information. Another way could be to use “Devices” screen and sort the APs based on Hostnames if you have them by locations and then check the radios stats:

 

     

 

Regards,

Ovais

systemscsn's profile image
systemscsn

Thanks for the links, ill be sure to check them out (today is the first day of school, and the students are back today, so its a bit crazy).

 

Quick couple of questions on AP’s close to each other AND using same channel.

 

If the channel is set to Auto on the AP’s, wouldnt the AP use a different channel from the one thats closest? Because in checking, wifi1’s channels are set to Auto on pretty much all our AP’s.

 

This issue only applies to AP’s close to each other AND using the same channel, right?  For instance, in one building there is an AP on channel 161, and then in a different building - a long distance away - there is another AP thats on channel 161.  Because of the distance, we wont have the issue you explained, correct?

 

I really appreciate your time on this.  The issue we had with our deployment, is we literally ran out of time from installing, and getting everything working.  We got it all working the day before school started (bonjour wouldnt work), and so couldnt do any kind of tweaking.

 

 

 

lastly, we have 160 AP’s throughout our campus, does this mean that i have to pick a channel for AP’s that are close enough together where there could be an issue of using the same channel?

 

thanks again for everything, and for helping me get my head around this, appreciate.

Best,

Jason.

systemscsn's profile image
systemscsn

Oh sorry, mean to make a comment on this part of what you typed

 

“I would recommend you take a look at the current channel plan to verify if you have APs that are close to each other and have been assigned the same channel. You can use Network 360 (if a floorplan is configured and AP are mapped) to check the AP locations, channel and power information. Another way could be to use “Devices” screen and sort the APs based on Hostnames if you have them by locations and then check the radios stats:”

 

We seem to have the rx/tx retries in one place more than others, the SU Lecture Hall. My first screenshot was of that one.  I just checked it, and its not on the same channels as the 3 closest AP’s.  At least its not now, but of course this is many hours after the issue.

 

Just wanted you to know.

 

thanks,

J.

Ovais Qayyum's profile image
Ovais Qayyum

Hi Jason,

Here is the answers to your first two questions:

If the channel is set to Auto on the AP’s, wouldnt the AP use a different channel from the one thats closest? Because in checking, wifi1’s channels are set to Auto on pretty much all our AP’s.

Auto channel assignment is enough usually but its not perfect and some network environments may require you to either have static channel plan OR have Auto channel assignment select the channels and then you tweak the channels as needed. It is more important in networks where only limited no. of channels are available for use, which means channels will be repeated many times over on different APs which may create CCI. I am not sure what your channel profile looks like, if you are using all non-DFS channel or just UNII-1 channels are being used. The recommendation would be to skip DFS channels and use all the remaining channels. I assume this is most likely already the case.    

 

This issue only applies to AP’s close to each other AND using the same channel, right?  For instance, in one building there is an AP on channel 161, and then in a different building - a long distance away - there is another AP thats on channel 161.  Because of the distance, we wont have the issue you explained, correct?

That’s correct, the issue will surface only when you have APs in the hearing range of each other while using the same channel. The following diagram should help visualize that; here channel 6 is the cause of CCI since two APs can hear each other on channel 6 which will cause the APs and their associated clients to compete for channel access.

 

 

We seem to have the rx/tx retries in one place more than others, the SU Lecture Hall. My first screenshot was of that one.  I just checked it, and its not on the same channels as the 3 closest AP’s.  At least its not now, but of course this is many hours after the issue.

In this particular location, your problem may not be the CCI due to other APs but If you have a large no. of concurrent clients on this AP which are all: 

  • Actively doing Tx/Rx
  • Using any application that utilizes broadcast/multicast etc.
  • Legacy 802.11b/g clients which are slower and drag the faster clients with them
  • Clients that are located far from the AP will also increase retries, because these clients won’t get Tx opportunity as often as the clients located closer to the AP, hence they keep re-trying.   
  • Clients that are sticky and not roaming away from the lecture hall AP when they really should move on to the next better AP. This is a very common problem with hand-held client devices. These clients will kill the performance of the AP.
  • If its a large lecture hall and the network is designed to cover the hall with single AP, you may have too many clients for a single AP, adding another AP will help load balance the clients across the two APs and free up the channel. But this should only be considered if the above mentioned conditions are not found to be true. 

You should investigate this AP in the light of above and check what’s the cause of high channel utilization and retries. Let me know if this helps.

Regards,

Ovais

 

 

systemscsn's profile image
systemscsn

Thanks Ovais,

 

the visuals, and your explanation really help.  The more im learning, the more there is to learn, and I also understand why this is/can be a single career, so much to learn and know.

 

About the lecture hall, it seats about 60 people and really isnt that big, maybe 30ft x 30 ft and for the most part people dont roam.  it sits between a corridor and another area, in the corridor there is an AP, and the other area has an AP.  So, if Auto on channels wasnt doing a good job, then it is possible that one or both of those other AP’s was using the same channel as the LH AP.

 

I guess I have a couple of options:  1) wait until i get a complaint and check the interface and see if any of those two AP’s are on the same channel, if so, then its possible thats the issue.  2) change all three of those AP’s to different channels, and then monitor that for complaints.

 

Im going to go through all the AP’s where they are fairly close, and make sure there isnt one on the same channel; at least this way, im taking a potential issue out of the equation.

 

Thanks SO much for your time in explaining all of this to me, its been extremely helpful.  Have a happy day.

Ill post back here what the results are; may be some time though (a couple weeks tops).

 

Best,

Jason.

Ovais Qayyum's profile image
Ovais Qayyum

Good deal, will wait for your feedback. 

BTW, don’t forget to pay attention to the following clients related points that I had mentioned. Those in the lecture hall may not roam while they are in the hall, eventually when the lecture is over they will move out and that’s where they may stick to this AP even if they have moved to an area few rooms across.    

 

  • Actively doing Tx/Rx
  • Using any application that utilizes broadcast/multicast etc.
  • Legacy 802.11b/g clients which are slower and drag the faster clients with them
  • Clients that are located far from the AP will also increase retries, because these clients won’t get Tx opportunity as often as the clients located closer to the AP, hence they keep re-trying.   
  • Clients that are sticky and not roaming away from the lecture hall AP when they really should move on to the next better AP. This is a very common problem with hand-held client devices. These clients will kill the performance of the AP.
  • If its a large lecture hall and the network is designed to cover the hall with single AP, you may have too many clients for a single AP, adding another AP will help load balance the clients across the two APs and free up the channel. But this should only be considered if the above mentioned conditions are not found to be true.

Regards,

Ovais

systemscsn's profile image
systemscsn

Hi,

 

So, i saw two AP’s that were experiencing an issue (power level too high), and made a change to power.  When i try to update the AP, either Delta or Complete, it fails.  This happened before and i had to go unplug the AP form the switch before it would take the update.  I tried that with these AP’s and it still fails when trying to update.

 

Here is the thread with all the steps ive tried without success:

 

 

The title is wrong though, no matter what I try, i cant update the AP.

Any help to resolve this issue is appreciated.

 

thanks,

J

Ovais Qayyum's profile image
Ovais Qayyum

Hi J,

If others of the same model accept it then your configuration is probably good. I would factory reset the problem APs and give it a Complete config push. Also make sure they're using the same firmware. It's very rare but I've seen it only once that various firmware act differently in terms of configuration pushes.

The next things that causes configurations to fail is CAPWAP stability. Some firewalls like Palo Alto & SonicWall like to close the UDP 12222 quickly and sever the connection to the cloud.

How does the CAPWAP client info look like on these two APs? try SSH into the AP and run “show capwap client” and post the output.

Regards,

Ovais

systemscsn's profile image
systemscsn

All of the AP650(AH) are on the same firmware version 10.0.5.0, we have AP1130’s on 8.2.4.0 as well as two of them on 10.0.5.0 and a single AP460C on 10.2.0.0.

 

I checked with the consultant that installed our firewall, and the ports are open, as well as persistent NAT being enabled.

 

Ill try the SSH, and see what it shows.  Thanks for the info, appreciated.

J