13.5 Comparing Results to the Baseline
Key Takeaways
After an optimization, measure the same metrics with the same method over a comparable time window, then compare them with the baseline.
Change one variable at a time so any improvement or regression can be attributed to that change.
AirMatch deploys plans on a 24-hour cycle, so RF optimizations should be judged over several days, not minutes after a change.
Keep changes that meet their success criteria, roll back changes that cause regressions, and record the result and the new baseline.
Typical comparisons include band distribution and retries after WLAN tuning, queue drops for priority traffic after QoS changes, and dropped broadcast counters after rate limits.
13.5 Comparing Results to the Baseline
Quick Summary: The final Optimize objective is compare results to baseline. After applying a WLAN or LAN best practice, you measure again, using the same metrics, tools, and time window as the baseline, and decide whether the change achieved its goal. Keep what worked, roll back what did not, and update the baseline so the next comparison starts from the new normal.
The Optimization Loop
Making a Fair Comparison
- Same metrics and tools: if the baseline used Central hourly reports, compare with Central hourly reports.
- Comparable windows: compare a normal week with a normal week, and peak hours with peak hours. A Monday after a holiday is not comparable to a regular Monday.
- One change at a time: if you change channel widths and QoS on the same night, you cannot tell which helped.
- Allow settling time: AirMatch re-plans on a 24-hour cycle and clients adapt over days, so judge RF changes over several days.
- Note confounders: a large event, an outage, or a new application during the window can explain the difference on its own.
Defining Success Before the Change
Write the success criteria into the change request. For example: "Reduce the 2.4 GHz share of capable clients in the auditorium, lower peak 5 GHz channel utilization, and keep UXI pass rates at or above the baseline, with no increase in authentication failures." Clear criteria prevent the common habit of declaring success because something improved while something more important got worse.
Example Comparisons
WLAN example: auditorium tuning
Problem from the baseline: high retries and channel utilization during lectures, and many dual-band clients on 2.4 GHz. Change: use 20 MHz channels in the auditorium's 5 GHz radios to create more non-overlapping channels, as recommended in Section 6.2, and confirm ClientMatch band steering is active.
| Metric (lecture hours, weekdays) | Baseline week | After-change week | Assessment |
|---|---|---|---|
| Peak 5 GHz channel utilization | High | Lower | Improved |
| Retry rate | High | Lower | Improved |
| 2.4 GHz share of 5 GHz-capable clients | Elevated | Reduced | Improved |
| Peak per-client throughput | Higher | Slightly lower | Expected trade-off of narrower channels |
| UXI test pass rate | Baseline level | Same or better | No regression |
(The table illustrates how to read a comparison; your own numbers come from your measurements.) Decision: keep the change, because the goals were met and the narrower channels' lower peak per-client throughput was an accepted trade-off.
LAN example: QoS for voice
Problem from the baseline: voice quality complaints, and show interface <port> queues on the uplink shows drops in the queue carrying voice during backups.
Change: trust DSCP from phones and the uplink, and make the voice queue strict priority in a schedule profile (Sections 13.1 and 13.2).
Comparison: voice-queue drops during the backup window should fall to near zero, while overall drops may shift to lower-priority queues. If best-effort users now complain, adjust DWRR weights and compare again.
LAN example: broadcast rate limits
Problem from the baseline: a misbehaving device floods broadcasts on access ports.
Change: rate-limit broadcast <pps> on access ports.
Comparison: show interface <port> qos shows broadcast drop counters; confirm that normal ARP and DHCP still succeed (DHCP success rate unchanged) and remember that the rate limit only contains the symptom, so the device still needs to be fixed.
Deciding What to Do
| Outcome | Action |
|---|---|
| Goal met, no regressions | Keep the change, document it, update the baseline |
| Goal met, minor accepted trade-off | Keep, document the trade-off and who accepted it |
| No measurable effect | Consider rolling back to keep the configuration simple; revisit the analysis |
| Regression | Roll back (for example with a checkpoint on AOS-CX), then investigate |
Updating the Baseline
After a change is accepted, the new measurements become the new baseline. Record the date, the change that created it, and the method. Without this step, the next comparison would be made against an outdated picture.
Reading the Numbers Carefully
- Peaks and averages: a change can lower the average while the peak hour, which users notice, stays the same. Compare both.
- Distributions, not single values: look at how many clients have good signal and SNR, not only the average. A few far-away clients can drag an average down.
- Enough data: one day of results is easily distorted by a single event; prefer the same multi-day window as the baseline.
- Related metrics: an improvement in one metric should not hide a regression elsewhere. Check authentication success, DHCP success, and UXI pass rates alongside the metric you targeted.
Reporting the Result
A short comparison report helps stakeholders and future engineers:
| Section | Content |
|---|---|
| Change | What was changed, where, and when (with the change ticket) |
| Goal | The success criteria written before the change |
| Method | Baseline window and after-change window, tools, and scope |
| Results | Before and after values for each metric, with notes on confounders |
| Decision | Keep, adjust, or roll back, and who approved it |
| New baseline | Date from which the new values apply |
Rolling Back Cleanly
If the comparison shows a regression, return to the previous state in a controlled way: checkpoint rollback <name> on AOS-CX (after checking checkpoint diff), the previous template or setting in Central, or the previous RF settings for the affected APs. Then measure again to confirm the network is back to its baseline before trying an alternative.
Common Exam Traps
- Comparing different conditions. A busy weekday cannot be compared with a quiet weekend.
- Bundling changes. Several simultaneous changes make the result impossible to attribute.
- Judging RF changes too early. Give AirMatch and clients time to settle.
- Forgetting to update the baseline. Future comparisons need the current normal.
An engineer narrows 5 GHz channels in a lecture hall and wants to know whether it helped. Which comparison is valid?
Metrics taken five minutes after the change versus the baseline's average for the whole week
Metrics from a quiet Saturday after the change versus metrics from a weekday before the change
The number of APs and clients in the lecture hall before the change versus after the change
Lecture-hour metrics from normal weekdays after the change versus the baseline weekdays, same tools
Why should an optimization change be applied and evaluated one change at a time?
Because a baseline expires as soon as any change is applied to the network
Because Central cannot report on more than one metric at the same time
So each improvement or regression can be traced to one specific change
Because AOS-CX accepts only one configuration change per device each day
After a QoS change, voice-queue drops on an uplink fall to near zero, but best-effort users report slower file transfers and best-effort queue drops increase. What is the best next step?
Adjust the DWRR weights for the data queues, then measure and compare again
Remove all QoS configuration immediately and set every port to trust none
Declare success because voice improved, and close the change without further action
Increase the MTU on all access ports so that file transfers need fewer frames
Sections you finish are checked off in the contents.