I once reported that one of my sites had lost 11% of its Google clicks after a change I had made. The real number was a gain of 25%. Nothing was wrong with the data. The comparison was wrong.
What happened
I wanted to know whether a change to how my pipelines choose topics had helped. I took clicks and impressions from Google Search Console for a "before" period and an "after" period and compared the totals.
The before period was 29 days. The after period was 17 days, because Search Console's final data lags by about three days and the change was recent. A period that is 58% as long will show about 58% of the totals even if nothing changes at all. Every site looked worse than it was.
With equal 17-day windows, the picture flipped. One site went from an apparent 11% drop to a 25% gain. Another, which looked down by about a third, was really down about 10%.
The rule
Any before-and-after comparison of search data must do one of two things:
- Use two date ranges of exactly the same length, or
- Divide each total by its number of days and compare daily rates.
Prefer equal windows, and take the before window from just before the change. If a site was growing steadily, a long baseline includes its slower early weeks and makes the recent period look better than it is.
Two more checks worth doing
- Look at the daily series before believing a big change. A real drop shows up as a visible step in the day-by-day data. A drop caused by mismatched periods does not.
- Remember that other things changed too. In my case, title fixes, indexing work and new site configuration all shipped in the same weeks. A fair comparison tells you what happened, not necessarily why.
How I apply it now
The charts on my results page use complete weeks, and the change figure compares the last four weeks with the four weeks before them, never a short period against a long one. When I report results to a client, the date ranges are always stated and always equal.
It is a small rule, but it is the difference between a report that helps you decide and one that sends you in the wrong direction.