How Accurate is the AP Top 25 Poll?
The college football world swoons when the first AP Top 25 poll is released in late August. For starters, it's a signal that football is close. But far too much stock is placed on where teams are ranked (or if they are ranked at all).
Here's the problem: the initial AP Top 25 has become less and less meaningful over recent years because its accuracy is so poor.
I decided to study the last 15 years of AP Top 25 polls (since 2000) and found that whether looking at the entire poll or just the top of it, the initial poll is inaccurate when evaluating the best teams in college football.
I took two approaches at evaluating consistency and correlation of the initial AP Top 25 poll to the final AP Top 25 poll. First, I only looked at those teams who were ranked in both the initial poll and the final poll. By ignoring teams who fell out of the AP Top 25 completely after being ranked in it to start the season, we can use regression analysis to examine linear relationships between initial ranking and final ranking.
But that won't account for teams who were ranked and then fell out, or teams who were not ranked at the start of the season who were ranked at season's end. So I took a far more granular approach and simply evaluated top and bottom 5 consistency, as well as looking at the number of total misses we've seen (unranked to start the year but ranked by the end).
In either approach, the result was the same: the initial AP Top 25 is less accurate and less helpful than ever before.
Examining Ranked Teams in Initial and Final Poll
There's a clear trend over time that shows ranked teams in the final AP Top 25 poll becoming less and less correlated to their initial AP Top 25 ranking. I broke the 15 year period in half (as close as possible) and analyzed the 2000-2006 period and the 2007-2014 period.
2000-2006
Below, you can see the results when you compare the initial AP Top 25 (x axis) with the final ranking (y axis). If the voters were perfect, the first, last, and every AP Top 25 in between would be identical. If the goal of the AP Top 25 is to rank the best 25 teams in college football, in order, the initial poll should be as close as possible to the final poll. Obviously, "any given Saturday" there are better teams who lose games to worse teams (aka upsets) but the goal of any poll should be to rank the best 25 teams in order. Thus, if the pollsters were perfect, the graph below would consist of nothing more than a straight line from the lowest, left-hand quadrant to the upper, right-hand quadrant. The initial #1 ends up #1. The initial #10 ends up #10. The initial #25 ends up #25, and so forth.
Obviously that never happens. Many teams are ranked initially and fall out entirely. Many teams are not ranked initially but are ranked by season's end. The graphic below depicts those teams ranked both in the initial and final AP Top 25 poll. The ultimate result is that from 2000-2006, in searching for a linear relationship between initial and final rankings, 22% of where a team is ranked in the final poll is explained by their initial AP Top 25 ranking. Essentially, a quarter of their final rankings is related to the initial ranking, which means three-quarters is related to other factors (assuming they don't drop out altogether).
2007-2014
This analysis is the same as the prior period. Except instead of 22% of the final ranking being explained by the initial poll, that percentage has dropped by almost 2/3rds. Now, only 8.5% of a team's final ranking is explained by their initial ranking.
When comparing the charts side by side, you can clearly see the effect of that big drop down to 8.5%.
The orange trend line's slope is far less steep (more horizontal) in the 07-14 period, indicating less of a relationship between initial and final ranking. And the teams littered at the far reaches of the graph, the top left or the bottom right (where inaccuracy rules), are far more numerous in the 07-14 graph.
To illustrate it more clearly, I added shading in the lower right of both periods. Teams in this area were initially ranked much worse in the first AP Top 25 as compared to where they ended the season. As an example:
- NO teams were ranked in the bottom 5 of the initial poll & finished top 10 from '00-'06.
- But 9 teams ranked in the bottom 5 of the initial poll & finished top 10 from '07-'14.
To further demonstrate the progressive weakness of the poll over time, refer to another example using the blue shading on the two charts:
- 8 teams are ranked inside the same blue shading in the earlier time period
- 19 teams are ranked inside the blue shading in the later time period.
Just the Top 5
In the early 2000s, the voters were pretty accurate with the top 5 teams. From 2001-2004, of the 5 top teams in each of those 4 years (20 teams in total), all 20 teams (100%) were ranked somewhere in the first AP Top 25 poll of the year. No total surprise teams existed, and in fact, on average 3 out of the 5 teams who finished top 5 at the end of the year were also ranked top 5 at the beginning of the year.
But recently, that hasn't been the case. Instead of 100% of the top 5 teams being found somewhere in the initial AP Top 25, in the last 3 years 33% of teams who finished top 5 at the end of the year were NOT EVEN RANKED in the first AP Top 25 poll! It's not that they were ranked 20th and really outperformed their ranking -- they were outside the top 25 intially yet finished the season among the 5 best teams in college football.
Some of these teams included:
- 2014 TCU -- finished #3 -- were not even ranked until 9/28 (25th)
- 2013 Auburn -- finished #2 -- were not even ranked until 10/12 (24th)
- 2013 Michigan State -- finished #3 -- were not even ranked until 10/26 (24th)
- 2013 Missouri -- finished #5 -- were not even ranked until 10/25 (25th)
These are massive miscalculations. In the early 2000s, we never witnessed these types of errors from the voters.
Just the Bottom 5
Obviously the bottom 5 should be harder to predict. From 2000-2006, voters had 14 of the 35 teams (40%) who ultimately finished in the bottom 5 somewhere in the top 25 to start the season. And those 14 teams finished an average of 7.8 seeds different than they were ranked in the first poll.
But since 2007, voters have been worse with the bottom 5. Only 11 of 40 teams (27.5%) who finished in the bottom 5 were in the initial AP Top 25, so their percentage fell 12.5%. Additionally, those 11 teams finished ranking 12.6 seeds different than they were ranked to start the season.
Bottom 15
Of the teams ranked 11-25 in the final AP Top 25 poll, only 49.6% of those teams were ranked in the initial AP Top 25 poll from 2007-2014. That number is consistent with the earlier period as well, as 49.5% of teams ranked 11-25 by the end of the year were ranked in the initial AP Top 25.
It's a further example of how voters are missing on teams. If we're seeing consistent examples of unranked teams finishing in the top 5, of course we will see examples of unranked teams finishing in the top 25. But when less than half of the teams in the bottom 15 at year end were even ranked at the beginning of the year, it begs the question "why should we care about the initial AP Top 25 poll?"
AP Top 25 is "Discredited" by the Playoff Selection Committee
College Football implemented a Playoff Selection Committee, and their first rankings of the year don't emerge until week 10. In 2015, that will be November 3rd. It's really pointless to consider polls prior to that point in time, as polls are now irrelevant when it comes to a committee - they consider far more than just polls.
In fact, the bylaws of the selection committee actually preclude them from even caring what the initial AP Top 25 poll said. Specifically: "Committee members will be required to discredit polls wherein initial rankings are established before competition has occurred."
But more than just the initial poll appears to be precluded from analysis by the Selection Committee. Because the AP Top 25 poll's initial rankings clearly are established before any game has occurred, it appears the Selection Committee members should be "required to discredit" the AP Top 25. Entirely. All year long, not just initially.
There are a myriad of reasons why it's getting harder to accurately rate the best 25 college football teams in the nation, particularly in mid-August. Many historically bad programs are becoming better, and in many cases, are capable of pulling off upsets over historically stronger programs. But regardless of the why, the fact is that the initial AP Top 25 is more meaningless now than ever. And placing much stock in it is assuredly a waste of time.