Note from Retro Report: Below is a recent analysis of the drafting of NFL quarterbacks provided by David Berri, a professor of economics at Southern Utah University who founded the sports statistic blog wagesofwins.com. This analysis updates a 2009 paper, “Catching a draft: on the process of selecting quarterbacks in the National Football League amateur draft,” (*Journal of Productivity Analysis*) which Berri published with Robert Simmons, a professor of economics at Lancaster University.

For the Retro Report video, go here:

Picking a Winner: The 1998 NFL Draft

——————

**Basic Question: Have NFL general managers improved with respect to the drafting of NFL quarterbacks since 1998? **

In our original paper, we simply reported the correlation (r) between where a player was selected and his NFL performance. And we reported this analysis across a variety of statistics and at a variety of points in a quarterback’s career. The results were quite clear. Looking at data from 1970 to 2007 (and no matter how we looked at this) the correlations were extremely low suggesting pick and per play performance were not related.

It is important to note, though, that simply looking at a correlation can overstate a relationship (obviously not a problem if the correlations are very low). If we wish to know how much of the variation in some factor Y is explained by another factor X, we need to look at r-squared (which is defined simply as percentage of Y that your model explains).

For example, consider the link between draft pick and a quarterback’s career performance after his first three years in the NFL.[1] If we look at this link from 1980 to 1998, we see correlations ranging from 0.02 to 0.18. This translates into a range of r-squared from 0.00 to 0.03. In other words, when we look at r-squared we see that draft pick explains no more than 3% of a quarterback’s performance from 1980 to 1998. And that appears to indicate that where a quarterback was drafted didn’t tell us much about the quarterback’s NFL performance.

Now we turn to the question posed above. Is there any evidence that general managers are drafting better? If we look at the correlation coefficient (r), we see that from 1999 to 2013, (r) increases to as high as 0.34 for Adjusted Yards per Attempt.

**Table One: Draft Pick and Career Performance (first three years)**

1980-1998 1999-2013 _______________________________________________________________

Statistics r r-squared r r-squared

_______________________________________________________________

Completion Percentage ^{i }-0.18 0.03 -0.19 0.04

Passing Yards per Attempt ^{i }-0.11 0.01 -0.26 0.07

Touchdowns per Attempt ^{i }-0.07 0.00 -0.28 0.08

Interceptions per Attempt ^{i }0.02 0.00 0.20 0.04

Quarterback Rating -0.11 0.01 -0.33 0.11

Wins Produced per Play ^{ii }-0.09 0.01 -0.32 0.10

Adjusted Yards per Attempts ^{iii }-0.09 0.01 -0.34 0.12

Adjusted Net Years Per Attempt ^{iii }-0.09 0.01 -0.32 0.10

________________________________________________________________

i – these measures are the components of the NFL’s quarterback rating measure

ii – Wins Produced per play was noted in The Wages of Wins, Stumbling on Wins, and other academic publications.

iii – these measures are noted at www.pro-football-reference.com/

One might be tempted to conclude that the evaluation process has improved significantly. But when we turn to r-squared (or the amount of NFL performance draft pick explains), we see that no matter the measure, at least 88% of performance cannot be explained by where a quarterback is taken in the draft.

It is also possible what we are seeing depends on the sample considered. When we consider performance from years 3 to 5, we see that the difference in the two time periods is much less. And the explanatory power in the two time periods is also quite low.

**Table Two: Draft Pick and Career Performance (Years 3-5)**

1980-1998 1999-2013 _______________________________________________________________

Statistics r r-squared r r-squared

_______________________________________________________________

Completion Percentage ^{i }-0.17 0.03 -0.31 0.10

Passing Yards per Attempt ^{i }-0.10 0.01 -0.05 0.00

Touchdowns per Attempt ^{i }-0.15 0.02 -0.03 0.00

Interceptions per Attempt ^{i }0.35 0.12 0.19 0.04

Quarterback Rating -0.29 0.08 -0.20 0.04

Wins Produced per Play ^{ii }-0.20 0.04 -0.18 0.03

Adjusted Yards per Attempts ^{iii }-0.20 0.04 -0.12 0.02

Adjusted Net Years Per Attempt ^{iii }-0.20 0.04 -0.14 0.02

_______________________________________________________________

i – these measures are the components of the NFL’s quarterback rating measure

ii – Wins Produced per play was noted in The Wages of Wins, Stumbling on Wins, and other academic publications.

iii – these measures are noted at www.pro-football-reference.com/

This analysis was repeated for other career snapshots as well as performance after a quarterback’s first year, second year, third year, etc. The lack of explanatory power we see in both Table One and Table Two was also seen in these samples.

So are NFL general managers getting better? Well, perhaps. But it remains the case that where a quarterback is drafted doesn’t tell us much about what that quarterback will do in the NFL.

And this is not surprising. As I noted with Brian Burke (from www.advancedfootballanalytics.com) in an article published[2] in 2012; the performance of veteran NFL quarterbacks is difficult to predict.

As the following table – taken from our article – reports, a quarterback’s performance in a given season is poorly explained by what that same NFL quarterback did the previous season (i.e. the r-squared we see for each measure is quite low).

**Performance of a quarterback explained by what the quarterback did the previous season **

Years: 1998 to 2010

minimum 224 passing attempts in consecutive seasons

Observations: 256

__________________________________________________________________

**Quarterback Statistic ****Percentage of Current Performance**

** Explained by Performance Last Season**

**__________________________________________________________________**

EP per play* 21.0%

WPA per play* 11.7%

Success Rate* 29.0%

WP100 16.9%

NFL’s QB Rating 15.0%

Completion Percentage 31.1%

Passing Yards per attempt 22.1%

Touchdowns per Attempt 10.1%

Interceptions per Attempt 0.6%

___________________________________________________________________

* - data for these statistics – found at advancedfootballanalystics.com — is only from 2000 to 2010. Number of observations is 213.

Because so much of what a quarterback does depends on his teammates (and coaches), a veteran quarterback’s performance tends to be quite inconsistent.

Once we understand that predicting what a veteran NFL quarterback does is difficult, it should not surprise us that predicting what a college quarterback will do in the NFL is also very difficult. And so, we should not be surprised that where a quarterback is selected on draft day doesn’t tell us much about what that quarterback will do in the NFL.

This suggests the same problems we identified in this process still persist.

[1] As was done in the paper, a quarterback had to average 100 plays per year to be part of the analysis.

[2] Berri, David and Brian Burke. (2012). “Measuring Performance in the NFL.” in *The Economics of the National Football League: The State of the Art*, ed. Kevin Quinn; Springer Publisher: 137-158.