Week 12: Analyzing Fantasy QB Projections Performance
The week 12 quarterback projections were once again mixed. I used non-linear methods and I honestly feel like they underperformed the more straightforward models I applied last week. Of course, last week I didn’t do much in the way of reflection on my projections – perhaps if I’d done the due diligence last week I would have been equally unimpressed with my performance.
Let’s get right into it. Here is a breakdown of the machine’s projections against those posted on CBSSports.com:
What is apparent to me in this comparison is that the guys at CBS aren’t interested in how good their average prediction is. They have a bias towards projecting players to perform better than they actually do on average (average projection of 18.2 verses the actual average of 16.0, where the average of the machine predictions is 15.9). The average amount by which the machine’s projections are off was -.11, i.e. a small negative bias. By contrast the average CBS miss is 2.22, a fairly large positive bias.
But for a fantasy sports enthusiast, the CBS projections are far more exciting than those of the machine. It predicts some players to dramatically outperform the average QB. While I had Brees projected as the number one quarterback with 21 points, they had Brees projected at number one with 31 points – the difference between a solid performance and a monster! Consider that the actual distribution of points had a mean of 16.0, and a standard deviation of 6.4 – a distribution that fits well with historical precedent. Predicting a 31 point game is predicting an event with a historical likelihood of maybe 2%. Sure, he was playing the crappy Falcons, and sure he’s Drew Brees – but a 31 point prediction is still well outside what you would expect his average production to be in these circumstances.
Looking at these comparisons made me realize why, traditionally, fantasy sports analysts predict rankings and not points. Put simply: relative value is easier to ascertain than absolute value. In a data scientist’s mind, the act of prediction is the act of minimizing the error between what you predict to happen and what actually happens. In the mind of a CBS fantasy sports analyst, prediction is the act of estimating what could happen – if your theories about the shape of this week’s match-ups turn out to be correct. And I’m not saying one is better than the other – in the world of fantasy sports, knowing that Drew Brees will get you 31 points of production would be much more valuable than knowing that Mike Glennon will get you 16. We just have two different ways of looking at the world. The CBS guys will miss more often, and their misses will be more dramatic. But they will also hit a few home-runs, like picking Brady to put up 25 points. That’s the inherent trade-off.
Last point on these results – notice the distribution of my errors compared the CBS errors. My projections are relatively poor for those guys who performed really, really well, and those that performed really, really poorly. My projections did a bad job of accounting for outliers, but a decent job in the majority of non-outlier cases. The CBS projections do a worse job on the really, really poor/mid-tier performances but a relatively better job at predicting up-side outliers like Tom Brady, Cam Newton, and Colin Kaepernick. Part of this is their aforementioned inherent postive-bias, but part of it is also the non-linearity of human-expert based prediction. With an appropriately fitted model we expect the distribution of errors to be normal, we expect to miss on the outliers; that’s why we call them outliers. Human experts, on the contrary, pride themselves on the ability to find those outliers. In week 12 it seems like the guys at CBS did a pretty decent job of it.
To dig a little bit deeper I wanted to convert my projections in to rankings, and see how those rankings fared against CBS and ESPN.
There’s a lot going on in the grid – but in general I’d say the machine had a few solid bright spots and some serious blemishes. It ranked Philip Rivers, the weeks top scoring fantasy QB, much higher than either ESPN or CBS. And it did similarly well with Carson Palmer, which it inexplicably ranked 5th for the week (Palmer actually came in at 7th). The only other bright spot to note is that Andrew Luck was ranked 14th by the machine, and no lower than 8th by the analysts, while he actually came in at 17th overall for the week.
The obvious blemishes include an exceptionally low rating on the 2nd ranked quarterback for the week Tom Brady (damn you Young Tom), and an exceptionally high rating on the 13th ranked Ben Roethlisberger.
As always when faced with rankings-based systems, I sort of throw my hands up after looking at the comparisons for too long. We all blew calls on guys who played well including Fitzpatrick, Smith, McCown and Tannehill. We certainly all missed the mark on guys who didn’t live up to expectations like Drew Brees and Peyton Manning.
At any rate, one more week in the books. Looking forward to a football-filled vacation weekend. Look for this weeks picks to be put out later today, and to include the Thursday competitions.