Welcome to PECOTA day, sponsored by DraftKings. Premium subscribers can now download the 2015 Weighted Means Spreadsheet under the Fantasy tab at the top of this page, or by clicking "manage your profile." Player pages have been updated with these projections, as have team depth charts (with projected standings) and the fantasy team tracker. Allow us to expand on a few details that might be helpful to you.
Why Does PECOTA Hate My Team?
Every year, fantasy owners and fans of teams ask this question, “Why Does PECOTA Hate My Team?” Last year, Deadspin compiled five dozen “(maybe) surprising player projections.” This season, there’s already been a Lineup Card with eight such surprising projections and Sam Miller did some Pebble Hunting to reveal some of the “winners” in the PECOTA pitching projections. This all raises the question of why Baseball Prospectus would keep publishing surprising projections. Shouldn’t these things be getting better with time, as the system is refined and there’s more data?
It would be disingenuous to suggest that projections never miss the mark. Sometimes by a lot. In fact, last season alone, 39 of the 362 position players for which Baseball Prospectus had projected 100 or more plate appearances actually amassed 100 or more plate appearances with very unexpected (to PECOTA) hitting performances. We looked at these players’ WARP-per-600 plate appearances, with FRAA removed (yes, FRAA is important, but it’s projected differently and is—sometimes—much more out of the player’s control than batting stats). Using this metric, 39 players missed by 3.0 or more WARP-per-600. It could almost have been called, “Craig’s List”, as Mr. Allen Craig was the no. 9 culprit with a WARP-per-600 difference of 4.6 … and as those who saw him play for Boston can attest, he was making a strong run to top this list. PECOTA had projected 1.8 WARP-sans-FRAA in 426 PA (March 22nd projections), and he ended up with -1.7 WARP-sans-FRAA in 505 PA. But Dan Uggla took the top honors, falling 5.7 WARP-per-600 short of projections. Steve Pearce was no. 3 and represented the top over-performer, bettering his WARP-per-600 projection by 5.4.
The above examples come from the most stable group of players—batters who were projected to play and who did play. Yet, some of the most surprising projections entering the 2014 season ended up being close to perfect. For example, people who saw A.J. Burnett pitch in 2013 thought PECOTA needed glasses, as it projected Burnett to have one of the 10 largest declines in 2014. It projected his ERA to be 4.24, which, considering the drop in leaguewide offense in 2014, would have been adjusted to 4.14. His FIP in 2014? 4.14. Projections for Bryce Harper and B.J. Upton, tabbed as “(maybe) surprising” in the Deadspin article, proved prescient.. Remember the reaction when Chris Davis had a .289 TAv projection (again March 22)? That number ended up being optimistic (he posted a .271), even though when he was coming off a .358 TAv season virtually everyone thought PECOTA hated the guy.
Seriously, though, PECOTA doesn’t hate any player or anyone’s team. There are no biases in it based on anything but historical track records. For completeness, it should be noted that results such as the examples herein are not just “shrugged off” – both accurate and inaccurate results are processed. So, while some projections are going to be surprising, it’s important to keep in mind that all-in-all, the results have been very accurate over the years (thank you, Nate Silver!).
Everyone who follows baseball at all has probably dabbled in the Baseball Prospectus Team Tracker—the most powerful tool of its kind available. For a reminder of some of the various things Team Tracker can do, both on the Team Tracker pages and elsewhere on the site, please refer back to Feature Focus articles on Team Tracker, Basics and Team Tracker, Advanced. The primary reason it’s being mentioned here is that 2015 PECOTA forecasts are now available. Shown is an actual portion of the Team Tracker page for the hitters on my Scoresheet team. (A team which was much better last season than it had any right to be. I had the second-best record among 24 teams entering the final week of the season and then, um, moving on… ) It can be seen that even for a 24-team league, hard times are likely ahead in 2015, based on PECOTA projections. The excerpt from my Team Tracker display is truncated on the right side as a reminder that there are many other stats which can be selected for the reports—allowing them to be tailored to each owner’s needs.
On March 11, 2004, Nate Silver wrote the following about forecasting.
Call yourself a forecaster and you're sure to get some dirty looks. It's a cultural tradition, at least in the parts of our country that has seasons, to criticize the accuracy of a weather forecast (you call this partly cloudy, Mabel?). Political pundits--you know, the guys in the bowties--are ranked somewhere between child molester and petty thief on the social hierarchy. The stock market analysts that were the toast of the town just a couple of years ago are now seen as charlatans at best, criminals at worst.
PECOTA likes the Astros' chances of producing insane strikeout totals. Also, more importantly, it likes the Astros' chances of scoring copious runs.
If you were on baseball Twitter Sunday, you saw the uproar new commissioner Rob Manfred caused by throwing out the possibility of perhaps thinking about maybe banning the shift to increase offense. Yeah, it doesn’t take much to get baseball Twitter angry. But it did lead to strong discussions about other possible causes of baseball’s scoring problem (and boring problem), such as the enlarged strike zone.
PECOTA likes a few pitchers quite a bit more than it did a year ago. What did those arms do to deserve better projections?
You know who had a really good year last year? Alfonso Alcantara. The guy obviously dedicated himself to the sport, got himself in great condition, learned a new trick or two, and/or was just darned focused like a great athlete should be. I have no idea who Alfonso Alcantara is.
Whose 2014 season, good or bad, had enough of an impact to drastically change how PECOTA views them?
Last year, around this time, I wrote an article here about the players who had most changed PECOTA’s mind in the previous year. The premise of said piece was that it takes extraordinary circumstances for PECOTA to acknowledge that the player had changed. Players don’t usually change much, at least in a year. People don’t usually change much. Last year, around this time, I wrote that article, and right now I’m writing that article, because I don’t usually change much.
Can the uncertainty in a player's projections be projected?
There are two important aspects of prediction. The first concerns the accuracy of the prediction—that is, how close a prediction is to the actual, observed result. The second is uncertainty, which is how sure a forecaster is about his or her projection. These issues are fundamental forecasting concepts, and similarly apply to predictions of the weather, the stock market, or the outcome of tomorrow’s ballgame. At present, only one of these facets of a prediction gets much attention in the world of baseball projections, and that is accuracy. Accuracy is measured by the absolute error, which defines how close, on average, a forecast is to the actual, observed result. Projectionists struggle primarily to minimize this number.
The under-examined facet of prediction that we will address in this article is the uncertainty. Whereas we know that predictions tend to be accurate to within a hundred or so points of OPS, we would also like to know whether we are more or less likely to be wrong on certain players. The uncertainty is often treated as a second-order concern because it is usually more difficult to estimate. However, as we show, it is possible to predict ahead of time which players’ forecasts are more uncertain than others. This concept is important because certain teams may prefer high versus low-risk players—a team with high win expectations (90+ wins) might prefer to reduce risk, whereas a middle-of-the-road team (80-85 wins) would presumably seek risk in order to “get lucky” and reach the postseason.
The error spectrum of projections shows the limitations of analysis, or the progress we can still make.
It’s around the time that projection engines are being tweaked, updated, and improved, in anticipation of the release of new predictions for the coming year. At Baseball Prospectus, Rob McQuown is hard at work ironing out the kinks for this year’s release of PECOTA. Given the present focus on predictions, the time is ripe for a retrospective look at how the projections fared last year.
There’s no better source for a large-scale comparison of projection algorithms than Will Larson’s Baseball Projection Project, which I will use for this article. Larson’s page houses the old predictions of as many different sources as he can get his hands on, including methods as diverse as Steamer, the Fan Projections a FanGraphs, and venerable old Marcel. It’s a rich storehouse of information concerning the ways in which we can fail to foresee baseball.
Hitters and pitchers who've defied their preseason PECOTAs, and players who've changed the projection system's mind.
Among the things a Baseball Prospectus subscriber might like to know, as we approach the midway point of the season, are the names of the players who’ve roundly beaten (or fallen fall short of) their preseason PECOTA projections, and the names of the players who will continue to do so. The first list of names is much easier to provide than the second. In Russell Carleton’s article today, he alludes to some relevantresearch by Mitchel Lichtman, who recently studied the subject of breakouts. Here’s how Russell explains what Lichtman did: