Sometime today, or thereabouts, we expect to release all 2016 PECOTA projections. As part of that, there is a Royals team projection. And as part of that, there will be rolling of eyes, because, surprise, it is not a very optimistic projection. So in anticipation of the full PECOTA release…
This is the projected win total that nobody wanted. I didn’t want it. I’ve rooted for a lot of baseball outcomes, but this is the first time I’ve rooted for a baseball-projection outcome. It’d be a lot easier for me to spend the next two months answering questions about a Royals-Win-89 forecast than this one. We’ll get into that.
You didn’t want it. You want to believe in these projections. You know that there are mysteries about baseball that we and our little human brains can’t grasp, and you want to believe that those mysteries can be solved by the power of science and math. If we as a species can’t project Wade Davis, how will we as a species ever fix global warming or colonize Mars? We’ll get into that.
Our staff didn’t want it. It’s not just that when we polled our writers for their own Royals predictions—before PECOTA had been run—not one of them went as low as 76. It’s not even that not one of them went lower than 80, or that only one of 27 responses was lower than 85, or that the plurality response was 90, or that the average was 88. It’s this: When I asked a follow-up question a few days later—“If I told you PECOTA projects them to win 76 games this year, does your answer change?”—the response was overwhelmingly “nah.” In fact… counting them out… 73 percent of staff said it didn’t change their answer at all. As one put it: “No, because the projections just seem to not like the Royals.” This seems awfully close to a crisis of confidence. We’ll get into that.
The only people who wanted this projection are probably the Royals themselves, who are chilling on the fourth level of that Gandhi quote. Shredded projections are the confetti in their World Series parades.
If I were going to make the case against projection systems—without doing any research, just a common sense objection—it would be that the formula that misses on a player (or team) one year is in no position to project what they’ll do the next. This isn’t meant as a blanket “bah humbug” to the idea of using events of the past to forecast the future—everybody does that, just in less systematic ways than a system does. But much of the information that goes into a 2014 projection is carried over into a 2015 projection, and then into a 2016 projection. To give a clear, hyperbolic example: Imagine a player was playing without eyes in 2011, 2012, 2013 and 2014. He gets an eye transplant before 2015. PECOTA, with its multi-year memory, is going to a) miss very badly on that guy in 2015, and b) continue to miss until the blind years are out of its memory. Real world instances of this phenomenon surely abound, out of our sight but embedded in the PECOTA “misses.” Long story short: If bad intel got into PECOTA’s 2015 Royals projections, and PECOTA is mostly the same in 2016, and the Royals are mostly the same in 2016, why take the 2016 projections seriously?
The Royals, coming into 2015, had outperformed PECOTA by 21 games in the previous three seasons. Only three teams (Baltimore, Pittsburgh, Oakland) had outdone PECOTA by more. That the Royals then overperformed yet again seems suggestive. That Pittsburgh (one of the teams in the parentheses one line up) was PECOTA’s second-biggest “miss” last year, winning 18 more games than projected, seems suggestive. That the A’s were PECOTA’s third-biggest “miss” would seem suggestive, except that the A’s won 15 fewer games than projected. Fewer. The team PECOTA had been consistently lowest on turned into the team that PECOTA was wrongfully highest on.
If we look at teams’ performances vs. PECOTA over the three year period entering 2015, and the same teams’ performances vs. PECOTA in 2015, there’s a correlation of about .30. That's weak, but still: If PECOTA was low on a team from 2012-2014, it was somewhat more likely to be low on the same team in 2015. But if we extend this to previous years, the correlation dissolves:
- Correlation between 2011-2013 "miss" and 2014 "miss": .07.
- 2010-12 to 2013: -.07
- 2009-11 to 2012: -.06
- 2008-10 to 2011: -.28
So, over a half-decade, there is no tendency for PECOTA to err consistently on the same teams. (Doing the same exercise year-by-year—so comparing a team’s performance vs. PECOTA over a single year to the same measure the following year—shows only slightly better, with a year-to-year correlation under .1.) So, PECOTA either doesn’t “miss”—but, rather, only looks like it misses when the least likely (but still acknowledged possible) outcomes happen—or PECOTA does “miss” but is quite adept at learning from its mistakes the next year.
Which is not to say that PECOTA got the Royals right this year. It’s just to challenge the preemptive claim that PECOTA definitely got them wrong, on account of a Royals blind spot. That’d be a very reasonable hypothesis, but it doesn’t bear out.
So to the Wade Davis problem. I did an autopsy on last year’s 72-win projection (which, hey, nobody gives PECOTA’s later runs credit for bumping it to 73 by the start of the season) for JABO last summer. You can read the whole thing, but here’s the intro and the bullets:
There are three foundations to a projection system: predicting who will play, predicting how well they will play, and converting those individual performances into team wins. Every projection "misses," in the sense that none predict with 100 percent accuracy, but we can see, in five critical projections, how the Royals especially diverted from their team-wide outlook.
- Wade Davis’ short history of dominance made him just about impossible to project algorithmically
- Lorenzo Cain’s unique baseball background caused his late bloom sneak up on us
- Edinson Volquez’s peripherals made a good case for him to be a mediocre pitcher, as he had been for most of his career. He then outperformed his peripherals. (He actually improved his peripherals in the final weeks of the season, after I wrote that piece.)
- Ben Zobrist (and, sort of, Johnny Cueto) were added five months after PECOTA signed its work, and contributed to the Royals’ final record.
- The Royals were super clutch.
Those five (as specific things, and as in-a-nutshell embodiments of the Royals as a whole) actually get you pretty close to the entire gap between projected and actual. There were other things going well—Chris Young and Mike Moustakas and Ryan Madson and so on—but there are always things going well for every team, just as there are things (Greg Holland, Alex Rios, Jeremy Guthrie) going terribly. The five bullets above are the combination of factors—massive breakout performance, good luck, literally unforeseeable roster changes, and (arguably) a projections blindspot—that led to a 23-win whiff.
We made some changes to the PECOTA this year. We’re using cFIP components instead of FRA to project pitchers, which everybody around here is extremely happy about. We’re using our new catcher stats. Outfield assists have been incorporated into FRAA. We’ve shortened PECOTA’s memory for pitchers and are weighting the years differently. We’ve got team defense incorporated into pitchers’ ERAs. These aren’t small changes—well, the outfield assists probably is, but they’re otherwise substantial—and we expect they’ll make PECOTA better and more accurate.
But they won’t help a ton with the bulletpoints. If the Royals are superclutch again, we won’t see it coming. If they trade for two stars in July, it’ll be a surprise. If pitchers outperform their peripherals—even accounting for the team’s defense, now part of their projected ERAs—they’ll outperform the projections.
Cain does get twice as good a projection as he had last year, and if it still reflects heavy regression—well, he’s 30, and it’s not that unreasonable for PECOTA to expect Cain to follow up a Michael-Brantley-in-2014 season with a Michael-Brantley-in-2015 season. Wade Davis—dang, I wish we could get Wade Davis right. This year is better—nearly a full run lower than last year’s projected ERA—but I bet most people would still take the under on a 2.64 ERA from him. Last year, Craig Kimbrel’s projected ERA was 1.33, so it has been possible for a reliever to overcome the small samples and natural volatility of the role and convince PECOTA to put a capital-E Elite projection on him. But just two outstanding years, with "lousy starter" still on his recent resume, aren’t enough to get PECOTA there on Davis. Of course, Kimbrel’s ERA last year was 2.58, so maybe I shouldn’t root for that to change.
So how am I going to defend this when I’m asked about it by radio hosts over the next two months?
First, I’ll note that I had the Royals winning 86 in that poll-the-staff exercise. I’ll tell them that our staff generally feels a lot more optimistic about the Royals’ chances than PECOTA does. I’ll say that most systems like the Royals’ defense more than ours does, and that most observers praise the Royals for things (like a great clubhouse, great advance scouting) that PECOTA doesn’t account for (though, if they’re consistent and valuable factors, they should be showing up in players’ stats and thus influencing their projections; whatever.) I'll allow that the Royals have a pretty mediocre starting rotation, that the bullpen after Wade Davis isn't all technically all that special, and that it's a fun lineup (that provides great defense) but there are three or four holes in it. I might mention that the Royals' deviant projection last year wasn't especially unusual; some team or another is shockingly high or low every year, but a combination of factors set the Royals projection up to be spotlighted when it went wayward. I’ll say that PECOTA follows a series of guidelines that work pretty well in the aggregate, but that there are unknowns that are just as unknown to it as they are to any of us. I’ll say that PECOTA can’t do much more than take an incredibly unpredictable sport and make it a little bit more predictable. It’s not fate. Obviously.
But I’ll also note that, unlike the 73 percent of my staff, seeing PECOTA’s 76 did change my answer. How could it not? It’s absurd to think that I, a guy with a depth chart and a bunch of untested hunches, have any real way of knowing with any precision how many games any team is going to win next year. My answer to a question like this should be extremely malleable; it changed when I saw my staff’s answers to the poll, because they as a group know better than I do, and it changed when I saw PECOTA’s answer, because as a well-designed and constantly evolving system it simply knows more than I do. Maybe those changes end up cancelling each other out, but they definitely change.
And then I’ll probably take the easy cop-out and point out that FanGraphs only projects 79, and Clay Davenport 74. Safety in, er, numbers.
What I probably won’t say, because it’s a complicated thought and I’m not sure how to express it, is that I’m still not totally convinced that PECOTA was exactly wrong last year, or is wrong this year. It doesn’t look great, to be sure, and at this point it’d be a lot easier if PECOTA would just apologize and come join us all over on this side of 81. Clearly, when you're an algorithm that exists to predict the future, the outcome is the grade. But the lesson I’d like to think Nate Silver wanted us all to take from PECOTA—with its percentile projections for players; with its application in playoff odds—is that there are a lot of ways for a season to play out, and we only get to see one of them. I’m happy for BP to take some abuse for last year’s whiff, don’t get me wrong. But isn't it a lot more fun to think of the Royals as the miracle team that defied a bunch of smart projections, rather than to think of the projections as just being dumb?
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.Subscribe now
Having said that, the nature of research is to test, measure, adjust, repeat. Before Bill James, the public were apt to believe all sorts of old saws about how baseball games were won and lost. So the questions revolve around which parts of the current "composition of the team" are being undervalued, in what way, by how much, and most importantly ... what steps can be done to adjust the valuations? This is really the core of Sabermetric analysis. Sam has pointed out some of the adjustments which have been made, with 2016 representing a year of big changes in several aspects of measuring player values ... and hence in projecting future values.
To be honest, that says more about your staff's failings that the failings of Pecota.
I'll invoke one of MGL's rules here with a wager offer:
If ANY of your staff wants to make a wager for any amount of $ (charity or otherwise) where they will take over 85 1/2 and I'll take under 85 1/2, just let me know!
Given that the plurality was 90 (whatever that means) and the average was 88, surely lots of that staff would jump at that wager.
I suspect that none of them will, which means, according to my rule, that they really are full of sh**! ;)
I am full of sh**, though, so I think your rule is still in tact.
I say "staff failing" because the staff at BP is NOT supposed to fall prey to that kind of irrational thinking.
Now, this whole controversy is specifically based upon the Royals stellar record the last 2 years compared to Pecota's projection for those years.
So BP has 2 and only 2 choices: Either include past season's w/l record into the algorithm or ignore them and
stick to your projections.
You can't have it both ways.
I'm glad that someone recognized my point. I knew that I would be ridiculed for my post, which is fine. I am very confident in my statement and my logic that goes with it.
It's a little bit (maybe a lot) like a poker player who has been on a tremendous rush and picks up a bad hand that he should not play (even when considering how other players may change their play against her based on the rush). 95% of the typical players will play the hand fully expecting it to have a positive expectancy. A professional player will not (or should not).
Or a blackjack player who has busted his hand 10 times in a row picks up a 16 versus a 7 and refuses to hit.
I could go on. That is the way it seems to me wrt to the Royals.
Either believe in your projection as the best over/under you can put out there, change it to reflect whatever drives your belief that the Royals projection is wrong, or trash Pecota. I don't see any other ration response.
If it is an otherwise smart and rational person who knows better, the offer to make a substantial wager usually cures whatever ails them at the moment (succumbing to irrational and/or suboptimal human thinking usually because of poor pattern recognition). In this case, I think Kevin is just being...well I can't think of the right word.
But everyone else - most of them know that their 88 or 90 wins is foolish, I believe. If they didn't that simply wouldn't be congruent with their mission at BP.
As soon as they have to put their money where their mouth is (obviously they don't "have to" in this case), that's the moment when logic and rational thought (and everything they work for on this web site) suddenly kicks in. That's the MGL rule.
The combined uncertainty when you layer player projections together into a team projection - coupled with the uncertainty of expertly informed - but still human - playing time estimates is enormous, as you note here. Then add on more uncertainty due to schedule and matchup quirks as two already-uncertainly projected teams meet each other 2430 times throughout the season, and yes, there's a lot of ways observed variation can depart from the expected outcome.
As an example, much was made during the postseason about the Royals' historically high contact rates through their whole lineup, and the ways in which that increased the pressure on opposing pitchers and defenses. Intuitively, this makes sense - it seems at least plausible that a lineup with very high contact rates 1-9 without any "holes" could outperform its OPS relative to a lineup with similar cumulative projections but distributed differently. Does PECOTA even attempt to take such considerations into account?
For instance, here is Hosmer's seasons WARP:
2016: 2.3 (projected)
Hosmer has been consistently inconsistent. If he hits for power, he's good. Also both of his best seasons have had BABIP spike 25-30 points above his career average.
His career average BABIP is .312. In that 1.4 WARP seasons his BABIP was... .312.
Goldschmidt (age 30 season): .313 TAv
Goldschmidt (age 37): .300
Miguel Cabrera (age 34): .305
Miguel Cabrera (age 39): .301
Kolten Wong (age 26): .250
Kolten Wong (age 34): .250
Matt Carpenter (age 32): .274
Matt Carpenter (age 36): .275
Why cFip in PECOTA instead of DRA?
Is there an adjustment made for mid-season sell-offs, like last year's Tigers?
Watching Cueto pitch for the Royals in the regular season, it's hard to argue he positively contributed to the win total.
"DRA is intended to be backward-looking (what should have happened, given what we now know about these players and stadiums) and cFIP is intended to be forward-looking (what we expect to happen in the future, based on the pitcher's own skills). cFIP is more predictive of future run-scoring, and it's difficult to say, in the context of DRA, which runs a pitcher will later "deserve" to be charged with when we don't yet know the context of those future plate appearances. So, we feel that cFIP is the better metric to use for projecting future pitcher runs allowed, and in PECOTA."