keyboard_arrow_uptop

Sometime today, or thereabouts, we expect to release all 2016 PECOTA projections. As part of that, there is a Royals team projection. And as part of that, there will be rolling of eyes, because, surprise, it is not a very optimistic projection. So in anticipation of the full PECOTA release…

Well, cuss.

This is the projected win total that nobody wanted. I didn’t want it. I’ve rooted for a lot of baseball outcomes, but this is the first time I’ve rooted for a baseball-projection outcome. It’d be a lot easier for me to spend the next two months answering questions about a Royals-Win-89 forecast than this one. We’ll get into that.

You didn’t want it. You want to believe in these projections. You know that there are mysteries about baseball that we and our little human brains can’t grasp, and you want to believe that those mysteries can be solved by the power of science and math. If we as a species can’t project Wade Davis, how will we as a species ever fix global warming or colonize Mars? We’ll get into that.

Our staff didn’t want it. It’s not just that when we polled our writers for their own Royals predictions—before PECOTA had been run—not one of them went as low as 76. It’s not even that not one of them went lower than 80, or that only one of 27 responses was lower than 85, or that the plurality response was 90, or that the average was 88. It’s this: When I asked a follow-up question a few days later—“If I told you PECOTA projects them to win 76 games this year, does your answer change?”—the response was overwhelmingly “nah.” In fact… counting them out… 73 percent of staff said it didn’t change their answer at all. As one put it: “No, because the projections just seem to not like the Royals.” This seems awfully close to a crisis of confidence. We’ll get into that.

The only people who wanted this projection are probably the Royals themselves, who are chilling on the fourth level of that Gandhi quote. Shredded projections are the confetti in their World Series parades.

***

If I were going to make the case against projection systems—without doing any research, just a common sense objection—it would be that the formula that misses on a player (or team) one year is in no position to project what they’ll do the next. This isn’t meant as a blanket “bah humbug” to the idea of using events of the past to forecast the future—everybody does that, just in less systematic ways than a system does. But much of the information that goes into a 2014 projection is carried over into a 2015 projection, and then into a 2016 projection. To give a clear, hyperbolic example: Imagine a player was playing without eyes in 2011, 2012, 2013 and 2014. He gets an eye transplant before 2015. PECOTA, with its multi-year memory, is going to a) miss very badly on that guy in 2015, and b) continue to miss until the blind years are out of its memory. Real world instances of this phenomenon surely abound, out of our sight but embedded in the PECOTA “misses.” Long story short: If bad intel got into PECOTA’s 2015 Royals projections, and PECOTA is mostly the same in 2016, and the Royals are mostly the same in 2016, why take the 2016 projections seriously?

Good question!

The Royals, coming into 2015, had outperformed PECOTA by 21 games in the previous three seasons. Only three teams (Baltimore, Pittsburgh, Oakland) had outdone PECOTA by more. That the Royals then overperformed yet again seems suggestive. That Pittsburgh (one of the teams in the parentheses one line up) was PECOTA’s second-biggest “miss” last year, winning 18 more games than projected, seems suggestive. That the A’s were PECOTA’s third-biggest “miss” would seem suggestive, except that the A’s won 15 fewer games than projected. Fewer. The team PECOTA had been consistently lowest on turned into the team that PECOTA was wrongfully highest on.

If we look at teams’ performances vs. PECOTA over the three year period entering 2015, and the same teams’ performances vs. PECOTA in 2015, there’s a correlation of about .30. That's weak, but still: If PECOTA was low on a team from 2012-2014, it was somewhat more likely to be low on the same team in 2015. But if we extend this to previous years, the correlation dissolves:

  • Correlation between 2011-2013 "miss" and 2014 "miss": .07.
  • 2010-12 to 2013: -.07
  • 2009-11 to 2012: -.06
  • 2008-10 to 2011: -.28

So, over a half-decade, there is no tendency for PECOTA to err consistently on the same teams. (Doing the same exercise year-by-year—so comparing a team’s performance vs. PECOTA over a single year to the same measure the following year—shows only slightly better, with a year-to-year correlation under .1.) So, PECOTA either doesn’t “miss”—but, rather, only looks like it misses when the least likely (but still acknowledged possible) outcomes happen—or PECOTA does “miss” but is quite adept at learning from its mistakes the next year.

Which is not to say that PECOTA got the Royals right this year. It’s just to challenge the preemptive claim that PECOTA definitely got them wrong, on account of a Royals blind spot. That’d be a very reasonable hypothesis, but it doesn’t bear out.

***

So to the Wade Davis problem. I did an autopsy on last year’s 72-win projection (which, hey, nobody gives PECOTA’s later runs credit for bumping it to 73 by the start of the season) for JABO last summer. You can read the whole thing, but here’s the intro and the bullets:

There are three foundations to a projection system: predicting who will play, predicting how well they will play, and converting those individual performances into team wins. Every projection "misses," in the sense that none predict with 100 percent accuracy, but we can see, in five critical projections, how the Royals especially diverted from their team-wide outlook.

Those five:

  1. Wade Davis’ short history of dominance made him just about impossible to project algorithmically
  2. Lorenzo Cain’s unique baseball background caused his late bloom sneak up on us
  3. Edinson Volquez’s peripherals made a good case for him to be a mediocre pitcher, as he had been for most of his career. He then outperformed his peripherals. (He actually improved his peripherals in the final weeks of the season, after I wrote that piece.)
  4. Ben Zobrist (and, sort of, Johnny Cueto) were added five months after PECOTA signed its work, and contributed to the Royals’ final record.
  5. The Royals were super clutch.

Those five (as specific things, and as in-a-nutshell embodiments of the Royals as a whole) actually get you pretty close to the entire gap between projected and actual. There were other things going well—Chris Young and Mike Moustakas and Ryan Madson and so on—but there are always things going well for every team, just as there are things (Greg Holland, Alex Rios, Jeremy Guthrie) going terribly. The five bullets above are the combination of factors—massive breakout performance, good luck, literally unforeseeable roster changes, and (arguably) a projections blindspot—that led to a 23-win whiff.

We made some changes to the PECOTA this year. We’re using cFIP components instead of FRA to project pitchers, which everybody around here is extremely happy about. We’re using our new catcher stats. Outfield assists have been incorporated into FRAA. We’ve shortened PECOTA’s memory for pitchers and are weighting the years differently. We’ve got team defense incorporated into pitchers’ ERAs. These aren’t small changes—well, the outfield assists probably is, but they’re otherwise substantial—and we expect they’ll make PECOTA better and more accurate.

But they won’t help a ton with the bulletpoints. If the Royals are superclutch again, we won’t see it coming. If they trade for two stars in July, it’ll be a surprise. If pitchers outperform their peripherals—even accounting for the team’s defense, now part of their projected ERAs—they’ll outperform the projections.

Cain does get twice as good a projection as he had last year, and if it still reflects heavy regression—well, he’s 30, and it’s not that unreasonable for PECOTA to expect Cain to follow up a Michael-Brantley-in-2014 season with a Michael-Brantley-in-2015 season. Wade Davis—dang, I wish we could get Wade Davis right. This year is better—nearly a full run lower than last year’s projected ERA—but I bet most people would still take the under on a 2.64 ERA from him. Last year, Craig Kimbrel’s projected ERA was 1.33, so it has been possible for a reliever to overcome the small samples and natural volatility of the role and convince PECOTA to put a capital-E Elite projection on him. But just two outstanding years, with "lousy starter" still on his recent resume, aren’t enough to get PECOTA there on Davis. Of course, Kimbrel’s ERA last year was 2.58, so maybe I shouldn’t root for that to change.

***

So how am I going to defend this when I’m asked about it by radio hosts over the next two months?

First, I’ll note that I had the Royals winning 86 in that poll-the-staff exercise. I’ll tell them that our staff generally feels a lot more optimistic about the Royals’ chances than PECOTA does. I’ll say that most systems like the Royals’ defense more than ours does, and that most observers praise the Royals for things (like a great clubhouse, great advance scouting) that PECOTA doesn’t account for (though, if they’re consistent and valuable factors, they should be showing up in players’ stats and thus influencing their projections; whatever.) I'll allow that the Royals have a pretty mediocre starting rotation, that the bullpen after Wade Davis isn't all technically all that special, and that it's a fun lineup (that provides great defense) but there are three or four holes in it. I might mention that the Royals' deviant projection last year wasn't especially unusual; some team or another is shockingly high or low every year, but a combination of factors set the Royals projection up to be spotlighted when it went wayward. I’ll say that PECOTA follows a series of guidelines that work pretty well in the aggregate, but that there are unknowns that are just as unknown to it as they are to any of us. I’ll say that PECOTA can’t do much more than take an incredibly unpredictable sport and make it a little bit more predictable. It’s not fate. Obviously.

But I’ll also note that, unlike the 73 percent of my staff, seeing PECOTA’s 76 did change my answer. How could it not? It’s absurd to think that I, a guy with a depth chart and a bunch of untested hunches, have any real way of knowing with any precision how many games any team is going to win next year. My answer to a question like this should be extremely malleable; it changed when I saw my staff’s answers to the poll, because they as a group know better than I do, and it changed when I saw PECOTA’s answer, because as a well-designed and constantly evolving system it simply knows more than I do. Maybe those changes end up cancelling each other out, but they definitely change.

And then I’ll probably take the easy cop-out and point out that FanGraphs only projects 79, and Clay Davenport 74. Safety in, er, numbers.

What I probably won’t say, because it’s a complicated thought and I’m not sure how to express it, is that I’m still not totally convinced that PECOTA was exactly wrong last year, or is wrong this year. It doesn’t look great, to be sure, and at this point it’d be a lot easier if PECOTA would just apologize and come join us all over on this side of 81. Clearly, when you're an algorithm that exists to predict the future, the outcome is the grade. But the lesson I’d like to think Nate Silver wanted us all to take from PECOTA—with its percentile projections for players; with its application in playoff odds—is that there are a lot of ways for a season to play out, and we only get to see one of them. I’m happy for BP to take some abuse for last year’s whiff, don’t get me wrong. But isn't it a lot more fun to think of the Royals as the miracle team that defied a bunch of smart projections, rather than to think of the projections as just being dumb?

You need to be logged in to comment. Login or Subscribe
bryanherr
2/16
It wouldn't be any fun if PECOTA was right all the time.
fantasyking
2/16
Doesn't the math check you ran assume that organizations run and teams build in the same way over a long period? What if PECOTA is wrong about the Royals because the way the team is built right now is undervalued, but the composition of the team--and therefore PECOTA's ability to estimate its performance--was totally different 10 years ago?
lyricalkiller
2/16
Well, I didn't look at correlations between misses 10 years ago and misses now. I looked over four- and two-year periods.
mcquown
2/16
I'm glad you asked this. I would like to assure everyone that we - in a general sense - agree with this intuitive take that there are still aspects of the game which aren't being measured with as much precision as we'd like, and concede that this may lead to biases in certain cases. Having said that, the nature of research is to test, measure, adjust, repeat. Before Bill James, the public were apt to believe all sorts of old saws about how baseball games were won and lost. So the questions revolve around which parts of the current "composition of the team" are being undervalued, in what way, by how much, and most importantly ... what steps can be done to adjust the valuations? This is really the core of Sabermetric analysis. Sam has pointed out some of the adjustments which have been made, with 2016 representing a year of big changes in several aspects of measuring player values ... and hence in projecting future values.
mdlehrman
2/16
As an unabashed Cubs fan who watched almost every game, I can say with little to no reservation that the Cubs were not a 97 win team last year, which is part of the reason they were so fun to watch. You do you, PECOTA.
zbrady
2/16
95.8 Third-Order wins and 94 BaseRun wins say otherwise.
sdsuphilip
2/17
I actually think the projection for royals make plenty of sense, and am hammering the under, the one I'm really questioning is the rays.
Kinanik
2/16
I think your reaction to PECOTA is a good one. Unless your probability that PECOTA is right is 0%, the projection contains information you didn't have before. Even if you're only 20% sure it's right, the logical response is to incorporate that 20%. If you were at 88 and PECOTA says 76, adjusting to 86 makes sense.
Chino5
2/16
Bill Pecota was a Royal. Perhaps PECOTA has a built-in bias against the Royals because the projection formula was named after a journeyman infielder for KC and not after SABERHAGEN or BRETT. Just sayin'.
zbrady
2/16
We might even say that PECOTA is the Bill Pecota of projection systems. (I kid)
lichtman
2/16
"Our staff didn’t want it. It’s not just that when we polled our writers for their own Royals predictions—before PECOTA had been run—not one of them went as low as 76. It’s not even that not one of them went lower than 80, or that only one of 27 responses was lower than 85, or that the plurality response was 90, or that the average was 88. It’s this: When I asked a follow-up question a few days later—“If I told you PECOTA projects them to win 76 games this year, does your answer change?”—the response was overwhelmingly “nah.” In fact… counting them out… 73 percent of staff said it didn’t change their answer at all. As one put it: “No, because the projections just seem to not like the Royals.” This seems awfully close to a crisis of confidence. We’ll get into that." To be honest, that says more about your staff's failings that the failings of Pecota. I'll invoke one of MGL's rules here with a wager offer: If ANY of your staff wants to make a wager for any amount of $ (charity or otherwise) where they will take over 85 1/2 and I'll take under 85 1/2, just let me know! Given that the plurality was 90 (whatever that means) and the average was 88, surely lots of that staff would jump at that wager. I suspect that none of them will, which means, according to my rule, that they really are full of sh**! ;)
TheArtfulDodger
2/16
I'll do it. You pick the cause and a not unreasonable amount. I am full of sh**, though, so I think your rule is still in tact.
lichtman
2/16
Lol. $100 to a charity of the winner's choice?
TheArtfulDodger
2/17
Sure! Let's do it.
markpadden
2/17
Did you just refer to yourself in the third person and then make an Internet bet-threat?
lichtman
2/17
Yes and I don't know what an "internet bet-threat is?" What did I "threaten" to do (if what didn't occur)?
jfcross
2/17
Great response and a great article. I was inclined to respond with something similar although I think we should say "human failing" rather than "staff's falling". We know that PECOTA is better at projecting wins and losses than a knowledgeable observer (without an algorithm) so a rational knowledgeable observer should prefer PECOTA's projection to their own (and move most but not all of way from their own estimate to PECOTA's). That most people don't budge is what requires explanation here.
lichtman
2/17
Exactly. Or at the very least figure out what Pecota is missing since clearly most of the staff think that Pecota is missing something big, at least with respect to the Royals. I say "staff failing" because the staff at BP is NOT supposed to fall prey to that kind of irrational thinking. Now, this whole controversy is specifically based upon the Royals stellar record the last 2 years compared to Pecota's projection for those years. So BP has 2 and only 2 choices: Either include past season's w/l record into the algorithm or ignore them and stick to your projections. You can't have it both ways. I'm glad that someone recognized my point. I knew that I would be ridiculed for my post, which is fine. I am very confident in my statement and my logic that goes with it. It's a little bit (maybe a lot) like a poker player who has been on a tremendous rush and picks up a bad hand that he should not play (even when considering how other players may change their play against her based on the rush). 95% of the typical players will play the hand fully expecting it to have a positive expectancy. A professional player will not (or should not). Or a blackjack player who has busted his hand 10 times in a row picks up a 16 versus a 7 and refuses to hit. I could go on. That is the way it seems to me wrt to the Royals. Either believe in your projection as the best over/under you can put out there, change it to reflect whatever drives your belief that the Royals projection is wrong, or trash Pecota. I don't see any other ration response. If it is an otherwise smart and rational person who knows better, the offer to make a substantial wager usually cures whatever ails them at the moment (succumbing to irrational and/or suboptimal human thinking usually because of poor pattern recognition). In this case, I think Kevin is just being...well I can't think of the right word. But everyone else - most of them know that their 88 or 90 wins is foolish, I believe. If they didn't that simply wouldn't be congruent with their mission at BP. As soon as they have to put their money where their mouth is (obviously they don't "have to" in this case), that's the moment when logic and rational thought (and everything they work for on this web site) suddenly kicks in. That's the MGL rule.
gpurcell
2/17
You have utterly unwarranted confidence in projection systems--PECOTA and all the others--that are currently able to explain perhaps one third of observed variation. You're completely out of line comparing the predictions of those prediction systems with the defined probabilities in a game of blackjack.
mcquown
2/17
I realize we all use the language that "PECOTA predicts" for team outcomes, but - citing wikipedia here - "PECOTA, an acronym for Player Empirical Comparison and Optimization Test Algorithm, is a sabermetric system for forecasting Major League Baseball player performance." And we account for more than a third of observed variance in players. The combined uncertainty when you layer player projections together into a team projection - coupled with the uncertainty of expertly informed - but still human - playing time estimates is enormous, as you note here. Then add on more uncertainty due to schedule and matchup quirks as two already-uncertainly projected teams meet each other 2430 times throughout the season, and yes, there's a lot of ways observed variation can depart from the expected outcome.
lichtman
2/17
You are missing my point. I am not commenting at ALL about the "accuracy" of the Pecota projections.
Maturin
2/16
Following up on the "team composition" question above, does PECOTA, being fundamentally a PLAYER projection system, even attempt to measure the effect of TEAM composition, other than as a sum of individual player projections and resultant calculated run totals? As an example, much was made during the postseason about the Royals' historically high contact rates through their whole lineup, and the ways in which that increased the pressure on opposing pitchers and defenses. Intuitively, this makes sense - it seems at least plausible that a lineup with very high contact rates 1-9 without any "holes" could outperform its OPS relative to a lineup with similar cumulative projections but distributed differently. Does PECOTA even attempt to take such considerations into account?
rudyrosen
2/16
How does PECOTA explain Hosmer going from 3.4 WARP in his age 25 season to a projection of 2.3 WARP in the upcoming season?
Shauncore
2/16
3 wins probably isn't his baseline. For instance, here is Hosmer's seasons WARP: 2012: 0.3 2013: 3.7 2014: 1.4 2015: 3.4 2016: 2.3 (projected) Hosmer has been consistently inconsistent. If he hits for power, he's good. Also both of his best seasons have had BABIP spike 25-30 points above his career average. His career average BABIP is .312. In that 1.4 WARP seasons his BABIP was... .312.
Pernellius
2/17
It's a lazy, "somewhere in between" projection. I think 3 bWAR isn't unreasonable if he's healthy, and is probably the floor of his projection, 4-4.5 representing his ceiling.
maphal
2/17
Man, if the Royals win 75 games this year.....
worldtour
2/17
Where can I download the spreadsheet? Is it up yet? Do tell.
lyricalkiller
2/17
Go to the top of this page; hover over Fantasy tab; and selected PECOTA weighted means spreadsheet. Enjoy!
worldtour
2/17
I have 2015 data. Am I missing something? Is it idiot-proof?
lyricalkiller
2/17
Hmmm. Try this? Go to "manage your profile" page (http://www.baseballprospectus.com/manageprofile.php) and it should be there as "2016 PECOTA Spreadsheet Digital Download"
worldtour
2/17
I have 2012 through 2015, but no 2016. Am I paying for a lower tier subscription?
mcquown
2/17
Responding via email.
markpadden
2/17
No one has presented an update as to how PECOTA works for the last several years. Has it been updated and/or tested at all during this time? Or is it still blindly cranking out whatever Colin Wyers' black box produces?
markpadden
2/17
As per usual, the aging curves are comical: Goldschmidt (age 30 season): .313 TAv Goldschmidt (age 37): .300 Miguel Cabrera (age 34): .305 Miguel Cabrera (age 39): .301 Kolten Wong (age 26): .250 Kolten Wong (age 34): .250 Matt Carpenter (age 32): .274 Matt Carpenter (age 36): .275
Tom9418
2/17
Even in the book, the aging seems to be weird. Guys who are over 33 are predicted for (albeit small) gains from 2016 to 2017. Why would PECOTA predict improvement? Is the park factor being removed in 2017 or is the offensive environment expected to change or ???
Pernellius
2/17
BP seems bearish on pitching development, with steep declines on existing pitchers. Defense also peaks young, so the run scoring environment is skewed lower now and higher in future years.
Pernellius
2/17
Sam, do you see any parallels between the Royals' and Pirates' projections missing? Do elite 'pens, defense give PECOTA projection issues? Why cFip in PECOTA instead of DRA? Is there an adjustment made for mid-season sell-offs, like last year's Tigers? Watching Cueto pitch for the Royals in the regular season, it's hard to argue he positively contributed to the win total. Thanks!
mcquown
2/17
I wanted to answer the DRA question in a way that answered some follow-up questions, as it's a good one - one I considered while working on the revamped pitching projections this season. Here's a quick summation from Jonathan Judge, whose articles can provide a lot more background on the hows and whys (and come highly recommended): "DRA is intended to be backward-looking (what should have happened, given what we now know about these players and stadiums) and cFIP is intended to be forward-looking (what we expect to happen in the future, based on the pitcher's own skills). cFIP is more predictive of future run-scoring, and it's difficult to say, in the context of DRA, which runs a pitcher will later "deserve" to be charged with when we don't yet know the context of those future plate appearances. So, we feel that cFIP is the better metric to use for projecting future pitcher runs allowed, and in PECOTA."
Pernellius
2/17
Thanks. I'm still wrapping my head around DRA, and this is a really great answer.