keyboard_arrow_uptop
BP360 is Back! One low price for a: BP subscription, 2022 Annual, 2022 Futures Guide, choice of shirt

Last year, the Kansas City Royals were the defending American League champions. They had just won 89 games in 2014, the American League pennant and had come oh so close to winning the 2014 World Series. And no one believed that they were for real. It was hard to find a pundit prediction which had the Royals finishing in third place in the AL Central, much less making the playoffs. If you were paying close attention last season, you know how that one turned out.

Chief among the Royal doubters last year was Baseball Prospectus’ own PECOTA projections. The story that’s gone around was that the Royals were well aware of everyone’s doubts, and in spring training they used that as motivation to prove everyone wrong as a result. How dare those “experts” doubt us!

Of course, if it were that easy, the Baltimore Orioles, coming off of a 96-win season and their own trip to the playoffs, and projected for a measly 78 wins, would have done something a little bolder than win 81 games.

I suppose that this time of year, with pitchers and catchers “reporting” and that marking the first signs of baseball in four months, it’s all part of the great lie we tell ourselves. If your favorite team had a good year last year, they’re ready. If they had a bad year last year, they just need a few things to fall their way. If PECOTA believes in them, well… of course, they’re awesome. If PECOTA hates them, it’ll be the fire in the belly that they need. Fans of all 30 teams are currently lying to themselves that their favorite team will win the World Series because of something like this. Well… 29 are.

Warning! Gory Mathematical Details Ahead!

The good news for PECOTA is that PECOTA is a better predictor of this year’s eventual win total than is last year’s win total (at least for 2009-2015). The bad news is that it only wins by a hair (correlation is .497 for PECOTA’s projections and .481 for last year’s win total). The difficult thing about a correlation around .5 is that it’s big enough that you can’t dismiss it and small enough that there’s a lot of room for randomness in there.

Most of the time, PECOTA agrees that a team will pretty much do what they did last year. Again, from 2009-2015, about half of PECOTA projections were fewer than five games away from what the team had done in the previous year. Then again, five wins can be a very wide range. Still, in these cases, PECOTA and last year’s total are in a virtual tie with each other in their predictive power (.510 vs. .507; I’d tell you which is which, but if you’re asking that question, you completely missed the point of this paragraph.) They correlate about the same because they are both saying the same basic thing.

What happens in the cases where PECOTA and last year’s win totals don’t really agree? I looked at cases where PECOTA predicted that a team would improve over last year’s record by five or more wins. Again, PECOTA and last year’s record actually finished fairly well tied as to who was better at predicting the eventual outcome of the season (.585 vs. .609, with last year’s record coming out on top). The good news is that more than 80 percent of teams whom we called to improve by at least five games actually improved, and more than half (58.3 percent) really did improve by five games or more.

On the flip side, when PECOTA sees a giant drop (five games or more) from last year, nearly three-quarters of those teams did perform worse than they did the year before, but the correlations between both the PECOTA projection (.318) and last year’s win total (.309) were both poor correlates to what they would do this year. It seems that when PECOTA predicts that a team will take a big tumble relative to last year, it’s actually a good sign that no one really knows what’s going to happen.

But I suppose that looking at the teams that PECOTA thinks will be movers and shakers can be a bit misleading. If my favorite team is scheduled to be a 70-win struggler rather than the 75-win struggler that they were last year, should I start counting down the days until 2017 now? Is that the same thing as a 96-win team that’s only being pegged to win 91 this year?

Maybe it’s better to look at tiers of team quality, or at least "quality" as PECOTA sees it. Here are correlations at different PECOTA bands and how well PECOTA does as a predictor of eventual wins vs. last year’s total.

PECOTA Projected wins

PECOTA correlation

Last year’s win total correlation

< 70

.553

.245

70-74

.034

.476

75-79

-.022

.301

80-84

.174

.226

85-89

.177

.182

> 90

.418

.247

For teams for whom PECOTA projected fewer than 70 wins, PECOTA tracked their eventual win total with a correlation of .553, compared to last year’s win total, which registered only a .263 mark. At the high end, PECOTA reigns supreme too. The problem is that PECOTA had some problems in the middle. PECOTA seems better suited to pick up the teams that are either going to be really good or really bad, even if their previous body of work doesn’t support that conclusion. But still, there’s a lot of room for error in a .418 correlation.

So if we’re being honest, for a lot of teams, PECOTA is a guess. An educated guess, but a guess nonetheless. There’s so much that we don’t know. In fact, one of the reasons that I hate doing pre-season predictions is that even with the best information, at the beginning of the year, it’s almost a guarantee that something weird is going to happen and at the end of the year, you’ll look like an idiot.

Last year, some idiot I know picked the A’s to beat the Nationals in the World Series. Oh wait, that was me.

PECOTA does seem to have some directional validity. If it picks a team to be better than last year, that’s generally the case. If it picks them to be worse, that’s at least a decent guess too. It’s got a good grip on who the really good and really bad teams are, even if those are teams that weren’t great last year. And it makes a complete muddle of the middle.

Did PECOTA rage fuel the Royals?
Maybe it did. The lesson here about PECOTA is that while it’s a very good projection system, it knows so little about what is about to happen. All of us know so little about what is to happen in the next few months. Some of us are just louder about it. One of the issues with PECOTA (and all projection or prediction systems) is that people focus on the eight words that come out of the end (“the Oakland A’s will win the World Series”) and not the massive amount of uncertainty that is behind those eight words. The reality is that there are probably 20 teams where, if you told me that you had zoomed forward into the future and actually seen what had happened and they won the 2016 World Series, I would say “Oh, OK, that makes sense.” It’s the same way with an individual team’s record. The range of “reasonably possible” is a wide span.

But maybe PECOTA rage really is to blame for the Royals' World Series win. It’s easy to dismiss the narrative that a team won because they were fueled by “No one believed in us!” We’ve been conditioned to treat all of that as taurine feces, but there’s something to be said for it. There might be a team that looks at the press early on in spring training, realizes they are getting very little respect from the papers and starts talking about that in the locker room. Whether it’s true or not is irrelevant. If enough people believe it, then the perception becomes reality. Within that frame, no one believes in that team and this is a great insult. If everyone buys into that narrative, though, then you’ve created a mostly closed system where that narrative can reinforce itself. We are all bound together by the fact that we need to prove to everyone that we are better than they think.

My favorite definition of team chemistry is that it’s the answer to the question “Why should I bother?” What’s my motivation? It may seem strange thinking of it in terms of baseball, but I guarantee, reader, that you belong to at least one group that has something even more insane as its central organizing principle. All central organizing principles of all groups are insane when you look at them from the outside, but it doesn’t mean that they don’t have real consequences. If the idea that “we must show them!” causes a pitcher to spend a little extra time prepping for his next start rather than just winging it, then it doesn’t matter whether his belief that everyone is against him is true or not. He’s now a better pitcher. It’s not that his “us against the world” belief is enough to make it to the World Series. Persecution complexes are easy and there have been plenty of teams who felt that they didn’t get any respect, and went 70-92. There has to be some talent and luck that goes your way too, but I think we make a mistake when we push aside the thought that a team also needs some sort of organizing principle to center on during that long, hot season. If it’s the fact that PECOTA gave them no chance back in February, then so be it.

The trope that “no one believed in us and it drove us” is actually a way of saying that “It’s not enough to just be talented. Everyone in this league has talent. Sometimes you just have to be lucky, but sometimes the difference is that some guys just give up after a while. We found an antidote to that.”

So, if any members of the two-time defending American League Champion Kansas City Royals are reading this and perhaps you saw that PECOTA has you guys projected for 77 (yes, 77) wins in the 2016 season, and are already nailing that to the bulletin board in the locker room, it’s actually not true that no one believes in you. We actually have no idea what to believe, at least if we’re being honest with ourselves and taking a sober look at our numbers. We have a crude and cloudy crystal ball and when we try to tune into what’s going to happen in October, we keep getting interference from the golf channel. It’s not that we don’t believe in you. It’s that you made the mistake of believing us. Actually, that’s probably our fault for sounding so self-assured when we put the numbers out there.

Then again, maybe I shouldn’t tell you that. It’ll spoil the motivation. Take PECOTA with a grain of salt. Or perhaps ignore the salt and instead add some yeast to make the loaf rise.

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
dfrissora
2/16
I enjoy glancing over PECOTA data, but as you wrote in the last sentance, I "take it with a grain of salt." There are several factors besides probability that can cause PECOTA to be off target, such as injuries, impact of new or departed players during the course of the season and players who experience outlier years. Until we can find a crystal ball to foresee these issues, PECOTA will be the best "estimate" we have in out tool box. Come to think of it, when we find that crystal ball, who needs PECOTA?
cmaczkow
2/16
"One of the issues with PECOTA (and all projection or prediction systems) is that people focus on the eight words that come out of the end (“the Oakland A’s will win the World Series”) and not the massive amount of uncertainty that is behind those eight words."

To be fair, the "massive amount of uncertainty" side of the ledger was definitely downplayed for a long time - *cough "deadly accurate" cough cough*. I've been very pleased not to have seen that phrase recently!
therealn0d
2/16
To be fair, "Deadly Accurate" only applies to cases where it is somewhat accurate, or I mean predicted what everyone knew and that actually happened. I mean, they only ever put seven or so examples on the back cover.
sbnirish77
2/17
Where are the team projection numbers?
jfranco77
2/18
Teams that are bad are generally incentivized to be worse. Shedding veteran contracts, playing for next year, outright tanking, etc. This makes a 60-win projection more likely to be right, or at least in the right direction.

Teams that are good are generally incentivized to be better. Trading for veterans, calling up good rookies sooner, etc. This makes an 85-win projection more likely to be right, or at least in the right direction.

Part of the reason the Royals crushed their PECOTA last year was because they traded for Cueto and Zobrist.

PECOTA is particularly bad at projecting 70-79 win teams because they won't know which direction they want to go until the season starts.
jspirk
2/18
Although I agree with your general sentiment in total (winners win, losers lose), I disagree with your assessment of the reason the 2015 Kansas City Royals outplayed their project so prolifically.

You've stated it was because of acquisitions of Cueto and Zobrist that the Royals outpaced the projection: The first appearance by either player for the Royals occurred July 30th as Zobrist got his first start, and the next day Cueto topped the hill against the Blue Jays in a quality start that the Royals lost. Through July 29th (w/o Zobrist & Cueto) the Royals were 61-39 (a 61% winning percentage and a 99 win pace). From there on out the Royals went on to finish 95-67 or 34-28 the rest of the way with Cueto and Zobrist (a 55% winning percentage and a 89 win pace).

The point being is that Royals didn't outpace their PECOTA projections because PECOTA didn't / couldn't foresee the eventual acquisitions of Cueto & Zobrist, they had missed entirely on their projections because they woefully misunderstood the value of the Royal's winning model and did a horrible job of projecting the performance of Eric Hosmer, Mike Moustakas, Lorenzo Cain, Kendrys Morales and Salvador Perez.

The system is and will continue to be broken because it doesn't evolve at the speed of the game. I don't know the inner workings of the system entirely, but I'd venture to guess that the weighting of pitcher performance for starters innings vastly outweighs the very same innings for relievers...as we've seen this winter that doesn't jive with reality and the way that GM's are building their pitching staffs these days. It will continue to be a system that at best guesses on the forward performance of teams based on the prior models of performance and understood relationships between roster construction, player output and wins.
aquavator44
2/20
How can you say the system is broken if you don't know how it works in the first place?

It's really easy to criticize PECOTA after the fact, but the subtext of much of this article is about how the error bars are usually much wider than we would like. It's right there in the first paragraph: "It was hard to find a pundit prediction which had the Royals finishing in third place in the AL Central, much less making the playoffs."

Please, point us to anybody that thought Kendrys Morales would do what he did.