Baseball Therapy: Do Bad PECOTA Projections Make Teams Mad?

Last year, the Kansas City Royals were the defending American League champions. They had just won 89 games in 2014, the American League pennant and had come oh so close to winning the 2014 World Series. And no one believed that they were for real. It was hard to find a pundit prediction which had the Royals finishing in third place in the AL Central, much less making the playoffs. If you were paying close attention last season, you know how that one turned out.

Chief among the Royal doubters last year was Baseball Prospectus’ own PECOTA projections. The story that’s gone around was that the Royals were well aware of everyone’s doubts, and in spring training they used that as motivation to prove everyone wrong as a result. How dare those “experts” doubt us!

Of course, if it were that easy, the Baltimore Orioles, coming off of a 96-win season and their own trip to the playoffs, and projected for a measly 78 wins, would have done something a little bolder than win 81 games.

I suppose that this time of year, with pitchers and catchers “reporting” and that marking the first signs of baseball in four months, it’s all part of the great lie we tell ourselves. If your favorite team had a good year last year, they’re ready. If they had a bad year last year, they just need a few things to fall their way. If PECOTA believes in them, well… of course, they’re awesome. If PECOTA hates them, it’ll be the fire in the belly that they need. Fans of all 30 teams are currently lying to themselves that their favorite team will win the World Series because of something like this. Well… 29 are.

Warning! Gory Mathematical Details Ahead!

The good news for PECOTA is that PECOTA is a better predictor of this year’s eventual win total than is last year’s win total (at least for 2009-2015). The bad news is that it only wins by a hair (correlation is .497 for PECOTA’s projections and .481 for last year’s win total). The difficult thing about a correlation around .5 is that it’s big enough that you can’t dismiss it and small enough that there’s a lot of room for randomness in there.

Most of the time, PECOTA agrees that a team will pretty much do what they did last year. Again, from 2009-2015, about half of PECOTA projections were fewer than five games away from what the team had done in the previous year. Then again, five wins can be a very wide range. Still, in these cases, PECOTA and last year’s total are in a virtual tie with each other in their predictive power (.510 vs. .507; I’d tell you which is which, but if you’re asking that question, you completely missed the point of this paragraph.) They correlate about the same because they are both saying the same basic thing.

What happens in the cases where PECOTA and last year’s win totals don’t really agree? I looked at cases where PECOTA predicted that a team would improve over last year’s record by five or more wins. Again, PECOTA and last year’s record actually finished fairly well tied as to who was better at predicting the eventual outcome of the season (.585 vs. .609, with last year’s record coming out on top). The good news is that more than 80 percent of teams whom we called to improve by at least five games actually improved, and more than half (58.3 percent) really did improve by five games or more.

On the flip side, when PECOTA sees a giant drop (five games or more) from last year, nearly three-quarters of those teams did perform worse than they did the year before, but the correlations between both the PECOTA projection (.318) and last year’s win total (.309) were both poor correlates to what they would do this year. It seems that when PECOTA predicts that a team will take a big tumble relative to last year, it’s actually a good sign that no one really knows what’s going to happen.

But I suppose that looking at the teams that PECOTA thinks will be movers and shakers can be a bit misleading. If my favorite team is scheduled to be a 70-win struggler rather than the 75-win struggler that they were last year, should I start counting down the days until 2017 now? Is that the same thing as a 96-win team that’s only being pegged to win 91 this year?

Maybe it’s better to look at tiers of team quality, or at least "quality" as PECOTA sees it. Here are correlations at different PECOTA bands and how well PECOTA does as a predictor of eventual wins vs. last year’s total.

For teams for whom PECOTA projected fewer than 70 wins, PECOTA tracked their eventual win total with a correlation of .553, compared to last year’s win total, which registered only a .263 mark. At the high end, PECOTA reigns supreme too. The problem is that PECOTA had some problems in the middle. PECOTA seems better suited to pick up the teams that are either going to be really good or really bad, even if their previous body of work doesn’t support that conclusion. But still, there’s a lot of room for error in a .418 correlation.

So if we’re being honest, for a lot of teams, PECOTA is a guess. An educated guess, but a guess nonetheless. There’s so much that we don’t know. In fact, one of the reasons that I hate doing pre-season predictions is that even with the best information, at the beginning of the year, it’s almost a guarantee that something weird is going to happen and at the end of the year, you’ll look like an idiot.

Last year, some idiot I know picked the A’s to beat the Nationals in the World Series. Oh wait, that was me.

PECOTA does seem to have some directional validity. If it picks a team to be better than last year, that’s generally the case. If it picks them to be worse, that’s at least a decent guess too. It’s got a good grip on who the really good and really bad teams are, even if those are teams that weren’t great last year. And it makes a complete muddle of the middle.

Did PECOTA rage fuel the Royals?
Maybe it did. The lesson here about PECOTA is that while it’s a very good projection system, it knows so little about what is about to happen. All of us know so little about what is to happen in the next few months. Some of us are just louder about it. One of the issues with PECOTA (and all projection or prediction systems) is that people focus on the eight words that come out of the end (“the Oakland A’s will win the World Series”) and not the massive amount of uncertainty that is behind those eight words. The reality is that there are probably 20 teams where, if you told me that you had zoomed forward into the future and actually seen what had happened and they won the 2016 World Series, I would say “Oh, OK, that makes sense.” It’s the same way with an individual team’s record. The range of “reasonably possible” is a wide span.

But maybe PECOTA rage really is to blame for the Royals' World Series win. It’s easy to dismiss the narrative that a team won because they were fueled by “No one believed in us!” We’ve been conditioned to treat all of that as taurine feces, but there’s something to be said for it. There might be a team that looks at the press early on in spring training, realizes they are getting very little respect from the papers and starts talking about that in the locker room. Whether it’s true or not is irrelevant. If enough people believe it, then the perception becomes reality. Within that frame, no one believes in that team and this is a great insult. If everyone buys into that narrative, though, then you’ve created a mostly closed system where that narrative can reinforce itself. We are all bound together by the fact that we need to prove to everyone that we are better than they think.

My favorite definition of team chemistry is that it’s the answer to the question “Why should I bother?” What’s my motivation? It may seem strange thinking of it in terms of baseball, but I guarantee, reader, that you belong to at least one group that has something even more insane as its central organizing principle. All central organizing principles of all groups are insane when you look at them from the outside, but it doesn’t mean that they don’t have real consequences. If the idea that “we must show them!” causes a pitcher to spend a little extra time prepping for his next start rather than just winging it, then it doesn’t matter whether his belief that everyone is against him is true or not. He’s now a better pitcher. It’s not that his “us against the world” belief is enough to make it to the World Series. Persecution complexes are easy and there have been plenty of teams who felt that they didn’t get any respect, and went 70-92. There has to be some talent and luck that goes your way too, but I think we make a mistake when we push aside the thought that a team also needs some sort of organizing principle to center on during that long, hot season. If it’s the fact that PECOTA gave them no chance back in February, then so be it.

The trope that “no one believed in us and it drove us” is actually a way of saying that “It’s not enough to just be talented. Everyone in this league has talent. Sometimes you just have to be lucky, but sometimes the difference is that some guys just give up after a while. We found an antidote to that.”

So, if any members of the two-time defending American League Champion Kansas City Royals are reading this and perhaps you saw that PECOTA has you guys projected for 77 (yes, 77) wins in the 2016 season, and are already nailing that to the bulletin board in the locker room, it’s actually not true that no one believes in you. We actually have no idea what to believe, at least if we’re being honest with ourselves and taking a sober look at our numbers. We have a crude and cloudy crystal ball and when we try to tune into what’s going to happen in October, we keep getting interference from the golf channel. It’s not that we don’t believe in you. It’s that you made the mistake of believing us. Actually, that’s probably our fault for sounding so self-assured when we put the numbers out there.

Then again, maybe I shouldn’t tell you that. It’ll spoil the motivation. Take PECOTA with a grain of salt. Or perhaps ignore the salt and instead add some yeast to make the loaf rise.