keyboard_arrow_uptop
Baseball Prospectus is looking for a Public Data Services Director. Read the description here.
Image credit: USA Today Sports

The annual spring release of PECOTA‘s projections almost always incites a controversy and this year has been no exception. The most aggrieved fanbase for 2019 belongs to the Chicago Cubs, who PECOTA predicts for a modest 79 wins. More importantly to the 2016-2017 NL Central winners, the projection system puts them in the last place in the division. With the Cubs’ fans, players, coaches, and even president of baseball operations throwing shade at Baseball Prospectus, it’s worth considering what’s going on under PECOTA’s hood, and why the prediction that drives the most scorn—the last-place finish—is also the most difficult part of a projection system’s job.

All projection systems have limits. Anyone who tells you that they have consistently predicted a team’s finish to within one win, as Theo Epstein did in criticizing PECOTA, is either exaggerating or very lucky. There is so much uncertainty in how a baseball season will go, from not knowing the strength of a pitcher’s elbow ligaments to whether a gust of wind will bat down a long fly ball in a pivotal September game. A single injury to an important player is enough to put a team’s playoff hopes to bed. A wicked new pitch might be enough to turn a sub-replacement-level fifth starter into an ace, vaulting an iffy rotation into a good one.

These are possibilities PECOTA can account for probabilistically, but not foretell with certainty. The best any algorithm can do is average over all the many thousands of possible ways a season could unfold, conditional on the knowledge it does have. What PECOTA knows is how each player did in the preceding seasons, as well as how similar players over baseball history have performed. With that information—and that alone—PECOTA generates a prediction about the player’s next year.

Once PECOTA gets a read on each individual player, projecting a team is simply a matter of combining the total contributions of the roster. (This sentence sweeps some of the complexity of assigning playing time under the rug, but that’s a topic for another day.) Conveniently, each player’s contributions can be measured in runs, and by summing the total runs saved and generated, it’s possible to build a prediction for how good a team will be.

Based solely upon the Cubs’ predicted runs scored and runs allowed, PECOTA would peg them as an average team—scoring roughly as many runs as they allow. There’s plenty of uncertainty around that 79-win total. Internal Baseball Prospectus confidence intervals suggest that the Cubs could end up from 73 to 85 wins, even if they are only an average team.

But PECOTA doesn’t stop at assessing team strength—it also gives us a measure of where the team will finish within its division. That’s an exponentially harder problem. Most teams vacillate between 65-105 wins in a typical year, which gives PECOTA one of about 40 reasonable choices for how a team will do. Most of these choices will be absurd for any given team; the Orioles won’t win 105 games, the Yankees won’t win 70.

By contrast, there are 120 possible configurations of finishes in a five-team division (and many more than that in baseball as a whole). Even to predict whether a single team will finish higher than another team in the division requires PECOTA to make twice as many guesses—one for the first team’s win total, and another for the second team. Throw in the fact that teams within the division also play outside the division, and the problem rapidly becomes exponentially more complex than guessing any single team’s true strength.

In a tightly-compacted division, with each team within 10 wins of each other, all configurations are reasonable guesses, unlike the Orioles winning 105 games. A conservative estimate therefore puts the difficulty of predicting division finishes at about three times harder than how a single team will do (120 possible outcomes compared to 40).

The Cubs suffer from playing in what may turn out to be the hardest division in baseball this season. The up-and-coming Brewers, perennially contending Cardinals, surprising Pirates, and new-look Reds form a murderer’s row of competition in the NL Central. In a normal division, the Cubs might be projected for a couple of extra wins and a third-place finish. In the NL Central, that same team might only be good enough for last place.

It’s an interesting thought experiment to consider how this year’s PECOTA rollout would have unfolded if the algorithm had projected the Reds and Pirates to be slightly worse—and the Cubs for third place (even with the same win total). Would the Cubs’ players and coaches have turned PECOTA’s wins forecast into bulletin board fodder? Would senior team officials be trashing the algorithm? I suspect PECOTA would have offended far fewer people by suggesting that the Cubs are mediocre than suggesting that they are a last-place outfit, even if they are both.

There are valid reasons to be skeptical of a team’s projection. PECOTA lacks knowledge on the specifics of each player’s makeup, from their ability to make adjustments to their health. Fans may know more about those factors than a projection system. To PECOTA, any player is only a sequence of numbers, nothing more, nothing less. All the algorithm does is guess the next digits in that sequence.

If you’re a Cubs fan, you may be inclined to credit Kyle Hendricks or Jon Lester (or Kris Bryant, or Yu Darvish) with abilities exceeding PECOTA’s purview. That’s fair; you probably know more than PECOTA does about any particular player on the roster. But what PECOTA knows, and what you likely don’t, is how all of the hundreds and thousands of previous players throughout baseball history who look like Hendricks and Lester did the next year. And what PECOTA does better than the vast majority of humans is synthesize that knowledge into a prediction.

PECOTA has its limits. One of them is forecasting the order of finishes in a tight division. You shouldn’t depend on an algorithm to tell you that one team will win one or two games fewer than a division rival. That’s not what projections are built to do, nor are they capable of doing so accurately. The best they can provide is a rough measure of a team’s strength. For better and for worse, and no matter how much it offends the Cubs themselves, PECOTA sees them as roughly mediocre. The rest—including whether they can chase down the Brewers to claim the division, or make the playoffs at all—is up to the random chance of baseball.

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
roarke
3/08
I have never understood why people get so upset about it. I lived in Kansas City during their playoff runs, when PECOTA kept underestimating them, and people were angry about it. I just can't figure that out - if PECOTA had projected KC to win their division and the World Series, would that have increased the fans' enjoyment of watching the team? Will Cubs' fans not enjoy watching their team as much because PECOTA predicted a mediocre season? I suppose it all relates to the general phenomenon (which is also crazy when you think about it) of people taking their sports fandom very personally, where every criticism of the team is an insult to the fan personally. The tribalism in our society is very strange.
batts40
3/08
I don't speak for all Cub fans, but:
-enabling a domestic abuser
-having a disgusting bigot as an owner
-pretending he's the owner
-not improving the team because we're pretending to be broke

will all make me enjoy the season less possibly. PECOTA? Not so much.
steve161
3/08
There's another reason PECOTA's projections may not be accurate: a team's roster, because of trades, will not be the same at the end of the season as it is in the beginning. Obvious, but not mentioned.

The consistent misses with the Royals are instructive, however. Any projection system is based on a theory of the game, an idea of what leads to wins and losses. PECOTA's theory couldn't account for the Royals. If its developers have asked themselves why that is or made any effort to adjust, I'm not aware of it.

Then there is this: "Internal Baseball Prospectus confidence intervals..." In real science, these are called error bars. Real scientists publish them along with their research, so other scientists and readers can assess just how reliable it is. Why don't sabermetricians?
Travis Leleu
3/08
What makes you think the misses with the Royals were instructive, instead of being random variance?
Robert Arthur
3/08
I think BP has been quite good about publishing error bars for the major statistics. (By the way, as a person with a PhD in a scientific field, confidence interval is a fine term for what we're talking about here.) You can find estimates of the error bars for DRC here, for example:
https://legacy.baseballprospectus.com/sortable/extras/drc_runs.php?run=default2&this_year=2018
(column SD)
Or a good discussion of CIs here:
https://www.baseballprospectus.com/news/article/38289/bayesian-bagging-generate-uncertainty-intervals-catcher-framing-story/
With that said, I do think your suggestion for publishing PECOTA win total confidence intervals is a good one and will pass that along. For what it's worth, it doesn't range much between teams--it's about 6-7 wins for all teams.
Jacob Cook
3/08
"book burning" argument above. Proceed with caution.
Matthew Gold
3/08
It would be interesting to understand why PECOTA has a projection that is so different for what the other major projection systems are showing. Usually they are closer together.
TucsonTumbleweed
3/08
Great explanation of what these projections offer vs what people often think they do. I also have wanted to see the "error bars," like Steve161. BP has provided more than just the weighted mean of player projections for as long as I can remember. Why doesn't BP provide each team's projected range of outcomes, such as the 10th and 90th percentiles at least?
Robert Arthur
3/08
I had no idea people would be interested in the error bars, to be honest. Every team's error bars are between 6-7 wins, if that helps. I will relay on your suggestion.
Pat Folz
3/10
It might be cool to see a histogram of wins or somesuch visualization of the distribution of outcomes from all the simulations. I'd guess the Cubs are much more likely to win 100 games than 60 (or even 70) and the Orioles the opposite - that may or may not be true, but it would be nice to see what the system thinks.

And/or, I've actually wondered this idly since about '03 - why not do percentile projections for each team's wins, like you do with individual players?
markjuliusweinert
3/08
I agree with those who say its silly to get angry about a projection but that does not change the bottom line: PECOTA is garbage. When end results lead to the inevitable conclusion that a system does not work, its time to throw it out and find a better solution.
Anthony Worrall
3/09
I disagree as a professional punter ,I marked Chicago up as an 80 win team with a variance of 2 either way, PECOTA will not be far from the mark, when all is said and done.
sbnirish77
3/11
Predicting that the Pirates might be better than the Cubs might rival the epic miss by BP in predicting Matt Wieters Hall of Fame career.