September 6, 2007
The Playoff Odds Report
September is the month where the great pennant races happen. In the current three divisions plus a Wild Card set-up, baseball's focus gets placed on the playoff games in October, but it's easy to forget that once upon a time, and for a very long time, the year-long drama of the regular season was often more compelling than what happened during a week or two of World Series action.
One of the key tools we use to look at pennant races probably doesn't need any introduction to many of you: it's the Playoff Odds Report, one of the most popular features on our statistics page. The report is popular in large part because it's so simple-next to each team in the majors we note their place in the real-world standings, then note the percentile chance that they will be division champions, or to take the league's wild card slot, and, cumulatively, their overall chance of making the playoffs. The Playoff Odds take into account the games left in each team's schedule, the strength of the opposition they'll face, and the potential playoff implications of each matchup, and boil it down to a single easy-to-comprehend number. For the St. Louis Cardinals, for example, it's 21.5 percent (21.4 percent chance of winning the division, 0.1 percent chance of being the wild card).
It's handy, but the whole thing does raise a few questions:
How the Playoff Odds Work
Starting at the top, there are four main elements of the Playoff Odds Report. First, there are the actual league standings to date; second, a ranking system based on winning percentage to determine the quality of each team on the field; third, those ranks are regressed to the mean; and fourth, the remaining schedule is simulated one million times by computer, using the regressed rankings to determine the winner of each game remaining in the season. The computer then tallies up how many games each team won in each simulated season, so that we can see the percentage of times each team won the division or wild card.
To understand the Playoff Odds ranking system, it's probably worthwhile to take a step back to talk about some of the basic principles involved. In the early 1980s, Bill James (yeah, him again) had a "eureka" moment, where he noted the relationship between a team's runs scored and runs allowed, and their winning percentage. He found that "the ratio between a team's wins and its losses will be the ratio between the square of their runs scored and the square of their runs allowed." The way this was expressed mathematically recalls a famous bit of basic high school algebra: the Pythagorean Equation. Accordingly, James dubbed this the "Pythagorean approach to won/lost percentage."
Since then, a number of students of the game have looked at the winning percentage/run differential relationship, and refined the Pythagorean approach, notably including Baseball Prospectus's own Clay Davenport. Clay's approach was jokingly dubbed "Pythagenport" by members of our technical staff, and the name eventually stuck. The details are a bit more math-intensive than we usually deal with in this space (although I've been able to hunt up a decent number of links in the "Further Reading" section below, for those of you interested in the math) but regardless of the exact formulation, the Pythagorean approach and its derivatives enable us to calculate expected won/loss records for teams based on their runs scored/allowed. You can see the results of this basic calculation, using Pythagenport, in the Current Adjusted Standings Report, as first order wins and losses (W1/L1). Davenport took things a step further, looking at expected wins and losses using normalized runs scored (EqR) and allowed (EqRA), which he termed second order wins and losses (W2/L2), and then added a strength of opposition adjustment to the normalized figures (AEqR/AEqRA) to produce third order wins and losses (W3/L3). Stripped of the distortions you get from league-wide offensive levels, ballparks, and varying strengths of schedule (a real concern in the unbalanced schedule/interleague play era) third order wins and losses can be considered a truer measure of a team's ability than their actual record-and third order win percentage is used in many places on this website-including the Prospectus Hit List-as the essential measure of team quality.
In the Playoff Odds Report, the third order winning percentage of each team is then regressed toward the mean-in other words, each team's ratings are then adjusted to somewhere between their third order winning percentage and .500. As a result, if a team has been doing extremely well, posting, say, a .650 W3 percentage, or extremely poorly, with a .350 W3 percentage, the Report bets that they won't be able to keep up that pace. In addition to that, the system applies a small randomizing factor.
Then we simulate the games, a million times every morning. Sometimes we're asked things like "I wonder what happened in that simulated season where the Devil Rays won the AL East?" People imagine that we simulate the season using a program like Diamond Mind or the computer version of Strat-o-Matic, and that we can answer with something like "Akinori Iwamura hit .390, Delmon Young smacked 43 bombs, and Scott Kazmir won 23 games." In truth, the simulations don't take place on that level, where we can pull stats and box scores. Instead, they're battles of winning percentages, using Bill James' log5 method for estimating the chances of one team beating another. The regressed, randomized third-order winning percentages are given one final adjustment-.020 is added to the home team, and the same amount is taken from the road team-and the winning percentages are plugged into the log5 equation. So, if the Red Sox, with a winning percentage of .600 after all those adjustments, were playing the Orioles with an adjusted winning percentage of .475, at Camden Yards, the log5 equation spits out a number, in this case 0.585. The computer then goes on to generate a random number between zero and one. If that number's higher than 0.585, the Orioles win, if lower, then the Red Sox win. That process if then repeated a million times for each game left on the schedule, the wins and losses are then tallied, and the number of times each team won the division or the wild card is divided by 10,000, to get the percentile chance of that team making the playoffs.
Three Playoff Odds Reports
Now that you understand how the basic Playoff Odds Report works, we can very quickly discuss the two variant reports we also feature on the Statistics page. The PECOTA version of the report functions by changing the regression scheme of the Playoff Odds Report. Instead of regressing teams' third order winning percentages to .500, the PECOTA version of the report instead regresses each team's winning percentage toward their projected PECOTA winning percentage. Why would we do this? Because sometimes, particularly early in the season, even a team's third order winning percentage might not be indicative of their underlying talent level.
Take the Yankees, for example. On May 30, they were 21-29, eight games under .500, the low point of their season in terms of their actual winning percentage. Thanks to a healthy runs scored/allowed differential, their third order win percentage, regressed for the Playoff Odds Report was .537 at that point-better than their real-life record, but still not as good as the team was expected to be prior to the season. The PECOTA version of the Playoff Odds regressed the Yankees' third order win percentage up toward .580 (the winning percentage that PECOTA predicted for them) rather than down toward .500, so for the PECOTA report, they effectively had a .552 third order win percentage. The difference could be seen in the playoff odds, where the regular report saw the Yankees as having a 10.7 percent chance of making it to the postseason, the PECOTA version of the report was much more optimistic, seeing the Yankees playing in October 18 percent of the time, based on the likelihood that the team would eventually play up to their PECOTA-projected level.
The third variant is the ELO version, which is based on a famous chess ranking system, modified by Nate Silver to be used in baseball. The principle behind ELO is that it ranks a team's quality at a particular moment in time, emphasizing recent performance, and carrying over from one season to the next (there's a series of articles that I link to below that deal with the ELO system in greater detail). In the ELO Playoff Odds Report, ELO ratings are substituted for third-order winning percentage as the ranking system by which the relative strength of the teams is judged. To use the Yankees as an example, again, on May 30 ELO ranked them with a 1538 ELO score (on a scale on which 1500 is a league-average, 81-win team, and 1600 is roughly a 104-win juggernaut), and gave the Bronx Bombers a 15.9 percent chance at the postseason. Throughout the season, ELO has seen the Yankees as more likely to overtake the Red Sox for the division lead than either the regular Playoff Odds report or the PECOTA version-to go back to the May 30 example, ELO gave the Yankees a 4.97 percent chance to take the division, as opposed to a 1.54 percent chance by PECOTA and a 1.24 percent chance by the regular report.
Answers to a Few Frequently Asked Questions
Which report should I use?
One of the features you'll consistently find at Baseball Prospectus is alternate versions of many of our statistics. We don't just give you WARP, we give you three varieties of WARP; we don't just give you WXRL and SNLVAR, we also give you the precursors of each of those stats, some of which eliminate the lineup adjustment, or aren't set against the replacement level. The default rule when looking at BP's stats is that the newer, fancier versions of a given stat tend to be the ones we recommend using.
The regular, old Playoff Odds Report is an exception to that rule. That's not to say that there aren't uses for the other two reports-as mentioned before, the PECOTA version provides a handy counterpoint in cases where a team is under/over-performing relative to expectations, particularly early in the season, and ELO is useful when playing the game of "Who's more now?" But at its core, the Playoff Odds Report is a model of expected performance, based directly on teams' past performance. PECOTA and ELO both fit that description, too, so when you combine one of those two with Playoff Odds, you're now looking at a model of expected performance…based, now, on another performance model. Simpler is better in this case, and the regular Playoff Odds Report-the one that's currently linked on the front page-is the one you want to consult.
Are changes in a team' composition, such as from trades or promotions of big prospects, reflected in the Playoff Odds?
No. Potentially, these types of changes (such as your team picking Bobby Abreu at the trade deadline in return for a bag of stale Milk Duds, or your team bringing up an unhittable prospect in September) could be reflected in a team's PECOTA projected win percentage, but at present, they're not.
You said the system simulates the season a million times. Why are the playoff odds calculated down to eight figures?
Ties. Rather than do tiebreakers, the Playoff Odds Report awards a few hundred-thousandths of a percent to teams who tied for a playoff spot. It seems that two-, three-, and even four-way ties happen pretty often when you play out the season a million times.
What's "statistical elimination?"
It's what we call it when a team hits a zero chance at the postseason on the playoff odds, and stays there. The Devil Rays are the only team technically eliminated from contention for the division title, in either league-and that just happened on Sunday. But judging from the Playoff Odds Reports, seven teams are stick-a-fork-in-'em done for any postseason activity: the Nationals, Marlins, Rangers, White Sox, Royals, and Orioles are the others. The Devil Rays were the first team statistically eliminated this season, hitting a 0.00000 percent chance at the playoffs on July 13, and flatlining for good on July 23.
How did you pinpoint exactly when the Rays hit rock bottom on the Playoff Odds Report?
One fun feature of the Playoff Odds is that if you click any of the team links on the report page, you get a day-by-day rundown of that team's playoff chances throughout the season. It's interesting for seeing things such as when your team's playoff chances peaked, or were at their lowest point, or when your team's third order win percentage or ELO rating was at its highest.
Do you ever run past seasons through the Playoff Odds system?
For a 2005 article, Clay Davenport used the Playoff Odds system to look back at the 1964 National League pennant race, and to trace the Phillies' down-the-stretch collapse that season. More recently, however, Clay adapted the Playoff Odds system to help identify the most exciting pennant races in major league history for our new book, It Ain't Over 'til It's Over. The fourteen great pennant races examined in the book aren't any sort of secret-they're printed right on the dust jacket-but it's less well-known that the Playoff Odds method also identified the ten dullest pennant races in baseball history:
Year League/Division Winner W% 2001 AL West Seattle Mariners .716 1902 NL Pittsburgh Pirates .741 1884 Union League St. Louis Maroons .832 1955 NL Brooklyn Dodgers .641 1939 AL New York Yankees .702 1969 AL East Baltimore Orioles .673 1999 AL Central Cleveland Indians .599 1995 AL Central Cleveland Indians .694 1977 NL West L.A. Dodgers .605 1885 American Association St. Louis Browns .705
As Will Carroll noted last week, the AL division leaders seem to have their races well in hand, but it seems unlikely that any of this year's races would be fit for the all-time yawners list. Seven of the ten leagues mentioned above were won by teams who won more than two-thirds of their games-records that are likely out of reach for any of 2007's champions. The worst winner of one of the boring races above, the 1999 Indians (.599 win percentage), played in a division where no other team was over .500; they eventually won the division by 21 1/2 games. The 1977 Dodgers led nearly wire-to-wire, and never saw their lead go below 7.5 games after July 1.
Baseball Prospectus, It Ain't Over 'til It's Over. Steven Goldman, Ed. (Basic Books, 2007): In the book's introduction, Steven Goldman gives a much more elegant explanation of how the Playoff Odds Report works than I ever could, and explains the selection process and methodology for the best (and worst) pennant races ever.
Tom Tippett, "May the Best Man Win…at Least Some of the Time": This review of the 2002 postseason provides a pretty thorough look at the Bill James Log5 method for predicting the head-to-head matchups based on winning percentage.
Clay Davenport, "How Far They Fell-Putting the Indians' Miss Into Context": A study looking at some of the surest shots in Playoff Odds history who failed to make the postseason, in honor of the 2005 Cleveland Indians.
Clay Davenport, "Probability and Possibility" and "The Playoff Odds Report-The Addition of PECOTA": These two articles deal with the integration of PECOTA into the Playoff Odds scheme, to produce preseason odds and to modify the odds in-season.
Nate Silver, Lies, Damn Lies, "More on ELO": In part two of the ELO saga, the system is used to generate "Eloport" estimated win-loss records; the 30 best teams from 1960-2005, per ELO, are identified.