November 9, 2006
We Have a Winner!
Just as I did last year, I'm here to follow up the HACKING MASS Wrap with a look at this year's Predictatron results. This is the second year we've done the Predictatron contest, and it continues to be popular, for obvious reasons--trying to predict the order of finish and teams' eventual records is one of the oldest hobbies of baseball fans.
For those that haven't had the pleasure to compete, Predictatron is the annual contest at Baseball Prospectus where entrants can win $500 by predicting the total wins for each of the 30 major league teams, and the results of the playoffs. Basic scoring is set up so that everyone starts with 1000 points, and you lose points for every win you are off for each team; you can win points back with the playoffs. There are also a few wrinkles, like the Mortal Lock, so I'd encourage everyone to read the full rules.
This year, we didn't have time for the detailed statistics investigation, but we will take a look at the entire pool of Predictatron guesses, the winning ballot, and a comparison to some other preseason predictions. We'll start by going through each division and seeing how the Predictatron guesses picked the playoff teams. To avoid any allegations of falling victim to media bias, we'll traverse the divisions randomly...well, pseudo-randomly, anyway.
By a pretty healthy margin, readers pegged the A's to take the AL West. In fact, the A's had the highest average win guess amongst all MLB teams, just barely higher than the Yankees. Curiously, the A's managed to outperform their third-order wins by a little over 10 wins to reach their total of 93 wins on the year. In instances where ballots didn't favor the A's to win the West, they almost all picked the Angels. The division shook out as expected, with the average guesses for the A's and Rangers basically dead-on, and the Angels and Mariners about three wins off.
Over 90% of ballots correctly picked the Cardinals to take the division, but like most preseason guesses, overestimated their actual wins. Predictatron ballots were optimistic about the Brewers, but were likely disappointed when they only won 75 games--injuries to their pitchers, especially to their frontline starter, Ben Sheets, didn't help. The Cubs vastly underperformed expectations, and didn't come close to the average prediction of 82 wins; they underperformed the average ballot prediction more than any other team. The average ballot pegged the Astros' win total, and were pretty close on the Pirates. The Reds almost made it to .500 this year, overperforming the average ballot by a few wins, even after a late season swoon following their trading two key hitters to Washington.
Most readers liked Cleveland to win the division, despite a tight bunch of three teams at the top. Of course, the division did play out with a relatively tight three-team race, but it was the Tigers in the thick of things, not the Indians. In fact, no team in the AL underperformed expectations as much as the Indians did. Readers picked the Royals for mortal lock 35.4% of the time, the highest for any team, and contestants were rewarded accordingly, as the average ballot was within a win and a half of their actual wins, which is critical for the mortal lock pick. Ballots were also close on the White Sox, but the other three teams' average win guesses were all off by at least 11 wins.
The Yankees and Red Sox tied for the lowest standard deviation of win predictions for all teams, meaning that readers tended to agree about how many games they would win more than any other team. Both teams were picked to make the playoffs the large majority of the time, usually with the Yankees taking the division and the Red Sox the wild card. Of course, the Red Sox disappointed everyone and underperformed by roughly six wins, and also finished third in the division and well out of the wild card race. Usurping second place in the East were the Blue Jays, who finished about four wins ahead of the average prediction. The Orioles and Devil Rays were both content to finish out the set in more typical fashion, although both underperformed expectations, by five wins and 10, respectively.
The NL West seems like it is consistently a little bit zany, and last year was no exception. The Giants and Diamondbacks looked like they might make a run at the division or the wild card, but faded down the stretch, leaving the Dodgers and Padres to duke it out. Predictatron ballots picked the Dodgers by a few wins, but anticipated a tight three-team rumble, similar to the AL Central. The majority of ballots had the Dodgers winning, analogous to the Indians, only this time the ballots got it right, while the Dodgers and Padres both made the playoffs.
This was the most hotly contested division in terms of picking the division winner. We have seen other divisions where the average win guesses for the top teams were very close, especially the AL Central, but in all other cases the leader for division winner was favored by at least 59% of ballots. Here, though, we saw the Mets and Braves basically splitting the bill, with the Phillies picking up some votes as well. We all know now that the Mets cruised for most of the year, overperforming the average predicted win total by almost 10 wins. The Braves stumbled, falling to third behind the Phillies (whose win total was basically pegged by the Predictatron ballots). Nobody saw the Marlins doing as well as they did, and they over shot the projection by about 11 wins. The Nationals' average win guess was basically dead on.
Now that we know how the divisions looked, we can break down how the Predictatron ballots envisioned the playoffs working out. We've already seen that the ballots favored the A's, Cardinals, Indians, Yankees, Dodgers, and Braves to win their divisions. That leaves the wild cards, which shook out as follows:
American League Wild Card
This is basically as expected, as the Red Sox and Yankees have been so strong in recent years that most people expected the wild card to come out of the East. That thinking gave the Red Sox over 40% of the wild card votes, while the Yankees also got almost 30% of the wild card votes, likely from those voters that gave the Red Sox the edge in the division.
National League Wild Card
Similar to the American League, the majority of ballots saw the wild card winner coming out the East. The top two teams on this list came out of the East, obviously, and the Braves also had 23% of the wild card votes, mimicking the Yankees wild-card voting in the American League. To summarize the wild card, 74.6% of wild card votes in the American League came from the East and 75.7% of wild card votes in the National League came from the East, which is pretty astonishing.
Once the entrants completed their predictions for the standings, each balloter was required to predict how the playoffs would proceed. We'll only consider analyzing the playoff ballots for teams who were picked to make the playoffs over 10% of the time. This excludes the random ballot picking the Mariners to win the World Series and its brethren.
Below is a chart describing how the playoff ballots shape up. Before you get too engrossed in the details, allow me to explain what the numbers are showing. The first column is simply what percentage of ballots picked the given team to make the playoffs. Each of the next six columns describe the team's performance in all ballots where it was picked to make it to the given round of the playoffs. Each pair of columns is the team's record and winning percentage in the Division Series (DS), Championship Series (CS), and World Series (WS).
For example, that means that the Padres were picked to make it to the playoffs 158 times; the were picked to go 27-131 in the division series, so they were on the ballot for the Championship Series 27 times. In those 27 chances, ballots picked them to win and advance to the World Series six times. In those trips to the World Series, balloters had them winning zero times.
So, without further ado, let's dig in...
As you peruse the table, here are somethings you might notice and find fascinating:
There are also a few other tidbits and curios which are not shown on the chart:
Like Nate Silver did recently, we'll compare the Predictatron average guesses to the PECOTA projections, the ESPN experts' average predictions, and the Diamond Mind simulation predictions. Below is a listing for all teams, similar to the one that Nate ran in his piece but not broken out by division.
Below is a chart showing how each set of guesses correlates with the actual wins:
As you can see, Predictatron has the closest correlation with the actual wins, narrowly edging out PECOTA. This lends credence to the theories that support the wisdom of crowds. Compared to PECOTA, the Predictatron ballots did especially well with the Angels, White Sox, and Pirates. Since both systems did so well, we can't really say that Predictatron is a superior method to forecast team wins, but it is pretty fascinating that a large group of relatively well-informed guesses could match wits with such a finely-tuned system.
Both Predictatron and HACKING MASS continue to bring entrants back, and we enjoy reviewing and recapping things every year. Thanks to everyone who competed, and we hope you're excited about taking another crack at it next season.