The answer is they are smart. Come on guys, be serious. Of course they're smart.
Once a year, ESPN’s Jerry Crasnick polls baseball executives on Hot Stove topics. This year Crasnick surveyed “22 general managers, assistant GMs, advisers, scouting directors and talent evaluators in the field for their opinions on seven questions that are likely to drive media coverage,” and the results are characteristically fun: Josh Hamilton to the Brewers! One year and $2 million for Melky! Many other things!
It’s one of my favorite pieces each offseason, and I hope it goes on forever, or at least until I die, because it makes me sad to think that the world will keep going when I’m gone. The hardest part of writing about baseball is the realization that, by not being in the room, I’m getting maybe two percent of the relevant information, which is probably canceled out by the flood of misinformation that does make it to me by virtue of being false and thus having nobody trying to protect it from the public. The guys Crasnick talks to are in the room, and they’re friends with other people in other rooms, and because they’re anonymous there are no obvious incentives for them to lie, so it’s fascinating to read what they believe to be true.
But “believe to be true” is still not necessarily true, and I wondered how much extra information these guys really have. Crasnick has been doing this long enough that a pretty good archive of these pieces is available. Thanks to all those phone calls Crasnick made, we can now review the executives’ predictions from 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, and 2011.
I put every prediction into a spreadsheet, a simplified version of which appears below, a more detailed version of which you can see here. I limited it to predictions that are specific and not vague. For instance, I deemed “Which contract was the worst signed so far” not specific enough, but “Which of (five players listed) will be best next year” specific enough. With one exception, I ignored executives' salary predictions, because of the wide range of responses and because the multivariable aspect of contracts (length + AAV) makes it hard to evaluate each prediction's correctness.
|2003||Who is most likely to be wearing a different uniform in 2004 — Alex Rodriguez, Garciaparra or Griffey Jr.?||Garciaparra||No||1|
|2003||If the price was right, which shortstop would you sign, Miguel Tejada or Kaz Matsui?||Tejada||Yes||2|
|2003||Where will Vladimir Guerrero play in 2004?||Baltimore||No||1|
|2003||Kevin Millwood or Bartolo Colon?||Colon||No||2|
|2004||Where does Adrian Beltre sign, and for how much?||Seattle||Yes||1|
|2004||Will Billy Beane trade Tim Hudson?||Yes||Yes||1|
|2004||Where will he (Hudson) go?||Dodgers||No||1|
|2004||Will Jason Giambi be the Yankees' Opening Day first baseman?||No||No||1|
|2005||Where will Manny Ramirez be playing on Opening Day?||Boston||Yes||1|
|2005||Which team will sign Paul Konerko, and how much will it take?||Chicago||Yes||1|
|2005||Will Roger Clemens return to pitch in 2006?||Yes||Yes||1|
|2005||If money were no object, which premium closer would you rather sign — Billy Wagner or B.J. Ryan?||Wagner||Yes||2|
|2005||Will Rafael Palmeiro return to play in 2006?||No||Yes||1|
|2005||Of Jeff Bagwell, Mike Piazza, Sammy Sosa or Frank Thomas, which player is most likely to have a productive, season?||Thomas most, Sosa least||Yes||2|
|2006||Given a choice between Barry Zito and Daisuke Matsuzaka, which free-agent pitcher do you prefer?||Matsuzaka||yes||2|
|2006||Where do you think Barry Bonds will sign?||Giants||Yes||1|
|2006||Alfonso Soriano or Carlos Lee?||Soriano||Yes||2|
|2006||Where will Roger Clemens be pitching in 2007, or will he retire?||Astros||No||1|
|2006||Which player with a no-trade clause — Manny Ramirez, Alex Rodriguez or Pat Burrell — is most likely to be traded?||Burrell||No||1|
|2006||Which "second tier" starter do you like best: Adam Eaton, Ted Lilly, Gil Meche, Vicente Padilla or Randy Wolf?||Lilly||Yes||2|
|2007||Where do you expect Alex Rodriguez to be playing next year?||Angels||No||1|
|2007||Where do you expect Barry Bonds to be in 2008?||A's or retire||By a small plurality||1|
|2007||Who would you want in center field, Torii Hunter or Andruw Jones?||Hunter||Yes||2|
|2007||What will the Minnesota Twins do with Johan Santana?||Keep and/or extend||No||1|
|2008||Which team will sign left-hander CC Sabathia?||Yankees||Yes||1|
|2008||Which team will sign Manny Ramirez?||Dodgers||Yes||1|
|2008||Will the Padres trade pitcher Jake Peavy?||Yes||No||1|
|2008||Will the Rockies trade outfielder Matt Holliday?||Yes||Yes||1|
|2008||Who would you rather sign to a four-year deal, Derek Lowe or A.J. Burnett?||Lowe||No||2|
|2008||Left fielder Pat Burrell or right fielder Bobby Abreu?||Burrell||No||2|
|2008||Will Ken Griffey Jr. play in 2009? If so, where?||Yes||Yes||1|
|2009||If money and length of contract were no object, which outfielder would you sign: Holliday or Bay?||Holliday||Yes||2|
|2009||Where will John Lackey, the top starting pitcher on the market, sign in the winter?||Yankees||No||1|
|2009||Which free agent do you most expect to be back with the Yankees in 2010 — Pettitte, Damon or Matsui?||Pettitte||Yes||1|
|2009||Where will Aroldis Chapman sign? For how many years and how much money?||Yankees||No||1|
|2009||Which player is most likely to be traded during the winter — Roy Halladay, Adrian Gonzalez or Felix Hernandez?||Gonzalez||No||1|
|2009||Which could pay the most dividends on a one-year contract: J. Smoltz, R. Johnson, K. Griffey Jr., P. Martinez or J. Thome?||Smoltz||No||2|
|2010||Which team will sign Cliff Lee?||Yankees||No||1|
|2010||If you had an outfield opening, which guy would you take: Carl Crawford or Jayson Werth?||Crawford||No||2|
|2010||Do you think the Royals will trade Zack Greinke this winter?||No||No||1|
|2010||Which soon-to-be free agent do you think is more likely to get traded this offseason: Prince Fielder or Adrian Gonzalez?||Fielder||No||1|
|2010||Which has the most left: Lance Berkman, Manny Ramirez, Vlad Guerrero, Hideki Matsui, Jim Thome or Johnny Damon?||Thome||No||2|
|2010||Which pitcher coming off arm problems would you be most interested in taking a shot on? Webb, Young, Bedard or Francis?||Webb||No||2|
|2010||Assuming Jeter signs with the Yankees, what will he get in years and dollars?||3/$55 million||Yes||1|
|2011||Who has the best chance of rebounding next season? Dunn, Crawford or Werth.||Crawford||No||2|
|2011||Which 2011 September-collapse team has a better chance of making the playoffs next year: Boston or Atlanta?||Boston||No||2|
The last column splits the predictions into two categories: 1 for predicting player movement; 2 for predicting player performance. We'll get to that later.
So that gives us one prediction (the Bonds prediction from 2007) that you could argue was right or you could argue was wrong. Of the other 45, there are 21 correct predictions and 24 incorrect predictions. Worse than coin flips!
(Quick aside: this exercise may remind you of Ben Lindbergh's piece last month on predictions that managers, executives, and players made for themselves before the season. It is similar, with two significant differences that I believe advance the discussion. One is that the predictions Ben collected were made publicly and on the record, so the predictors had incentive to lie, spin, or boast. The other is that Ben's piece had zero tables and my piece has two tables.)
Except that most of these aren’t either/or questions; there’s not really a one-in-two chance of picking which team will sign Aroldis Chapman, when there are 30 potential answers. There’s not really a one-in-30 chance, either, as not every team was linked to Chapman, had scouted Chapman, etc. So it’s a bit trickier to compare the executives to random chance.
But if we split these up into categories by how many options there are, then we get a bit more insight:
|Possible answers||#||Correct rate|
|3, 4, 5, or 6||10||30%|
The remaining 15 have an unknown number of true options. But of the questions that have a finite number of possible answers, the executives’ success rate is just a shade better than random chance: 52 percent rather than 50 percent when there were two possible answers; 30 percent instead of the 26 percent random expectation on the three-to-six-answer questions.
Surprisingly, to me, the executives don’t seem to do any better or worse when they’re predicting player performance than when they’re predicting transactions. They were 47 percent successful when asked which player or team would be best; they were 46 percent successful when asked where a player would sign, whether he would be traded, of it he would return the next season.
Obviously I’m going to include a few of the most excellent excerpts. Three:
- "I saw (Kaz) Matsui four years ago, and I thought he was a better player than Ichiro," said an AL GM. (2003)
- "If the Yankees were able to lock Lackey into a rotation with Sabathia and Burnett, it could be the start of another dynasty," a National League assistant GM said. (2009)
- Which player is more difficult to trade — Cubs outfielder Milton Bradley, because of his reputation and other baggage, or Blue Jays outfielder Vernon Wells, because of his contract? Responses: Wells 20, Bradley 0. (2009. Bradley was traded that offseason in a bad-contract swap. Wells was traded for value a year later.)
I think there are two possible ways of explaining this low success rate. One is based on the work of Philip Tetlock, who spent 20 years studying thousands and thousands of pundit predictions. He concluded that expert pundits are barely more predictive than random chance.
First, as the skeptics warned, when hordes of pundits are jostling for the limelight, many are tempted to claim that they know more than they do. Boom and doom pundits are the most reliable over-claimers.
There's a difference here, in that the "experts" we're evaluating aren't in the limelight and have no obvious incentive to over-claim. But it is possible that, because they self-identify as experts, they naturally view and present themselves as more certain than they should. It might also be the case that it is more satisfying to have an interesting opinion (Kaz Matsui > Ichiro; Alex Rodriguez most likely to sign with the Marlins; Melky Cabrera to sign for one year and $2 million) than a conservative but accurate opinion.
The other is that baseball falls under the efficient-market hypothesis:
In finance, the efficient-market hypothesis (EMH) asserts that financial markets are "informationally efficient". In consequence of this, one cannot consistently achieve returns in excess of average market returns on a risk-adjusted basis, given the information available at the time the investment is made.
Crasnick is asking these questions specifically because they are perceived, based on publicly available information, to be difficult to answer with much confidence. He is asking GMs because they are perceived to have more than the publicly available information. But it might just be the case that, when it comes to evaluating major-league players and predicting offseason moves, most of the relevant data is already publicly available and priced into our expectations; that the relevant data that isn't publicly available is not shared between teams; and that, when it is shared between teams, it tends to trickle out to the public, via excellent reporters like Crasnick. In other words: for the purposes of these two types of questions, there's not a significant advantage to being in the room.
Like I said at the beginning, of course GMs are smart. A winning organization depends on so many skills in the front office: contract negotiating, strategizing, scouting for weaknesses, managing risk, motivating players, motivating non-players, generating revenue, working with other teams' front offices, keeping players healthy, developing young players, collecting information, measuring and assessing performance accurately, approving timecards. GMs are, presumably, smart at those things, and many others.
But predicting baseball might just be impossible, and a team that puts too much faith in its own predictions might be falling into a trap. So don't feel bad if you're bad at it. It's not your fault. It's baseball's fault.