CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

Futures Guide 2014 is Now Available in Paperback and Three E-book Formats.

Premium and Super Premium Subscribers Get a 20% Discount at MLB.tv!

<< Previous Article
The Lineup Card: 13 Of... (11/07)
<< Previous Column
Premium Article Pebble Hunting: Balls ... (11/05)
Next Column >>
Pebble Hunting: 17 Tea... (11/09)
Next Article >>
Premium Article Prospects Will Break Y... (11/07)

November 7, 2012

Pebble Hunting

Are GMs Smart or Not Smart?

by Sam Miller

The answer is they are smart. Come on guys, be serious. Of course they're smart. 

But...

***

Once a year, ESPN’s Jerry Crasnick polls baseball executives on Hot Stove topics. This year Crasnick surveyed “22 general managers, assistant GMs, advisers, scouting directors and talent evaluators in the field for their opinions on seven questions that are likely to drive media coverage,” and the results are characteristically fun: Josh Hamilton to the Brewers! One year and $2 million for Melky! Many other things!

It’s one of my favorite pieces each offseason, and I hope it goes on forever, or at least until I die, because it makes me sad to think that the world will keep going when I’m gone. The hardest part of writing about baseball is the realization that, by not being in the room, I’m getting maybe two percent of the relevant information, which is probably canceled out by the flood of misinformation that does make it to me by virtue of being false and thus having nobody trying to protect it from the public. The guys Crasnick talks to are in the room, and they’re friends with other people in other rooms, and because they’re anonymous there are no obvious incentives for them to lie, so it’s fascinating to read what they believe to be true.

But “believe to be true” is still not necessarily true, and I wondered how much extra information these guys really have. Crasnick has been doing this long enough that a pretty good archive of these pieces is available. Thanks to all those phone calls Crasnick made, we can now review the executives’ predictions from 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, and 2011.

I put every prediction into a spreadsheet, a simplified version of which appears below, a more detailed version of which you can see here. I limited it to predictions that are specific and not vague. For instance, I deemed “Which contract was the worst signed so far” not specific enough, but “Which of (five players listed) will be best next year” specific enough. With one exception, I ignored executives' salary predictions, because of the wide range of responses and because the multivariable aspect of contracts (length + AAV) makes it hard to evaluate each prediction's correctness.

Year Question Consensus Correct? Type
2003 Who is most likely to be wearing a different uniform in 2004 -- Alex Rodriguez, Garciaparra or Griffey Jr.? Garciaparra No 1
2003 If the price was right, which shortstop would you sign, Miguel Tejada or Kaz Matsui? Tejada Yes 2
2003 Where will Vladimir Guerrero play in 2004? Baltimore No 1
2003 Kevin Millwood or Bartolo Colon? Colon No 2
2004 Where does Adrian Beltre sign, and for how much? Seattle Yes 1
2004 Will Billy Beane trade Tim Hudson? Yes Yes 1
2004 Where will he (Hudson) go? Dodgers No 1
2004 Will Jason Giambi be the Yankees' Opening Day first baseman? No No 1
2005 Where will Manny Ramirez be playing on Opening Day? Boston Yes 1
2005 Which team will sign Paul Konerko, and how much will it take? Chicago Yes 1
2005 Will Roger Clemens return to pitch in 2006? Yes Yes 1
2005 If money were no object, which premium closer would you rather sign -- Billy Wagner or B.J. Ryan? Wagner Yes 2
2005 Will Rafael Palmeiro return to play in 2006? No Yes 1
2005 Of Jeff Bagwell, Mike Piazza, Sammy Sosa or Frank Thomas, which player is most likely to have a productive, season? Thomas most, Sosa least Yes 2
2006 Given a choice between Barry Zito and Daisuke Matsuzaka, which free-agent pitcher do you prefer? Matsuzaka yes 2
2006 Where do you think Barry Bonds will sign? Giants Yes 1
2006 Alfonso Soriano or Carlos Lee? Soriano Yes 2
2006 Where will Roger Clemens be pitching in 2007, or will he retire? Astros No 1
2006 Which player with a no-trade clause -- Manny Ramirez, Alex Rodriguez or Pat Burrell -- is most likely to be traded? Burrell No 1
2006 Which "second tier" starter do you like best: Adam Eaton, Ted Lilly, Gil Meche, Vicente Padilla or Randy Wolf? Lilly Yes 2
2007 Where do you expect Alex Rodriguez to be playing next year? Angels No 1
2007 Where do you expect Barry Bonds to be in 2008? A's or retire By a small plurality 1
2007 Who would you want in center field, Torii Hunter or Andruw Jones? Hunter Yes 2
2007 What will the Minnesota Twins do with Johan Santana? Keep and/or extend No 1
2008 Which team will sign left-hander CC Sabathia? Yankees Yes 1
2008 Which team will sign Manny Ramirez? Dodgers Yes 1
2008 Will the Padres trade pitcher Jake Peavy? Yes No 1
2008 Will the Rockies trade outfielder Matt Holliday? Yes Yes 1
2008 Who would you rather sign to a four-year deal, Derek Lowe or A.J. Burnett? Lowe No 2
2008 Left fielder Pat Burrell or right fielder Bobby Abreu? Burrell No 2
2008 Will Ken Griffey Jr. play in 2009? If so, where? Yes Yes 1
2009 If money and length of contract were no object, which outfielder would you sign: Holliday or Bay? Holliday Yes 2
2009 Where will John Lackey, the top starting pitcher on the market, sign in the winter? Yankees No 1
2009 Which free agent do you most expect to be back with the Yankees in 2010 -- Pettitte, Damon or Matsui? Pettitte Yes 1
2009 Where will Aroldis Chapman sign? For how many years and how much money? Yankees No 1
2009 Which player is most likely to be traded during the winter -- Roy Halladay, Adrian Gonzalez or Felix Hernandez? Gonzalez No 1
2009 Which could pay the most dividends on a one-year contract: J. Smoltz, R. Johnson, K. Griffey Jr., P. Martinez or J. Thome? Smoltz No 2
2010 Which team will sign Cliff Lee? Yankees No 1
2010 If you had an outfield opening, which guy would you take: Carl Crawford or Jayson Werth? Crawford No 2
2010 Do you think the Royals will trade Zack Greinke this winter? No No 1
2010 Which soon-to-be free agent do you think is more likely to get traded this offseason: Prince Fielder or Adrian Gonzalez? Fielder No 1
2010 Which has the most left: Lance Berkman, Manny Ramirez, Vlad Guerrero, Hideki Matsui, Jim Thome or Johnny Damon? Thome No 2
2010 Which pitcher coming off arm problems would you be most interested in taking a shot on? Webb, Young, Bedard or Francis? Webb No 2
2010 Assuming Jeter signs with the Yankees, what will he get in years and dollars? 3/$55 million Yes 1
2011 Who has the best chance of rebounding next season? Dunn, Crawford or Werth. Crawford No 2
2011 Which 2011 September-collapse team has a better chance of making the playoffs next year: Boston or Atlanta? Boston No 2

The last column splits the predictions into two categories: 1 for predicting player movement; 2 for predicting player performance. We'll get to that later.

So that gives us one prediction (the Bonds prediction from 2007) that you could argue was right or you could argue was wrong. Of the other 45, there are 21 correct predictions and 24 incorrect predictions. Worse than coin flips!

(Quick aside: this exercise may remind you of Ben Lindbergh's piece last month on predictions that managers, executives, and players made for themselves before the season. It is similar, with two significant differences that I believe advance the discussion. One is that the predictions Ben collected were made publicly and on the record, so the predictors had incentive to lie, spin, or boast. The other is that Ben's piece had zero tables and my piece has two tables.)

Except that most of these aren’t either/or questions; there’s not really a one-in-two chance of picking which team will sign Aroldis Chapman, when there are 30 potential answers. There’s not really a one-in-30 chance, either, as not every team was linked to Chapman, had scouted Chapman, etc. So it’s a bit trickier to compare the executives to random chance.

But if we split these up into categories by how many options there are, then we get a bit more insight:

Possible answers # Correct rate
2 21 52%
3, 4, 5, or 6 10 30%

The remaining 15 have an unknown number of true options. But of the questions that have a finite number of possible answers, the executives’ success rate is just a shade better than random chance: 52 percent rather than 50 percent when there were two possible answers; 30 percent instead of the 26 percent random expectation on the three-to-six-answer questions.

Surprisingly, to me, the executives don’t seem to do any better or worse when they’re predicting player performance than when they’re predicting transactions. They were 47 percent successful when asked which player or team would be best; they were 46 percent successful when asked where a player would sign, whether he would be traded, of it he would return the next season.

Obviously I’m going to include a few of the most excellent excerpts. Three:

  • "I saw (Kaz) Matsui four years ago, and I thought he was a better player than Ichiro," said an AL GM. (2003)
  • "If the Yankees were able to lock Lackey into a rotation with Sabathia and Burnett, it could be the start of another dynasty," a National League assistant GM said. (2009)
  • Which player is more difficult to trade -- Cubs outfielder Milton Bradley, because of his reputation and other baggage, or Blue Jays outfielder Vernon Wells, because of his contract? Responses: Wells 20, Bradley 0. (2009. Bradley was traded that offseason in a bad-contract swap. Wells was traded for value a year later.)

***

I think there are two possible ways of explaining this low success rate. One is based on the work of Philip Tetlock, who spent 20 years studying thousands and thousands of pundit predictions. He concluded that expert pundits are barely more predictive than random chance. 

First, as the skeptics warned, when hordes of pundits are jostling for the limelight, many are tempted to claim that they know more than they do. Boom and doom pundits are the most reliable over-claimers.

There's a difference here, in that the "experts" we're evaluating aren't in the limelight and have no obvious incentive to over-claim. But it is possible that, because they self-identify as experts, they naturally view and present themselves as more certain than they should. It might also be the case that it is more satisfying to have an interesting opinion (Kaz Matsui > Ichiro; Alex Rodriguez most likely to sign with the Marlins; Melky Cabrera to sign for one year and $2 million) than a conservative but accurate opinion. 

The other is that baseball falls under the efficient-market hypothesis

In finance, the efficient-market hypothesis (EMH) asserts that financial markets are "informationally efficient". In consequence of this, one cannot consistently achieve returns in excess of average market returns on a risk-adjusted basis, given the information available at the time the investment is made.

Crasnick is asking these questions specifically because they are perceived, based on publicly available information, to be difficult to answer with much confidence. He is asking GMs because they are perceived to have more than the publicly available information. But it might just be the case that, when it comes to evaluating major-league players and predicting offseason moves, most of the relevant data is already publicly available and priced into our expectations; that the relevant data that isn't publicly available is not shared between teams; and that, when it is shared between teams, it tends to trickle out to the public, via excellent reporters like Crasnick. In other words: for the purposes of these two types of questions, there's not a significant advantage to being in the room. 

***

Like I said at the beginning, of course GMs are smart. A winning organization depends on so many skills in the front office: contract negotiating, strategizing, scouting for weaknesses, managing risk, motivating players, motivating non-players, generating revenue, working with other teams' front offices, keeping players healthy, developing young players, collecting information, measuring and assessing performance accurately, approving timecards. GMs are, presumably, smart at those things, and many others.

But predicting baseball might just be impossible, and a team that puts too much faith in its own predictions might be falling into a trap. So don't feel bad if you're bad at it. It's not your fault. It's baseball's fault.

Sam Miller is an author of Baseball Prospectus. 
Click here to see Sam's other articles. You can contact Sam by clicking here

Related Content:  Predictions,  General Managers,  GMs

3 comments have been left for this article.

<< Previous Article
The Lineup Card: 13 Of... (11/07)
<< Previous Column
Premium Article Pebble Hunting: Balls ... (11/05)
Next Column >>
Pebble Hunting: 17 Tea... (11/09)
Next Article >>
Premium Article Prospects Will Break Y... (11/07)

RECENTLY AT BASEBALL PROSPECTUS
Daily League Strategy: Two Servings of Fland...
Fantasy Article My Model Portfolio: Three-and-a-Half Months ...
Notes About Baseball, 7/22
Premium Article What You Need to Know: Boston Gives 'Em A Li...
Premium Article Minor League Update: Games of Monday, July 2...
Premium Article Moonshot: Accounting for Count
Premium Article Prospects Will Break Your Heart: What Did I ...

MORE FROM NOVEMBER 7, 2012
Premium Article Prospects Will Break Your Heart: Chicago Cub...
The Lineup Card: 13 Offseason Needs
Premium Article Sobsequy: Let Me Qualify That
Premium Article Rumor Roundup: Wednesday, November 7
Premium Article Collateral Damage: 2012 Year-End Injury Summ...
Premium Article Transaction Analysis: Rays Go Longish on a R...

MORE BY SAM MILLER
2012-11-09 - Pebble Hunting: 17 Teams That Would Have Won...
2012-11-09 - BP Daily Podcast: Effectively Wild Episode 8...
2012-11-08 - BP Daily Podcast: Effectively Wild Episode 7...
2012-11-07 - Pebble Hunting: Are GMs Smart or Not Smart?
2012-11-07 - BP Daily Podcast: Effectively Wild Episode 7...
2012-11-07 - BP Unfiltered: Jason Bay and Evaluating This...
2012-11-06 - BP Daily Podcast: Effectively Wild Episode 7...
More...

MORE PEBBLE HUNTING
2012-11-14 - Pebble Hunting: The 10 Best Slides: A Slide ...
2012-11-12 - Premium Article Pebble Hunting: How Many MVP Votes Will Ryan...
2012-11-09 - Pebble Hunting: 17 Teams That Would Have Won...
2012-11-07 - Pebble Hunting: Are GMs Smart or Not Smart?
2012-11-05 - Premium Article Pebble Hunting: Balls That Look Like Strikes
2012-11-02 - Premium Article Pebble Hunting: Why Darren Oliver Might Real...
2012-10-31 - Premium Article Pebble Hunting: Tying Up October Loose Ends
More...

INCOMING ARTICLE LINKS
2014-01-09 - BP Unfiltered: YCPB and You Never Could