May 9, 2002
How Sure Is A
Can a "can't-miss prospect" miss?
Will young pitchers really break your heart?
Last year, in a highly entertaining Baseball Prospectus roundtable Joe Sheehan remarked:
I would submit that there are no circumstances in which a pitcher is the best prospect in baseball.
To which Derek Zumsteg replied:
If a pitcher can't be the best prospect in baseball, can he be the #2? Can he be the #3?
It's a valid question. There is plenty of anecdotal evidence to suggest that pitching prospects are more likely to flame out--but how much more likely? Enough to drop them a few notches down on a top prospects list? If so, by how many spots?
The good folks over at Baseball America take a view somewhat different from Sheehan's. Will Lingo, in an article accompanying this year's Top 100 Prospects list, wrote that "position players are safer bets than pitchers"--but BA still rankedJosh Beckett and Mark Prior 1-2. In a chat session later that day, Jim Callis defended the choices of Beckett and Prior on the grounds that "there's no way of telling which pitchers aren't going to make it. In theory, our list represents who we'd take over whom. Pitchers may be shakier than hitters, but we'd still take Josh Beckett or Mark Prior over anyone."
Risk, Reward, and Expectation
The choices of Beckett and Prior for the top two slots can be justified if the rankings are intended to reflect the players' ceilings, the highest level of performance that they can reasonably be expected to reach. It's easier to imagine them turning in Hall of Fame careers than anyone else in this prospect class except maybe Hank Blalock. And the preference for higher-risk, higher-reward players does seem to be common among prospect analysts. Gary Huckabay expressed this view in the BP Roundtable: "I think the upside/risk metric should favor upside."
But how much risk should it take to cancel out an increased possible reward? Dave Pease captured some of this idea, saying, "I also don't really buy not allowing a pitcher to be the top prospect, as long as he's the top prospect when taking an increased injury probability into account." This sort of suggests the idea of taking an average of the future possibilities: a 25% chance of swinging 20 games into the win column (with a 75% chance of career loss due to injury) should be worth about as much as a 100% chance of gaining five wins. You might tinker with the formula and give it to the higher-risk guy on tiebreakers; still, the two players should be considered more or less equivalent as prospects.
But how to evaluate the risk/reward tradeoff? Are we left at the mercy of our own subjective whims?
Career Value and the Historical Top 100
As it happens, we do have a metric that will allow us to study this question with at least some degree of objectivity: Value Over Replacement Player (VORP). Using this tool--developed by BP's Keith Woolner--we ask, "Suppose this player were lost due to injury, and replaced by a readily available player; how much would it hurt his team?"
Woolner's study in Baseball Prospectus 2002 develops a detailed set of standards for replacement level, depending on position (including starter vs. reliever). A good rule of thumb is that replacement level will be about one run per game worse than average; for example, in a five-run-per-game environment, a team of replacement-level players would score about four runs per game and give up about six. With some further calculations, we find that a league-average regular will contribute two wins per season compared to a replacement-level player. A borderline All-Star will typically be worth about five wins, an MVP candidate eight to ten, and a season for the ages (such as Babe Ruth's 1920, or Barry Bonds's 2001) around 15.
Using this, we can find an objective comparison of hitting and pitching prospects by looking in Baseball America's historical Top 100 lists, and asking questions like, "what's the average career VORP for a hitter ranking in the top 10?" "How about a pitcher in the top 10?" "How do they compare?"
If we find that the average is about the same for pitchers as for hitters, then we can conclude that the risk/reward tradeoff is working as it should. If not, then we will have a reasonably objective basis for suspecting that BA may be overrating one group relative to the other.
For this study, I estimated career VORP for the BA's top 100 prospects from 1990 through 1997, that is, those who have had at least five years to prove themselves. I used the rule of thumb that 10 runs of value moves one game into the win column. This is what I found:
Ranking Hitters Pitchers 1-10 18.2 6.5 11-25 10.9 5.8 26-50 8.3 5.2 51-75 5.2 4.4 76-100 4.9 3.7
What this tells us is that an average pitcher in the top 10 contributes scarcely a third as much as an average hitter in that group. It tells us that pitchers in the top 50 are only slightly better than hitters in the second 50.
Not only do hitters ranked in the top ten include superstars like Alex Rodriguez, Chipper Jones, Nomar Garciaparra, Derek Jeter, and Manny Ramirez, but those guys are backed up by a host of players like John Olerud, Tim Salmon, Greg Vaughn, and Mo Vaughn, all who have been above-average ballplayers for several years. The mediocrities in the group (Wil Cordero, Alex S. Gonzalez) and flops (Eric Anthony, Ruben Rivera) are but a small minority in a star-studded group.
Among pitchers ranked 1-10, only Pedro Martinez has become a superstar. A few other guys have had some success (Steve Avery, Ben McDonald, Arthur Rhodes), but even those are the exception rather than the rule. The career success of pitchers ranked 1-10 is a better match for hitters ranked 41-50 or so (that is, about the 25th through 30th-ranked hitters in their prospect classes).
What then? Should we answer Zumsteg's question by saying that no pitcher should be ranked in the top 25?
Not quite. Let's dig a little deeper.
Sometimes we hear that some groups of pitchers may be at higher risk than others: college versus high school, say, or perhaps lefties versus righties. Let's try a college/high school split, and also separate out pitchers from south of the border:
Ranking College High School Foreign 1-25 9.0 3.5 9.6 26-50 7.9 3.2 5.9 51-75 6.1 2.2 6.0 76-100 3.8 3.5 3.8
(For this, I didn't separate out the top 10; the samples were small enough already. In particular, the Foreign group has only 11 to 13 players in each of its four subgroups, so that Pedro Martinez--he of about 53 wins of career value, a marginal Hall of Fame total already--distorts the numbers heavily when he checks in, once at #10 and once at #62. Also, for some players who have been out of the game for a while, I had to make my best guess as to whether they went to college or not.)
Now we're getting somewhere. Not only does the collegiate group come out on top, but the high-school group consistently ranks worse even than their foreign brethren. This is surprising, because these pitchers--in this study, all from Latin American countries--face the rigors of professional ball from age 18 or even younger, so that we wouldn't expect them to have any less arm strain than North American high-school kids. It could be a small-sample effect, although the Latins still come out better even if you take Martinez out of the equation, so I suspect that the difference is real.
Indeed, the results for high-school pitchers are stunningly poor. Even those ranked in the top 10 have contributed an average of only 3.6 games to their teams' win columns. Not even the best of the high school pitching prospects perform better than the hitters at the bottom of the top 100.
What about a possible lefty/righty split?
Type Value College RHP 6.9 College LHP 4.1 Foreign LHP 7.4 Foreign RHP 5.9 High-School LHP 3.2 High-School RHP 3.1
I'm not sure about the significance of the college lefty/righty split; if anything, I would have expected more of a difference among high schoolers. Perhaps it's just the random effect of a small sample; Andy Pettitte was the only really good (20+ games career VORP) college lefty among 35 such pitchers in the study, compared with Mike Mussina, Kevin Appier, Darryl Kile, Charles Nagy, Roberto Hernandez, and Ben McDonald among 92 guys pitching from the right side. A couple more good lefties would have balanced things a bit. Let's call this one undecided for now.
The foreign left-hander group contains only eight pitchers, including Wilson Alvarez twice, and he accounts for almost the entire total by himself; let's not try to draw any conclusions there.
The Trouble with High-School Pitchers
Now let's return to the problem of why high-school pitching prospects perform so poorly at the major-league level. If the problem were overworking their arms in the pros, we'd expect that to affect the foreign pitchers just as much.
My current theory--only a theory, mind you--is that these results are driven by the fact that college pitchers are a self-selecting group. A talented pitcher who's also a good student is more likely to go to college; a guy with a golden arm but less interest in book-learning will usually prefer to go straight to the pros. This would, of course, be much less relevant to Latin American pitchers, who often have little or no educational opportunity; no matter how smart a Latin pitcher is, signing the pro contract will almost always be his best career option. Pedro Martinez never pitched in college, but by all accounts he's one of the smartest pitchers since Christy Mathewson.
I suspect that the fallout rate for pitchers, especially those drafted out of high school, may be more than just the matter of injury risk. There may also be a substantial risk that the pitcher won't master the mental game of pitching at the big-league level.
It's also possible that self-discipline may be a factor. A guy who will face three years of college may be more likely to have the discipline to keep himself on the sort of rigorous physical conditioning program needed for long-term major-league pitching success (Roger Clemens is the current leading example). A guy who's in a hurry to sign a contract may be less likely to put in the long-term effort needed to reach his career goals.
I'm not saying that high-school pitching draftees are necessarily lazy idiots--Greg Maddux being an obvious counterexample. It's just a matter of general tendencies. One interesting study would be to see if pitchers who signed out of high school in spite of good grades and test scores do any better in the majors than pitchers who were poor students. Good luck getting the necessary data for that one.
Back to the Original Question
Zumsteg asked in last year's BP Roundtable: "If a pitcher can't be the best prospect in baseball, can he be the #2? Can he be the #3?" Let's see now if we have enough data to answer that.
We know that the average hitter in BA's top 10 from 1990-97 has produced about 18 games of VORP in his career. The prospect lists tend to contain about 60% hitters and 40% pitchers, so this is equivalent to about the top six hitters per year. To rank the best of the pitchers in the top six or so, therefore, would require being able to identify pitchers that can be expected to produce at the same level (on average).
It's clear that we cannot do that. Even if we limit our scope to the best of the college pitchers, we'll have a hard time identifying pitchers expected to produce more than 10 or 12 games of value. The two college guys rated at #2 during those years, Ben McDonald and Paul Wilson, average out to a VORP of 12 (McDonald scores at 23, but Wilson, so far, only 1). The only other two college pitchers ranked in the top 10 were Alan Benes and Kris Benson, who don't pull up the average any.
Let's suppose, hypothetically, that Mark Prior can be counted on for 12 games added to the win column over the next several years (halfway between McDonald and Wilson). Where, then, would he rank?
Not in the top five, certainly. Even if the top five hitting prospects (let's say Hank Blalock, Sean Burroughs, Carlos Pena,Nick Johnson, and Joe Borchard) don't all become A-Rod-type superstars, they're still very likely to have John Olerud-type careers with several years above average. With Prior, there's a very heavy risk of not even reaching that level. Even if a hitter and a pitcher have similar ceilings, a hitter has a much better chance of achieving it--or even of just becoming a good ballplayer--than a pitcher does.
Let's look again at the performance of the top hitting prospects:
Ranking Avg. # of Hitters in Group VORP 1-10 7 18.2 11-20 5 12.9 21-30 6 9.3
A top collegiate pitcher like Prior would then seem to come in at around the bottom of the second group, #10 or so (depending on the quality of the top hitting prospects). Even in the best possible case among college pitching prospects--say, Roger Clemens in 1984--I wouldn't put him in the top five. Certainly Clemens's performance since then would have more than justified the ranking; but in 1984, there would have been too much room for doubt, too much that could have gone wrong.
What about Josh Beckett? Well, he's a very talented pitcher. He's also a pitcher out of high school, and one with an injury history already (although he seems to be recovered at this point). Most of what can be said for Beckett was also said for Rick Ankiel a couple years ago; and that's not what's being said about Ankiel now.
Certainly Beckett could turn in a Hall of Fame career. So could Todd Van Poppel have, or Brien Taylor or Roger Salkeld orMatt White. Maybe Kerry Wood will; we'll see if his elbow holds out. Even so, the risk is very, very high. My estimated future value for Beckett? I'll go on the high side of the historical evidence--maybe the high-school pitching mega-prospects of the '90's were a bit unlucky--and say perhaps seven or eight games expected value, and a prospect ranking around #30. To put him above that would require finding some clear means of setting him apart from all the failed uberprospects before him; and at this point, I don't see such an argument.
What now? Should we say, along with Sheehan, that "the phrase 'spectacular pitching prospect' is an oxymoron"?
I, for one, would agree very strongly. The data certainly support it, at least based on eight years of BA rankings. There's just too much that can go wrong with a pitcher, and if that's true for pitchers in general, it's doubly or triply true for pitchers signed out of North American high schools.
This raises the question: How can a team build a pitching staff? One way, of course, is to spend for veterans; if you can trade for an impending free agent like Roger Clemens and sign him to an extension, and then go get Mike Mussina when he hits the market, that will take a lot of the uncertainty out of the process.
But what can a more budget-limited franchise do? Build a young-guns rotation through the farm system? That's a risky proposition. It seems to be working for the A's, who have Tim Hudson, Barry Zito, and Mark Mulder (all college draftees, you'll notice), but Ryan Anderson, Freddy Garcia, and Gil Meche looks like about one out of three at this point for the Mariners. Paul Wilson, Bill Pulsipher, and Jason Isringhausen never became what the Mets hoped they would, and the A's have nothing to show for their "four aces" draft of Van Poppel, Kirk Dressendorfer, Chris Peters, and Dave Zancanaro.
However, even if Ryan Anderson shouldn't have been rated in the top 30 last year, much less #1, I still like the Mariners' approach of a pitching-deep farm system. If Anderson gets hurt, maybe Joel Pineiro can step up; if they lose Jeff Heaverlo, they'll see what Rafael Soriano can do; and even if those guys don't work out, there are eight or ten others who realistically could. I dislike spending first-round choices on pitchers (except for the occasional college draftee); those picks are usually better spent on more-reliable hitters. But I'm all for bulking up on pitchers in the middle rounds.
To return to Jim Callis's example: if I had Josh Beckett on my team, and Callis offered me Blalock or Burroughs or evenWilson Betemit for him, would I do the deal? In a heartbeat.
Among pitching prospects, there is no such thing as a sure thing.
Paul Covert has an advanced math degree, works as an airplane systems reliability engineer, and credits Bill James's books with having taught him how to think. He may be contacted at firstname.lastname@example.org.