February 8, 2006
Lies, Damned Lies
PECOTA Takes On Prospects, Part One
I’ve spent the better portion of the last two months thinking about how we can do a better job of evaluating the futures of baseball players and, particularly, the futures of young baseball players. It goes without saying that I’ve done this thinking in the vehicle of PECOTA, and essentially every aspect of the system has been rethought and reevaluated. But above and beyond anything PECOTA might have to say, one conclusion has jumped off the page:
When it comes to prospect analysis, a lot of disagreements result from improper valuation, rather than true differences of opinion about how well a particular player plays the game, or how well he is likely to develop.
The Baseball America scouting report on
But exactly how valuable would that sort of performance be? If we accept that performance as gospel, how should we rank Lu among Dodgers prospects? Among shortstop prospects? Among all prospects? Which players would we trade for Chin Hung Lu, and which wouldn’t we?
Bourn offers the quickness, aptitude and offensive approach required of a leadoff hitter. He’s the system’s best defensive outfielder and also has an above-average arm.
Here too, PECOTA largely agrees with BA’s assessment. Bourn’s speed scores are off the charts, and he rated as a +9 in center field last year. It doesn’t think Bourn’s a sure thing--his strikeout rates are high for a player with his profile, and he almost certainly won’t develop home run power. But it gives him about a 25% chance of getting up to an EqA in the .265/.270 range, heavy on the OBP side and coupled with good baserunning, which would in fact make him an adequate leadoff hitter.
But how does Bourn rate compared to
So we have:
Which of these players is the more valuable commodity? The answer to that seems obvious enough, and PECOTA’s various long-term valuation metrics say that Victorino is perhaps two or three times more valuable than Bourn. But BA has Bourn ranked as the #3 player in the Phillies system, whereas Victorino doesn’t crack in their Top 10 (in one of the weakest systems in baseball).
This is what I mean by improper valuation. There’s relatively little difference of opinion about what sort of prospect Bourn is, and what sort of prospect Victorino is; PECOTA doesn’t have Victornio going all
I’m picking on Baseball America because they can afford to be picked on. BA employs dozens of extremely bright and hard-working people who scout thousands of baseball games across the country and have Rolodexes the size of a small Walmart. They’ve earned their position as the most influential voice in the scouting community.
Besides, this sort of problem is just as pervasive among analysts. In fact, it may be more so; BA, at least, has the advantage of primary source information, whereas the rest of us are left to fight for the scraps and some video clips from the Futures Game. And so there’s an awful lot of debate about whether
This is why I think PECOTA has significant potential as a prospect ranking tool. It can consider a large number of factors at the same time. It can translate those factors into some quite tangible notions about player value. And it’s completely immune from groupthink. It’s a primary source, albeit in a very different way than scouting information, and it’s objective.
Just how close PECOTA is to fulfilling that potential I’ll leave for you to decide. But, as a result of the improvements I’ve made to the system, I think it’s finally developed to the point where it can produce a credible ranking of prospects, the kind of ranking that we can put into a time capsule, and come back in ten years and compare against other artifacts of its day. I’ll go without describing all the changes that I’ve made to PECOTA, which are discussed at much length in this year’s book. But the most important fixes, from a prospect evaluation point of view, are as follows:
There are a lot of presentational improvements too. We now provide an explicit forecast for each of a player’s next five seasons: how many home runs is
I’m going to be ranking prospects by combining two metrics from PECOTA’s peak forecasts, which are designed to assess a player’s medium-to-long term value. The first of these is a player’s peak WARP score, and the second is his peak UPSIDE score. The WARP forecast is the more conservative of the two. It’s focused on value above replacement, and therefore gives somewhat more emphasis to certainty and playing time.
UPSIDE, on the other hand, is focused only the possibility that the player develops into an above average major leaguer. It doesn’t care whether a player winds up riding the major league bench, gets stuck in Double-A, becomes the new Luis Rivas, or goes off to Australia to smoke ganja with Ricky Williams. Each of these outcomes is equally undesirable, and UPSIDE recognizes that.
Because UPSIDE is denominated in runs, and WARP is denominated in wins, we use the following formula to combine the two:
Combined Ranking = Peak WARP * 10 + Peak UPSIDE
That is, we assume that ten runs are equal to one win, the intention being to give equal weight to the two metrics.
You may also notice that, for position players, the WARP rating accounts for his projected defensive prowess, whereas the UPSIDE rating does not (although UPSIDE does account for the player’s position). This is an intentioned feature of the combined rating--essentially, we’re giving half-credit to a player’s defensive forecast. I have a lot of trust in the defensive ratings that Clay turns out for minor league players, and the studies we’ve done internally suggest that they are in fact quite good predictors of major league defensive performance. But considering that a large number of minor league players undergo position changes, that they have to contend with field conditions of varying quality, and that defense is hard enough to get a hold of to begin with, this seems like the fairest thing to do.
Caveats and Limitations
Although PECOTA is very good at what it does, it has some limitations:
In keeping with BP tradition, we’re using Rookie of the Year standards to determine which players are classified as prospects and which aren’t. There’s some confusion about what this means; a RoY candidate must have:
I don’t want to promise that we’ll bat 1.000 on these classifications. The roster time information can be hard to track down, and I can understand why Baseball America doesn’t bother with it. But we’ve been reasonably careful, and with the possible exception of
Besides that, I’ll be running supplemental lists of players who have lost their RoY classification, but are (i) no older than 24, or (ii) split time last season between the majors and the minors. Personally, I think the entire distinction between "prospect" and "good young player" is silly, especially when applied to evaluate an organization’s scouting and development capabilities--why punish a team that successfully graduates a player to the big league level? (It also seems that players who make a mid-season debut--guys like
Catchers: Real Prospects
Player WARP Upside Combined 1. Jarrod Saltalamacchia, C, ATL (21) 16.9 137.0 306.4 2. Josh Willingham, C, FLO (27) 15.3 126.2 279.3 3. Kenji Jojima, C, SEA (30) 14.0 94.7 234.9 4. Jeffrey Mathis, C, LAA (23) 14.0 86.3 226.1 5. Michael Napoli, C, LAA (24) 15.3 71.8 224.7 6. Ron Paulino, C, PIT (25) 16.1 59.4 220.4 7. Russell Martin, C, LAN (23) 16.3 52.3 215.4 8. Neil Walker, C, PIT (20) 14.0 60.9 200.6 9. Miguel Montero, C, ARI (22) 11.5 59.2 174.5 10. John Jaso, C, TBA (22) 12.0 49.6 169.2 11. Curtis Thigpen, C, TOR (23) 13.6 28.8 164.4 12. Jeffrey Clement, C, SEA (22) ** 9.2 59.6 152.0 13. Kelly Shoppach, C, CLE (26) 11.2 39.6 151.2 14. Brandon Snyder, C, BAL (19) 10.7 41.6 148.4 15. Taylor Teagarden, C, TEX (22) ** 9.4 39.8 133.4 16. Carlos Ruiz, C, PHI (27) 10.4 20.8 124.9 17. Justin Knoedler, C, SFN (25) 9.8 26.3 123.9 18. Eliezer Alfonzo, C, SFN (27) 9.6 26.8 122.7 19. Kurt Suzuki, C, OAK (22) 9.1 31.1 121.6 20. Francisco Hernandez, C, CHA (20) 9.9 21.3 120.6 21. Christopher Iannetta, C, COL (23) 8.6 30.0 116.1 22. George Kottaras, C, SDN (23) 8.3 28.3 111.0 23. Shawn Riggans, C, TBA (25) 7.6 29.4 105.4 ** Fewer than 150 translated PA Line breaks indicate ratings differences of at least 25 points
I recognize that these numbers, new as they are to all of us, will not mean very much in the abstract. But broadly speaking, it takes a combined rating of about 300 to attain the status of an elite, Top 20 prospect. By that definition, only
I’m fairly comfortable that, on the question of players like Willingham, the scouting side of the aisle does have it wrong. But, to reiterate, the error is not so much in failing to define the commodity appropriately--nobody doubts that Willingham will hit his home runs and draw his walks in the big leagues, and not do much else--but in failing to put it into its proper context. It’s nice to talk about a player’s upside, but by definition, relatively few players will realize their upside. And sometimes, like in Michael Bourn’s case, that upside just isn’t all that exciting to begin with. By contrast, players who have already realized their upside are liable to be underrated, provided that upside means being an average-to-good big league regular, warts and all.
Speaking of warts, it’s hard to identify a player who has more of them than Michael Napoli, a player whose power hitting ability may be even better than Willingham’s, and contact skills may be even worse. Unlike Willingham, however, Napoli is a born-and-bred backstop who fields his position well, and flaws that might end the career of another player can come to be tolerated when a guy plays a legitimate catcher. The $64,000 question is whether they’ll be tolerated by The O.C., which usually prefers eye candy to substance.
Otherwise, this is a pretty boring list. I don’t have any particular feelings about
Catchers: Almost Prospects
Player WARP Upside Combined Joe Mauer, C, MIN (23) 33.9 283.8 622.3 Brian McCann, C, ATL (22) 24.7 210.6 457.5 Dioner Navarro, C, LAN (22) 18.9 131.7 320.2 Ryan Doumit, C, PIT (25) 15.5 120.6 275.2 Yadier Molina, C, SLN (23) 14.7 28.8 175.7 John Buck, C, KCA (25) 11.3 51.1 164.5 Guillermo Quiroz, C, TOR (24) 10.1 56.2 156.8
Before we get on too much of a "where have all the catchers gone" lament, let’s remember the two young players who could represent their leagues in the All-Star game for the next decade to come. It’s rare that PECOTA renders this unambiguously favorable a verdict on a player, but that’s what we get with
Another of the biases that creeps into prospect analysis is what we might call Shiny New Toy Syndrome.
On the docket for next week: outfielders and first basemen. After that, the middle infielders, and the real controversy begins.