Baseball Prospectus is pleased to announce five new additions to the 2014 player cards, three which have been missing for a few years.
- 10-Year Forecast: A projection of player performance over the next decade.
- UPSIDE: Not all players with the same projection have the same potential for greatness. UPSIDE gives us a way to distinguish between them.
- Comparable Players: The "heart" of PECOTA. The player cards display each player's 100 closest comparables.
- Percentile Forecast: The 10th- through 90th-percentile projections for each player, encompassing the full range of projected performance from worst case to best case.
- Diagnostics: As also seen on the PECOTA weighted-means spreadsheet, each player's Breakout Rate, Improvement Rate, Collapse Rate, MLB%, and Attrition Rate, based on the weighted similarities of his most-comparable players.
The original intended use for UPSIDE and long-term projections was to aid in discerning how various prospects might do during the time when their organization had control over them (or roughly five full-time seasons). And the methodology was based on using actual performance of the most-comparable players over the years in question. There have been some changes to the details over the years, but the core concept has been brought back for 2014, with projections through 2023. Here are some clarifications of what we're doing now:
- Our current definition of PEAK is the one set forth in the Glossary. PEAK doesn't refer to a particular statistic; it refers to a sum total of any given statistic over a contiguous set of five seasons. These five seasons will be the next five for player ages 24 or higher. But for player younger than 24, they will be the most productive five-year window up through and including the age-28 season. For example, either the age 23-27 or age 24-28 span will be used for a for a 23-year-old player, whichever represents the highest sum total of the statistic being measured. Continuing the example, such a player could have a "PEAK FRAA" that is the sum of his FRAA values from his age-23 through age-27 seasons and could also have a "PEAK WARP" that is the sum of his WARP values from his age-24 through age-28 seasons.
- UPSIDE is now a composite of the PEAK values of non-negative WARP for the top 20 most-comparable players (weighted by similarity). As Nate Silver observed in 2006:
UPSIDE…is focused only the possibility that the player develops into an above average major leaguer. It doesn’t care whether a player winds up riding the major league bench, gets stuck in Double-A, becomes the new Luis Rivas, or goes off to Australia to smoke ganja with Ricky Williams. Each of these outcomes is equally undesirable, and UPSIDE recognizes that.
We'll be using UPSIDE to compare PECOTA's top prospects to those of the BP Prospect Staff in an upcoming article series, so stay tuned for that.
- The top 20 most-comparable players are the same ones used for the 2014 PECOTA projections and can include both major-league and minor-league seasons, though players are considered much more comparable to players at similar classifications.
Now for the fun stuff. To get to the new features without scrolling all the way down the player card, simply click on the "More PECOTA" tab located near the middle of the navigation bar:
From there, all the new features follow one after another:
The "PEAK 5" UPSIDE value for Sogard is the sum of his first five UPSIDE scores, as he's beyond the age where the system considers other options.
Last but not least are the most-comparable players based on similarity score. "The comparables," as Colin Wyers put it, "represent a lot of tedious number crunching (measuring Euclidean distance in n-th dimensional space, if you want to be precise)." The "Similarity Index" is based on the Similarity Scores of the top 100 most-comparable players. And the Similarity Scores are based on the Euclidean distances between the player in question (in this case, Sogard in 2014, which will be his age-28 season) and every other player in our database at the same age (for example, Jeff Keppinger in 2008—his age-28 season).
The "Trend" column can be a bit confusing—it's a simple up/down/neutral metric based on whether the comparable player over- or under-performed his baseline projection by 20 percent. For the "baseline" projection, only a generic aging curve is used for comparisons—the system doesn't evaluate the player's projection based on his comparables. Protip: ​Note the easily-overlooked arrow on the bottom right, which allows for the selection of comparable players 11-100.
We hope you enjoy these PECOTA-related offerings. If you have questions about methodology or navigation, please post them in the comments below or email customer service.
Thank you for reading
This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.
Subscribe now
Fine, I'll just go search Mike Trout myself!
- Chris Davis will never hit more than 26 home runs again (and rarely hit more than 20)
- Bryce Harper will never hit more than 27
- Javier Baez will never hit above .240
- Andrew McCutchen will never bat above .287
- Hanley Ramirez will peak at a .272 average
- Billy Hamilton will never steal more than 59 bases
* For Davis, it seems that his comparables had trouble keeping regular playing time, as his HR rate isn't projected to decline much, but his playing time does.
* For Harper, personally, I'd say the 19 HR estimate for 2014 is the "conservative" number here. Believing that his peak projectable (projection peaks will never match actual peaks) HR is eight higher than his 2014 projection seems entirely reasonable to me.
* As a Cubs fan, I've often reviewed Baez projections this offseason. While I agree that his "hit skill" should definitely translate into better batting averages over time, it clearly didn't happen often enough among his comps to bring up his average. We do project him for a very high ISO, at least.
* Batting average is a young player's skill. For 'cutch to never improve it (much) over his 2014 projection is perfectly reasonable. Obviously, statistical variance makes it incredibly likely that he will actually exceed that .287 figure, having that as his projection is quite reasonable.
* See McCutchen comment. Though with Hanley, it's sort of baffling that his comps went down so much in average, given how high his average was recently.
* Hamilton is a strange one - we may need to revisit how we project playing time for guys like him. But we do nothing to project pinch-running steals, and his SB-per-PA is still astonishing (as it should be), considering how infrequently he's projected to get on base by his own means.
I'll investigate him in specific, he was coming out with a higher peak in earlier trials. He is an interesting case study.
I was wondering if you have done any after-the-fact regression analysis (or something similar) to see how valid these type of predictions have been in previous years.
When I did a quick look at players with high breakout rates in 2013, there did not seem to be a high correlation with actual success.
Thanks!
Obviously, you'll tweak and improve the model as time goes on (using 2014 data as it becomes available), but as a result it wouldn't be fair to compare the September 2014 model to 2014 performance. Pre-season snapshots will clearly show your progress to the audience.
However, the "best" match does not necessarily mean it has good predictive capability. It would be interesting to know if these UPSIDE numbers have a high likelihood of predicting future performance or not (and what that likelihood is).
Thanks!
What I mean by test bench evaluations is running the same code against past seasons and looking at exactly the sort of comparisons you are suggesting here for purposes of optimizing the accuracy. Obviously, we strive to make these projections as accurate as possible.
As far as eyeballing 2013 breakout projections versus actual 2014 results, a "breakout" is defined as a season of +20% over established past performance levels, so for a full-time player to have a "break out" season is rather rare.
Please do keep suggesting this sort of thing for content - I'm sure there are plenty of articles in this vein which can be written.
I was looking at Jurickson Profar's FRAA projections, and I see that his FRAA numbers get worse as the percentile increases. That doesn't make sense to me, and could be a bug.
Second, the old projections always seemed conservative, but you had to remember they weren't about a player's ceiling. I just found them useful for comparing players, especially in a keeper league. TY for bringing these back.
I'm not sure when the BP Articles section changed, but it appears that there is now one entry for each author of each article. Is that working as intended?
We're definitely still considering ways to make UPSIDE more easily compared to other stats, such as projected 1-year WARP.
Thanks for all the effort on this. Been waiting for this for a while.
Addison Russell
age 20 (2014): .249 TAv
21: .265
22: .255
23: .249
24: .256
Carlos Correa
19: .245
20: .275
21: .270
22: .271
23: .269
Joc Pederson
22: .269
23: .278
24: .250
25: .253
26: .278
PECOTA is saying all of these guys are essentially major league ready for 2014 (very bullish), but will experience declines in performance from 2015 to 2017 (extremely bearish/bizarre). In addition to the general curve shapes not looking right, there is the issue of lack of sufficient smoothing of the data (see Pederson's predicted roller coaster from age 23 to 26). I had high hopes for the revamped long-term projections, but honestly these numbers do not instill confidence.
As good as year 1 projections might be in a general sense, they are not *that* accurate. And it just gets harder every year further out.
The problem with taking these away is that now that they've come back at a long hiatus people are giving them a more critical look and realizing that there just is not that much value to them, imo.
.260, .262, .258, .295, .255
Or some other weird spike several years out.
That was suggestive of too tight a fit/overfit to similar players (and one of the many reasons a locked model for future evaluation is important instead of just back testing is desirable) or some sort of other bias or problem.
Those projections look fairly reasonable to me if you take them for what they are: the performance of comparable players. Naively, what would you expect from guys knocking on the door? They should have a high chance of flameouts/career ending injuries/general disappointment, shouldn't they? Most prospects are risky. But take a 22 year old who is nearly major league ready, and project him to still be playing 4 years from now, what do you get? The numbers should tick up. Especially for an outfielder, since they get fewer shots to come back. The ones who are around at 26 tend to actually be good.
My problem with the projections, though, is exactly this factor. If there's a 60% chance a guy is putting up a .220 league-adjusted TAv in AAA in 4 years, and a 40% chance he's putting up a .300 TAv in the majors, then it's not really accurate to project him as a .252 TAv, and it won't coincide with scout projections. I know it's nice to have a single number, but in this case the average just really doesn't convey useful information. Upside seems like a better approach.
Maybe for a single number it would be better to use the median value among the comps? Weighted by similarity, perhaps, so you'd add up all the similarity scores in the top 100, divide by 2, then count down until you checked off that much similarity, and just use that player's performance in that years (in other words, re-order by performance in a given year, then count off the similarity).
These are projections, not raw data dumps. The idea of an algorithmic projection is to use past data to predict the future as accurately as possible by whatever means available. No one is being constrained at BP to publishing raw data derived solely from similarity scores; nothing is preventing them from using comparable player data as one input to a more sophisticated predictive model. For all we know, they already are doing so. My argument is simply that the current black box model's output is not looking terribly logical, and doesn't appear to me to represent the best estimate of what a given player's TAv will be in a given year in the future.
As for the curves looking reasonable to you, only a high prevalence of *non*-career ending injuries would cause a lot of ramps in skill followed by immediate declines in a players' early 20s. Not sure there are enough damaging-but-not-catastrophic injuries occurring to young position players (plus legitimate age ~21-onset skill declines) to make that the average case. Note that in many cases, both the ramps up and the declines are projected; it's not like the model is simply predicting regression for outperforming minor leaguers.
I agree with you on the utility of upside. Would like to see 25th and 75th percentile projections for each year, so we get an idea of the volatility/uncertainty levels.
Career-ending injuries and other forms of culling explain the ticks back up in even later years - players who stick around that long tend to play well or they wouldn't be employed.
1) 70th percentile of what sample and what data value does it go by? All players by WARP? 2nd basemen by WARP?
2) The 70th percentile data has him 273 plate appearances. Just over half a season. This just doesn't make sense to me.
Can anyone clarify? Thanks.