Deep, But Playable: Trust, Buckets, and Prospect Evaluation Frameworks

Image credit: USA Today Sports

Given that the basic framework of our prospect lists hasn’t been altered very much since at least the Jason Parks-era format, it might seem that we don’t spend a lot of time thinking about how to best convey the information within them to our audience. I promise this is not the case—Jeffrey Paternostro, Bret Sayre, and I take time every year to create a wish list of items we’d like to see on the top-10 lists, or tweaks we’d like to make to it. We then argue over whether any of those items actually enhance or compromise our ability to communicate the evaluations, and the concepts on which those evaluations are based.

An offseason discussion with Jason Wojciechowski regarding the A’s list gave me reason to question the effectiveness of our current setup. There are two prospects who appear at the back end of the list who couldn’t be more different in terms of player type, and yet both received the same 55/45 OFP/Likely grade. While Sean Murphy (ranked 10th) earned those grades with a more standard distribution of skills and risk, James Kaprielian checked in one spot above him with a much more … eclectic mix of present talent, projection, and risk.

To refresh, here is what we had to say on each

Kaprielian:

The Good: At his best, Kaprielian runs a big fastball into the upper-90s, pairs it with three offspeed pitches with potential to be above-average, and has strong command and a feel for pitching. The fastball is only sometimes that hot on velo, and he was a low-90s guy out of the draft, so it could be fool’s gold. On the other hand, the slider is showing up plus already and the change and curve both flash potential. Four-pitch potential with good stuff usually leads to writers bestowing their ace blessings on a young man. Why not here?

The Bad: Well, he had Tommy John surgery in April 2017 after barely pitching in 2015 and 2016 due to recurring elbow problems. He’ll return sometime in 2018 as a 24-year-old with 29 pro innings in three seasons, all in A-ball. He’s also never maintained the high-end velocity without his elbow barking nearly immediately after.

The Role:

OFP 55—Mid-rotation dude with tantalizing velocity/good Nate Eovaldi
Likely 45—Dude bouncing between roles with tantalizing velocity/normal Nate Eovaldi

The Risks: We have absolutely no idea whether he can stay healthy throwing a baseball, let alone whether it’s as a starter, a reliever, or somewhere in between. Since the improvement in stuff was so closely correlated with his elbow problems, we also have little idea what he’s going to look like as a healthy pitcher. There’s way more upside than the ranking indicates, but there’s also major concern he’ll never pitch effectively again.

Murphy:

The Good: Murphy is an excellent defender, with all of the building blocks you look for in a future above-average catcher. The arm is among the better ones currently in the minor leagues, with strong velocity that holds its line, and he maxes it out with fluid pops that will register plus-plus times on the regular. He controls the zone at the dish, demonstrating a clear plan of attack with a swing built to bring above-average power potential into games.

The Bad: It’s a strength swing that can get rigid through the zone, and he was exposed a bit by the more advanced arms of the Texas League, with loads of rolled-over ground ball contact the most frequent result. He puts the ball on the ground so often it’s unclear just how much of his power will play at higher levels, and while he’s not a black hole of running speed like some of his catching compatriots, he’s also not going to make it a habit of beating out too many of those left-side grounders, either.

The Role:

OFP 55—Above-average catcher
Likely 45—Solid second-division catcher

The Risks: The bar for catcher offense is so low, and Murphy’s defensive baseline high enough, that it’s not hard to envision him hitting enough to produce comfortably above-average value behind the dish. It’s wise to temper expectations for the offense panning out to ceiling, but the fundamentals here are such that Murphy’s a higher likelihood prospects than most.

As you can see, Kaprielian poses a problem within our valuation framework: by all accounts, when he’s healthy he projects as a potential top-of-the-rotation starter, with an upper-90s fastball and the potential for multiple above-average breakers. That type of ceiling generally earns an OFP (Overall Future Potential) of higher than 55—even if OFP isn’t representative of a true ceiling. Conversely, the health issues he’s already endured put his realistic floor well below that of a back-end starter, especially factoring in that even if he returns to the mound, it could be with diminished stuff.

So we see that Kaprielian and his high-beta outcomes pose a particular problem that his neighbor to the South, Murphy, does not. A more honest accounting of Kaprielian’s OFP might be a 60, or even a 70, when he’s healthy and at his finest. A more honest representation of his floor might read: “60-day disabled list.” But these honest representations belie the complexities of projecting the probabilities of the various permutations of a player’s career. Those complexities are present across every OFP/Likely grade we generate, but players like Kaprielian push the framework we employ close to its breaking point.

So, we’re left with a few options: switch to a different framework—perhaps one with an all-encompassing grade, such as Future Value; create an entirely different framework that better explicates the ways in which these probabilities shake out; or live with what we’ve got and understand there are going to be some guys who just won’t fit the system well.

We’ve seen the usage of one singular grade, that bakes in the good, the bad, the risks, and so forth. We have our current system that demonstrates a strong developmental outcome, as well as a likely one, hopefully demonstrating the distance in risk that some players have relative to others in more than just words. But there’s another way, that I’d argue both the singular figure produced in Future Value and the OFP/Likely both draw upon, but don’t state explicitly.

In fact, it might well be the best way for us to convey this information on a guy like Kaprielian, and arguably on every prospect—and that is clearly delineating our estimates of their likelihood to land in each grade bucket from 20-80.

For Kaprielian that could look like:

Whereas for someone like Murphy, it could look like:

The exact percentages aren’t the point so much as recognizing the way the distributions are so starkly different, and that’s without getting into the notion that Kaprielian’s likelihood of being a “20” is probably misleading because he’s probably either better than that or just not on the field (thus, not even an organizational player).

The reason we don’t (or haven’t) presented the information in the above format to our readers isn’t because they wouldn’t get it—we have some of the smartest, sharpest readers on the planet—but rather because there’s a significant element of false precision in this framework. False precision is something I worry about a lot. There’s already a fair amount of it baked into our current model, and opting to choose something that would introduce more of it has always concerned me, even if I think the above better represents how my prospect team and I go about thinking of these players.

So, for the same reason that I shy away from using grades such as 35 and 65 (yes, this is a real point of contention within the community—some do not consider them “real” grades), I shy away from adopting this approach for every player. Sometimes the broader view (OFP/Likely) allows for better signal, as the details are obscured and the bucketing of percentages and outcomes and projection provides a cleaner presentation and a cleaner take away. The granularity that the percentage approach exposes is a better way to understand potential prospect outcomes, but it also allows someone to be right no matter the outcome.

Spread the possible outcomes over the different percentages and you’re never wrong—this version of reality just played out a certain way, which “you” already conceded was a possibility. The OFP/Likely approach bakes in the more granular effort, while also forcing one to a decision point: What do you believe the ceiling version of this player is, incorporating the risks involved with health, development, and so forth. Furthermore, what do you believe a realistic outcome is for that profile? How do you think those same risks are likely to play out?

I used the term “honest” before, when describing the bucketing approach. While that is the most accurate approach, I’m not sure that the OFP/Likely designations don’t force us to be more honest in our assessments of those likelihoods, or at the very least, more useful to our readers as long as they trust us to be honest in the first place.