Creating a Mechanism for Evaluation of Draft Strategy

Part of what drew me to Baseball Prospectus, other than my respect for Jason Parks and his vision of a scouting-department-style “Prospect Team,” was the allure of stepping into a ready-made readership eager and able to help me explore baseball on both a macro and micro level. As far as the draft is concerned, that means not only breaking down draft prospects from a scouting perspective on a player-by-player basis, but also working to understand what goes into formulating an overarching approach to player acquisition through the draft. This includes general strategies relating to draft acquisitions, as well as draft-class-specific game planning.

Over the years, I've found one of the easiest assertions an amateur scout can make is, "I really liked [insert successful prospect name] coming out of [high school/college]." During the early days of my journey to grow as an evaluator, whenever any familiar prospect showed a leap in development at the professional level, my mind always ran back to whether or not that was something I was surprised to see. Almost always the answer was "no". But that doesn't really make sense, does it? I was, and really still remain, relatively new at the evaluation game, and it seems highly unlikely that I was able to effectively evaluate complex propositions such as the future success of young baseball prospects with so few years of earnest scouting under my belt.

Furthermore, when inking a draft report, I have always trended conservative in my appraisals, with a focus on identifying risk level and probability (probably an offshoot of my training as an attorney, which places a premium on issue-spotting and rainy-day planning). It would follow, then, that my outlooks generally aren't all that rosy, but rather focus on the things that might go wrong. So why was I seldom surprised with prospect success stories? The short answer is that it is easy to "like" potential and, for the most part, any draft prospect worth expending evaluative resources possesses potential in spades.

As basic as this concept seems, I firmly believe it is an overlooked truism, both for those in the industry and for those covering the scouting game as media members, informal bloggers, or fans (granted, to a greater extent with the latter two than with the former). Hitting on a prospect's upside, or "getting it right,” is intoxicating. You were able to look at a table of ingredients and correctly predict the taste profile of the meal to be shaped—it's a talent, and one that requires intelligence and fortitude to shape. Further, for those in the industry, it is a skill that has the potential to change the fortunes of an organization, particularly when you hit big on an impact talent.

Of course hitting on high-ceiling talent is the ultimate goal, but the nature of the draft acquisition game requires more than evaluating the broad swath of draft-eligible prospects to the best of our ability. At the decision-making level, concerns expand past individual player appraisal to a need to recognize what opportunities an organization’s particular draft slot affords the org, and how best those opportunities can be leveraged to build a draft class that will pay dividends down the road.

How we dissect the draft

Writers covering the draft have historically done so with (1) short scouting reports that paint a broad picture of draft prospect profiles, and (2) rankings. This is an excellent approach to introducing readers to players with whom most are unfamiliar, and for giving a 30,000-foot look at a draft class on the whole. Without fail, these reports historically accentuate the positives associated with players, primarily because readers want to know why they should be excited about a particular player their team selects (and not why that player is likely to fail).

For area scouts, the story is in a lot of ways the same. Scouting is often measured by successes—those who hit on guys get promoted and area scouts with years under their belts and no portfolio of MLB players run the risk of not being renewed when their contract expires. Missing on a guy isn't a badge of shame, so long as your report isn't too far out of whack with what the crosschecker(s), scouting director, special assistants, and the like see. In fact, it could be argued that evaluators on the whole grade themselves too lightly when it comes to misses.

I've had more than a few evaluators repeat the mantra to me, "Even the best in this business are right less than half of the time." This is certainly true, but to accept a low success rate among draft prospects without challenging ourselves to improve our decision-making process is to stagnate the evaluative process altogether. So, in an industry where successes tend to define the quality of the evaluations (both by the industry, and in the eyes of a fan base), and failures are more or less tolerated on a large scale, who is truly evaluating well and where can we identify areas in which improvement is needed?

As far as writers are concerned, the answer most often shouted out is "Rankings.” Rankings put you on the record as to whom you like, and in what order, assuming all factors have been taken into account unless otherwise indicated. I was initially a strong proponent of rankings, and continue to find merit in the process. I have kept track of my own class-wide rankings for five or six years now, and will continue to do so. But if "even the best in this business are right less than half of the time," how harshly are we to grade our own rankings in retrospect? Does a top-40 rating for Tyler Austin make up for the fact that you had Tyrell Jenkins 20 spots ahead of Taijuan Walker in 2010? If you listed Buster Posey as your top draft talent in 2008, is that selection made less impressive by the fact that you had Tim Beckham as the second-best overall talent (and Kyle Skipworth as a top-10 talent)?

To some extent, there is valuable information to be gleaned from a slot-by-slot review of a class-wide ranking. Most notably, you may be able to identify general areas where you tend to be heavy or light on a particular draft profile. Perhaps more important, not all writers are concerned with digging into the dirty details of the draft. Readerships are not shaped from cookie cutters, and for many, a couple of bullet points that can be regurgitated on a message board or over beers at the bar is all a writer needs to provide.

My hope is that you are interested in exploring the draft a little more deeply (and if you have made it this far through this introductory piece, I assume you are interested in such an exploration). To do that, we are going to use this space at Baseball Prospectus to continue a project I’ve been working on for several years—the running of a shadow draft and the monitoring of a shadow system. The process is simple: Each year, we’ll select a team and, at each of that team’s draft slots, we’ll make a selection as if we were the decision makers in the draft room. The selections that are signed and begin their respective pro careers will be placed in our “shadow system,” which we’ll monitor and periodically appraise with an eye toward improving our process.


We pick this project up five years in, with the results thus far set forth in the below table, breaking down the current shadow system by level. As a general matter, and one we’ll explore in more detail in future pieces, I have taken a portfolio approach—looking to balance probability with upside, in each case across the spectrum of position profiles:
















Ty Buttrey

Mitch Brown

Matt Wisler

Chad Bettis

Zack Wheeler

Brian Matusz

Rookie Davis

Matt Smoral

Matt Bischoff

Brody Colvin

Sonny Gray

Nick Tepesch

Dillon Maples

Daniel Norris

Kyle Winkler

Ian Krol

Justin Grimm


Jared Ray

Kevin Brady

Tim Melville

Kendal Volz



Ryan Berry

Cody Kukuk

Colton Murray

Alan Oaks




Madison Younginer






Eric Semmelhack






Justin Amlung






Ian Gardeck






Samuel Stafford






Bennett Klimesh













Johnny Eierman

Albert Almora

Kyle Gaedele

Todd Glaesmann

Nick Castellanos

Brandon Crawford

Bralin Jackson

Josh Elander


Ryan Strausborger

Bryce Brentz


William Argo

Cody Koback


Tyler Holt

Roger Kieschnick



Jeremy Rathjen







Jake Cave















Kenny Diekroeger

Jack Marder

Chris Dominguez




Matt Snyder

Connor Powers





BJ Guinn







Tony Thompson














Blake Swihart

Micah Gibbs



To better put the system in context, BP Prospect Team member Mark Anderson kindly took the time to put together a quick ranking of the system (set forth below), noting he liked the upside at the top but that the system falls off quickly as you enter double-digits:

  1. Zack Wheeler
  2. Albert Almora
  3. ​Nick Castellanos
  4. Matt Wisler
  5. Justin Grimm
  6. Sonny Gray
  7. Blake Swihart
  8. Matt Smoral
  9. ​Bryce Brentz
  10. Daniel Norris
  11. Chad Bettis
  12. Todd Glaesmann
  13. Nick Tepesch
  14. Mitch Brown
  15. Bralin Jackson
  16. Ty Buttrey
  17. Kenny Diekroeger
  18. Cody Kukuk
  19. Kyle Winkler
  20. Dillon Maples

Mark went on to note he liked the collection as a middle-of-the-pack system, which I take as a solid endorsement, particularly considering this represents talent acquisition solely through the draft, without further strengthening via international signings or through trade. The major-league successes are limited thus far, but there is potential impact talent set to arrive at some point this year and (hopefully) fairly regularly thereafter. As an additionally selling point, I believe the cross-section of talent sets up well for purposes of trade package construction, which is to my mind an important aspect of farm-system and player-development utility. There are clear failures in appraisal, and some strategic shortcomings that I look forward to discussing further in future pieces, but overall, I have been pleased with the direction of this project, and very much look forward to the challenge of bringing in the next wave of talent this June.

Looking to 2013—best available, but all else equal…

With just five weeks between now and the draft, in the short term this series is going to focus on the process of formulating a game plan for a particular draft class. For 2013, we will be stepping into the shoes of the Boston Red Sox and making a selection for each of their first 20 picks. With a top-10 overall pick (seventh overall), the first step in setting our strategy is going to be to determine our likely targets for our first selection. From there, we’ll lay out several potential approaches depending on who falls to us and the opportunities we expect to see at each of our next few slots.

Based on my looks at this class’s draft-eligibles over the past 10 months, I’ve narrowed targets for 1-7 to 14 players (listed below in alphabetical order):

1. Mark Appel, RHP, Stanford University

2. Trey Ball, LHP/OF, New Castle HS (New Castle, IN)

3. Ryan Boldt, OF, Red Wing HS (Red Wing, MN)

4. Kris Bryant, OF/3B, University of San Diego

5. JP Crawford, SS, Lakewood HS (Lakewood, CA)

6. Jon Denney, C, Yukon HS (Yukon, OK)

7. Clint Frazier, OF, Loganville HS (Loganville, GA)

8. Jonathan Gray, RHP, Oklahoma University

9. Sean Manaea, LHP, Indiana St. University

10. Reese McGuire, C, Kentwood HS (Covington, WA)

11. Braden Shipley, RHP, University of Nevada

12. Dominic Smith, 1B/OF, Serra HS (Los Angeles, CA)

13. Ryne Stanek, RHP, University of Arkansas

14. Kohl Stewart, RHP, St. Pius X (Houston, TX)

It is important to note that this is not necessarily a snap shot of the top 14 overall talents in the draft class. Rather, this collection represents those talents I currently view as falling in one of three categories: 1) an above-average value for the slot, or talent beyond that which I’d typically expect to be available at this slot; 2) solid value for the slot, or talent I’d typically expect to be available at this slot, and 3) fallback talent, or talent roughly on par with those in category 2, but better suited to fit (as part of my desired draft class on the whole) with talent I expect to grab in the next few rounds.

Generally, the rule of thumb for draft acquisition is “take the best available talent.” However, outside of a few bright lines in each draft class, the construction of draft profiles, taking into account probability, upside and signability, seldom lends itself to clear-cut preferences at a player-to-player, or micro, level. More commonly, we will examine the fit of a player into the broader overall draft approach, arriving at a few potential courses of action. From those narrowed options, we’ll make our ultimate selections. To summarize: “Take the best available talent, but all else equal, take the talent that best fits our draft-wide goals.”

Looking ahead

As this introductory piece has stretched on some time already, we’ll cut it off here with an eye to the coming pieces in this series. As noted above, our first step will be to identify our cross-section of potential selections at the top of the draft (i.e., breaking our 14 noted selections out into the three stated categories: 1) above-average value, 2) average value, and 3) comparable value setting up future selections. In between Dissecting the Draft pieces, I will be posting player-specific reports through BP Unfiltered (in the blog section of the site). A link will be provided in the DTS pieces, but please check in on BP Unfiltered periodically if you are interested in reading more on these specific players. Additionally, broader player profiles will be provided through the Scouting the Draft Series.

If you made it through this whole piece, you deserve a gold star. Thanks very much for your time, and I look forward to picking this conversation up soon.

Nick J. Faleris is a practicing structured finance attorney and Sports Industry team member in the Milwaukee office of Foley & Lardner LLP. The views he expresses in Baseball Prospectus are his own, and not necessarily those of the law firm.

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
I'm looking forward to the series Nick. I have a question about sigability. How have you handled that in the past? I noticed in Castellanos at least one player who fell in the draft and required a substantial overslot bonus.

With the new CBA the draft class budget issues are now front and center more than ever. Will you also be constraining yourself to the Sox bonus pool as well? If so, it'll be interesting to hear your reasoning for whatever approach you take (ie Astros like cutting a deal at the top and spreading the excess around, Tor like and spending everything at the top and going for 1k senior signs after the top few rds, etc).


Thanks, philly. It should be a fun series leading up to the draft (with a focus on the actual shadow draft), and thereafter discussing larger draft issues.

I have kept signability and total expenditures in mind in the past, as I think that's the only way this exercise really works. Exploring the challenges of operating under the new system's slot allotments will absolutely be a large part of what we discuss. As far as last year is concerned, I drafted in place of the Astros, meaning I had a good amount of flexibility. My strategy was similar to Houston's (I was really happy to see this on draft day) -- here is how the top 11 rounds went:

1. A. Almora (looking for $1.5-2.5 MM savings on $7.2 MM allotment)
1S. M. Smoral (planning to require $500K-1MM add'l to slot)
2. M. Brown (plan called for HS arm -- 4 lined up/two were available)
3. K. Diekroeger (plan called for HS arm but went with upside college player when it looked like Buttrey might fall; had to build in possibility of overslot)
4. T. Buttrey (plan called for one more overslot early-rounder)
5. B. Jackson (plan called for portfolio balance -- here, needed a HS position player)
6. J. Elander (portfolio -- college position player and value for me)
7. J. Rathjen (money save to add cushion for pick 2 and 4)
8. Semmelhack (money save to add cushion for pick 2 and 4)
9. M. Snyder (portfolio pick -- college corner bat)
10. K. Brady (portfolio pick -- college arm)
11. Alex Bregman (insurance/negotiating leverage for Buttrey/Smoral)

I'll dissect this in greater detail, maybe in July as a comparison piece with the 2013 draft, but the general strategy I settled on was taking Almora at a discount on the full allotment and spreading the excess to cover 1) Smoral or McCullers in the supplemental round, and 2) an available early-round talent in the 4th Round. The Bregman pick in the 11th Round locked in insurance for Buttrey/Smoral negotiations, though that's an example of real world planning that is difficult to implement in this format (obviously I don't get to actually negotiate these deals, so I run the risk of the actual drafting team not signing a player).
It was interesting to watch the NFL draft unfold last week. I would be curious to see a contrast/comparison of the influences on both the MLB and NFL drafts. Signability, team needs, draft pool talent, etc. There are so many factors involved, many unique to each sport. It might be a fun excersize.
It would absolutely be an interesting exercise -- unfortunately, outside of routing for players from my alma mater to be selected early and often, I don't have much insight into the evaluation of NFL draft-eligible players. I know one glaring difference between the NFL draft and MLB draft is the simple difference in time-to-production, meaning top NFL draftees are usually expected to contribute almost immediately, whereas even the top baseball draftees are routinely expected to log at least 1.5 years in the minors, with an adjustment period upon arrival to The Show.

I've now exhausted my NFL draft knowledge....
I only followed the NFL draft because my son showed a lot of interest---gotta seize every bonding moment possible. His football knowledge ran laps around mine, but who cares, it was fun.
On the surface, it seemed NFL teams put a lot of emphasis on team need, as opposed to best athlete available, but it could have been a product of overall quality in the draft class. In the later rounds, the "draft experts" seemed to keep trying to will teams to take the best player available, but teams mostly stuck to addressing their needs. Like you said, it speaks to time-to production. Anyway, looking forward to your next installments.
Nice article, Nick. I love the concept even though there are a lot of ways to second guess process doing these.

BTW, Kendal Volz retired.
Thanks; I should probably have made clear in the article (and will do so in the next installment) that the goal is certainly not to question any approaches teams have and are taking. I pick a team to shadow simply so there is a construct -- the goal isn't to compare my 2013 draft class with Boston's.

In fact, and this is something we'll explore in more detail when the draft is in the rearviewmirror, I am a strong believer that anyone promoting one draft strategy across the board probably isn't doing enough digging into the various approaches (and accordingly the various benefits of taking different approaches) to draft acquisitions. Teams are coming at the draft from very different circumstances, beyond slots. The Red Sox, Rays, Rockies, Twins and Angels will all have a multitude of variables particular to that organization that should help determine the best draft strategy possible.

It's about maximizing opportunity, and those opportunities are not only slot specific, but are also organization specific.
I didn't mean to second guess the team you are choosing for, more on how to keep track of who you picked going forward.

Doing a shadow draft is hard because you haven't talked to these guys about money or have access to medical records. In this scenario, if Smoral wouldn't have signed, do you sign up Bregman even though he went to LSU? Where do you draw the line? It can be complicated because MLB doesn't have one feeder league like NBA, NFL.

It's nice to see another intelligent source of draft analysis. I look forward to reading more.
Got it. Well, I keep it pretty simple. If the player doesn't sign with his real drafting team, I don't get to add him to the shadow system. You're absolutely right I don't have control over actual negotiations, so the best I can do is look at the process.

Using Smoral/Bregman, had Smoral not signed I would have been out the player, but could look at the approach I took and take a side note that in the "real world" I might have had the flexibility to notch Bregman. I keep track of the non-signs, as well, as it's a good stat to follow. If those non-signs go on to be high drafts later, it's at least an indication I'm ID'ing some talent.

Likewise, let's say Bregman DID sign, I'd have to make the decision as to whether to take both Smoral and Bregman, along with the MLB instituted penalty, or only add one to the shadow system in order to stay within my allotted budget.
wondering what some of your hits & misses have been over the past 5 years. Like did you take Matusz where the ORioles did or did you pass on Hosmer for him? What about Sonny Gray?
Matusz was taken while I stood in the shoes of the O's (this project spun out from a shadow draft that originated with Jon Shepherd over at Camden Depot (the ESPN SweetSpot Orioles affiliate), so that was the starting point. That year the top five I had marked as targets for Baltimore in our draft went as follows:

1. Posey
2. Beckham
3. Alvarez
4. Matusz
5. Smoak

If you recall, the night before the draft Posey floated a $10MM signing bonus demand (or something in that range). Keeping in mind we were trying to work within expected confines of the Orioles organization, I struck Posey from the list (face palm) and Matusz was the best available when the pick came up.

Re: Gray, I had selected, and failed to sign, Karsten Whitson the previous year, so I had the 9th and 12th picks (drafting for Houston). The short explanation is that, when my pick came up, my "must haves" were off the board. The pairings I was hoping to have available after about pick five were Lindor/Gray, Lindor/Fernandez, and Lindor/Bradley. With Lindor gone, I shifted gears and went pure upside with Swihart. That meant I wanted a little less risk with my next pick and opted for the college arm (Gray) over the prepsters.
well i'm sure the Rays agree w/ the Face Palm over not taking Posey.
After we're through the 2013 draft, we'll definitely take a closer look at the misses and the hits in an effort to discern if there are patterns we should be avoiding moving forward. Keep in mind, a miss in a vacuum isn't really a useful data point in this context, since a one-in-five hit of a particular profile type could be a worthwhile investment. Should be fun to explore and discuss!
Thanks, Nick...very interesting. How do you select which team to shadow each year? Obviously the Astros and Red Sox present very different challenges and opportunities.
I honestly don't remember what led me to the Astros after shifting over from Baltimore. This year, after a fair amount of work, I've developed a process for loosely ranking the "advantages" of the various organizations. I then looked at teams outside the top and bottom third of that loose ranking and pared down options based on which slots I thought presented the most interesting options for this draft class.