February 6, 2013
Comparing Mock-Draft Data
On the most recent episode of the Towers of Power Fantasy Hour (or 2), Paul Sporer and I had the opportunity to interview Greg Ambrosius of STATS LLC. Ambrosius runs their National Fantasy Baseball Championship (NFBC) events, which feature the all-stars of high-stakes fantasy baseball. While the industry may not be as lucrative as the World Series of Poker, NFBC’s top players have made a rather nice living though the competition. The most famous of the players may be Lindy Hinkleman, a pig farmer from rural Idaho who was featured in a USA Today article last season. A few years ago, he pulled off an improbable sweep of both the live and online events, and earned a check for $241,500 from NFBC.
During the podcast, Ambrosius mentioned that NFBC hosts its own mock drafts on the site, as their players wanted to have the opportunity to do research while interacting with other like-minded players, rather than with the diverse communities at places such as MockDraftCentral, CouchManagers, and RTSports. Ambrosius maintains that the ADP data that his users produce is more meaningful than what happens elsewhere, because it is data from live individuals—rather than bots that fill empty spots in mock drafts—and the participants are unlikely to employ wacky strategies to skew the data, such as filling all pitching slots before drafting hitters. Given the fact that MockDraftCentral is the most widely known of the other sites, I decided to do a comparison between NFBC and MDC data to see how different the two sets truly are.
There are two caveats in comparing the data between the two sites. Many of the MDC drafts only go 256 players deep (12 teams, 23 man rosters), while NFBC events are typically at least 450 players deep (15 teams, 30 man rosters). Of the 427 players listed in the most recent ADP report at MDC, 258 of them went undrafted in at least one of the 140 mock drafts run in the last five days. It is understandable that Corey Hart would go undrafted given his knee injury, but players such as Coco Crisp, Nick Markakis, Ryan Doumit, and even Mariano Rivera has suffered the same fate. Anyone that had a “ND” as their lowest draft number was assigned a rank of 450 for the purposes of this study. Conversely, because of the different formats in NFBC, each of the 450 players they have ranked has been drafted in each of the 32 mock drafts the site has run over the past week.
The second caveat is the purpose of each site. NFBC does not allow any post-draft trades, so its mock drafters have no interest in seeing what a draft may look like when you punt saves, because punting categories in a league that emphasizes the draft so heavily and where free agent acquisitions are tough to come by is a bad strategy. On the other hand, MDC participants may use its mock drafts as a sand box to see what happens when someone punts a category or two to emphasize other ones.
To calculate the range each player has in both reports, I compared the players’ lowest and highest reported draft position. The table below shows the range for each report, broken down in groups of 25:
It takes until the low 300s for MockDraftCentral’s figures to show less volatility in players’ ranges than the NFBC data. In graph form, here is what the volatility of the data above looks like:
If we compare only the players that were drafted in every mock draft run on either site, the graph evolves to this:
The large amount of undrafted players in MDC drafts does indeed skew the data, but removing that player pool shows that the two sites are not that far apart in their player ranges for the top 75 players, or the first five rounds of a 15-team draft.
The other question would be how each site treats the players. As mentioned earlier, NFBC participants are less likely to punt categories given the rules of the league, but even hitters are treated differently at both sites.
The following players have all been drafted at MockDraftCentral, but have not been taken in the latest NFBC drafts: Aaron Harang, Al Alburquerque, Brian Matusz, Jake McGee, Jonny Venters, Oscar Taveras, Pedro Ciriaco, Scott Feldman, and Zach McAllister. Each of those players had an ADP of 247 or higher at MDC, led by Alburquerque at 247.
The 10 largest player-ADP differences between players drafted in all drafts at MDC and NFBC belonged to:
Conversely, these were the largest differences for players that are drafted in earlier rounds in NFBC mock drafts:
It’s worth noting that eight of the 10 players on that list are either relief pitchers or middle infielders. Based on the previous table, MDC users (or bots?) are reaching for certain players that the more-serious community at NFBC is waiting to take.
The two sites agreed on player ADPs for just 10 players: Brandon Belt (239), Lance Lynn (193), R.A. Dickey (65), Felix Hernandez (38), Jed Lowrie (244), Ryan Braun (2), Andrew McCutchen (6), Nick Markakis (150), Mike Trout (2), and Albert Pujols (7).
If we state that any range within -25>x<25 is agreeable, then the two sites agree on approximately 150 players. Beyond that, results vary greatly, as illustrated below.
Most of the disagreements come when comparing the MDC ADPs to those at NFBC. All positive values reflected on the scatterplot below result from comparing MDC to NFBC, while negative values result from comparing NFBC ADP to MDC.
In summary, the confidence Ambrosius expressed in the NFBC data was well-founded. That said, the data at MockDraftCentral should not be ignored, as it is quite prevalent in the marketplace and allows you to look at the same information as your competitors. If you do decide to utilize only one source of ADP data, this analysis shows that you are better off going with NFBC. Hence, we will be making that data available on a weekly basis throughout the next few months here on Baseball Prospectus—thanks to an agreement with STATS LLC to have them release it to us every Thursday.