There will be a very short planned maintenance outage of the site tonight (7/22) at 11 PM ET
April 2, 2014
Just How Well Does TrackMan Work?
Alan Nathan#, Jeff Kensrud*, Lloyd Smith*, Eric Lang#
#Department of Physics, University of Illinois
*Sports Science Laboratory, Washington State University
Let us begin by answering the question posed in the subtitle: Pretty darn well! Now let’s see how we arrive at that conclusion.
Back in early 2013, one of us wrote a Baseball Prospectus article entitled "How Far Did That Fly Ball Travel?". The article posed the question: How well does the initial velocity vector (speed and angles) determine the landing point of a fly ball? Utilizing home run data from actual MLB games, it was determined that the initial velocity vector poorly determines the landing location. Much of the rest of the article was devoted to speculation about why that is the case. Three possible reasons were identified and investigated: variation in wind, backspin, and the baseball itself. The latter is the most intriguing possibility, since variation in the seam height and/or surface roughness of the ball might have a significant effect on the air resistance experienced by the ball.
As a follow-up to this research, we decided to do an experiment under more controlled conditions rather than use MLB game data. Since we wanted to eliminate wind as a possible factor, we approached the Houston Astros organization about using Minute Maid Park (MMP) with the roof closed for our experiment. To our delight, they agreed, so the four of us gathered in Houston for two cold days this past January to perform the experiment.
A very fancy pitching machine, designed and constructed by the Washington State part of our collaboration, was set up at home plate and was used to project baseballs either as fly balls or as line drives, with complete control over the initial speed, angles, spin rate, and spin axis, as well as the baseballs that were used. A high-speed video camera viewed the initial part of the trajectory in order to measure both the initial velocity vector and the spin rate. The landing point was measured with excellent precision using a tried-and-true, albeit old-fashioned technique: a very long measuring tape. But the really neat part of the experiment was that we had redundant measurements of all of those quantities (landing point, spin rate, initial speed and launch angle) using the Trackman unit that is permanently mounted in MMP.
While the primary goal of the experiment was to determine what factors besides the initial velocity vector determine the landing point, the redundancy in our experiment afforded us the opportunity to test how well Trackman stacks up against more conventional techniques. It is that comparison that we report in this article.
At this point, you might ask why anyone cares how well Trackman works. For that, we’ll give you two answers, one of which might not be obvious. The four of us are researchers in the science of baseball. Trackman, both the MLB stadium version and the portable versions, is one of the tools that we use in our research. While it is probably true that our understanding of the inner workings of Trackman is better than most, it still is essentially a “black box” that spits out numbers. If we are to use this as a reliable tool in our research, we need to know both its strengths and its limitations. It is primarily in that spirit that we undertook the analysis presented here.
However, there is another reason why it is important to benchmark the Trackman system, one more closely tied to the game itself. Trackman is currently installed in a number of MLB ballparks and data from it are utilized by team analysts as one of the tools in their evaluation, scouting, and decision-making processes. Moreover, reports abound that starting in 2015 Trackman will be installed in every MLB ballpark and will be used to track every pitched and batted baseball. Once again, for the data to be useful as an analytic tool, it is important to know their accuracy and precision.
Before going into the results, we first present some additional details about the experiment. The pitching machine was used in two different configurations, both utilizing an exit speed in a narrow range about 96.0 mph. All baseballs were projected with pure backspin, with spin rates in the range 1100-3200 rpm. The horizontal (“spray”) angle pointed close to 00, corresponding to straightaway centerfield. For 73 fly balls, the vertical launch angle (VLA) was in a narrow range around 27.80, resulting in distances in the range 320-400 ft. For 48 line drives, the VLA was about 0.50. Tape-measure distances were not measured for the line drives. Three different types of baseballs were used: MLB balls, MiLB balls, and raised-seam NCAA balls. (Indeed, much of the variation in fly ball distances can be attributed to differences among the types of baseballs and even to differences within each type, but that is a story for another day.) Let’s now move on to talk about the results.
First let’s take a look at the landing points. Figure 1 is a plot of Trackman distance vs. tape-measure distance.
The blue points are those for which Trackman tracks the ball for the full trajectory, from start to ground level. The red points are those for which the radar does not track the ball all the way to ground level. The usual reason for incomplete tracking is that the ball falls outside of the field of view of the radar, which is an especially common occurrence for high popups. For all such cases in our experiment (27 out of 73 fly balls), the tracking terminated just prior to the apex of the trajectory. For these points, Trackman has a proprietary algorithm for extending the trajectory beyond the last measured point to come up with an extrapolated landing point.
The dashed line is a linear fit to the blue points—those with full tracking. The slope of 0.998 indicates that Trackman and tape measure are in excellent agreement. There is more scatter of the red points about the line, almost surely due to the incomplete tracking. The root-mean-square (rms) deviation of the points from the line is 0.5 ft for complete tracking and 2.5 ft for incomplete tracking. It is worth noting that the Trackman report provided to MLB teams has a field labelled “last tracked distance” in addition to the extrapolated distance. For our data with incomplete tracking, the ratio of last-to-actual varied between 0.40 and 0.65. The data showed no obvious correlation between the size of this ratio and the discrepancy with the actual distance.
Figure 2 compares the “bearing angle”, which is Trackman’s terminology for the horizontal angle of the landing point, with 00 pointing to straightaway centerfield.
Once again we see excellent agreement between Trackman and tape measure for the complete tracking points and more scatter for the extrapolated points. If bearing is translated into distance perpendicular to the 00 line, the rms scatter of points about the line is about 0.2 ft for complete tracking and 5 ft for incomplete tracking.
We take a closer look at the landing points in Figure 3, which is a scatter plot of the difference between actual and Trackman data, with distance on the vertical axis and bearing on the horizontal axis, using the same color coding as the previous figures.
We can clearly see the excellent agreement for full tracking and more scatter for incomplete tracking. Nevertheless, it is worth pointing out that even the incomplete tracking does very well, with the worst case missing by less than 6 ft on a 380-ft fly ball. From this we conclude that Trackman has a very good algorithm for making the extrapolation. The complete-tracking points are shown in an expanded view in Figure 4, once again showing the excellent agreement with the conventional measurements.
Note however that the Trackman distance is about 0.6 ft less on average than the tape measure distance, a tiny amount that may be due to the origins of the two measurement systems being slightly offset from each other. Regardless of the reason, we do not consider this small offset to be significant.
We next take a look at the spin data in Figure 5, in which we plot Trackman versus video spin rate.
The individual points are plotted in blue, and the dashed curve is a line corresponding to equality between the two measurements. With some exceptions, the data are tightly bunched about this line, indicating substantial agreement. But there are exceptions, indicated by the 6 points (out of 94 total) for which the Trackman spin is well below the video spin.
Inspection of the raw Trackman data, which are not generally available to Trackman users, indicates that this effect is almost surely an artifact of how the raw data are processed. Indeed, five of the six points differ from the video spin by exactly a factor of two and the other by exactly a factor of five. The red points on the figure result from multiplying the points that are obviously wrong by the appropriate factor of two or five, and these corrected points fall right on the line. It is interesting to note that each of the six points is a line drive; no fly balls that we measured showed that effect. Given that pitched baseballs are nothing more than line drives going in the opposite direction, it would be well to keep this occasional problem in mind when using Trackman to measure the spin of a pitched baseball.
Figure 6 is a histogram of the differences between the two spin measurements, including the six points that we have corrected by hand.
We see that the distribution is approximately normally distributed, with a mean difference of only 8 rpm and an rms spread of 35 rpm. We conclude from this analysis that except for cases when Trackman makes an obviously wrong determination of the spin (e.g., half the “expected” value), it agrees with conventional techniques very well.
Finally we examine in Figure 7 the comparison between Trackman and video for the initial velocity vector.
Data are presented only for fly balls because of calibration issues with the video data for line drives. The figure shows the difference between video and Trackman values, with VLA on the vertical axis and speed on the horizontal axis. There is a very small offset between the two sets of measurements, amounting to about 0.5 mph for speed and 0.250 for VLA. An error in the camera calibration is the most likely cause for the offset. For example, the speed offset would be accounted for by an error in the camera calibration of only 0.5 percent, which is within the margin of error of our calibration. The scatter of the points about the offset is small, with rms values of 0.5 mph and 0.10 for speed and VLA, respectively. We consider this to be excellent agreement between the two measurement systems.
What are we to conclude from this experiment? Perhaps it is useful to summarize with some bullet points:
Taking all of these bullet points together, we arrive at the conclusion we presented at the start of this article: Trackman does pretty darn well measuring the things that we have independently measured.
It is a pleasure to thank the Houston Astros organization for allowing us to use Minute Maid Park for our experiment and for helping us make it a success. A special word of thanks goes to Mike Fast and Ben Lowry for their help. We also gratefully acknowledge the Trackman people for their help in setting up the stadium unit at MMP and for their patience in answering our many questions.