keyboard_arrow_uptop
Image credit: USA Today Sports

Every major-league game you watch, ball-tracking technology is used to deliver live views of where the ball passes through the strike zone. This is an entertaining and seemingly unbiased image of where the ball traveled through the zone. As a result, it’s no surprise that there’s been a steadily increasing call to replace the home plate umpire and rely entirely on machine measurements to call balls and strikes in real time, i.e. a “Robo Zone” or “Robot Umps.”

The reality, however, is that there are myriad issues with implementing a ball-and-strike-calling system like this for real games. The primary issue is that using machine measurements to call balls and strikes will simply shift disagreements with the call from the umpire to the machine, or to the machine’s operators.

People watching a game also tend to forget that their viewing perspective alters how the pitch appears to them, which can create the same animosity when the machine report does not match what people believe they saw. Your perception of the pitch can change significantly depending on whether your point of view is from beyond the outfield wall, or from the perspective of pitcher, umpire, or batter.

In order to fully assess the implications of embracing our mechanical strike zone-judging overlords, we’ll take a look at the issue from two standpoints. First, we’ll look at the potential issues that could limit or impact the implementation of the technology, followed by an assessment on the impact the switch would have on the game.

Potential Issues

If you’ve ever seen people posting images of a “live strike zone” during games (Twitter, we’re looking at you), it’s clear that too many fans believe these systems are 100 percent accurate. This is simply not true, as there are limited accuracy tolerances within which the machines are reliable, and human tolerances in which the machine operator decides to set the top and bottom of the zone. Sportvision claimed that their PITCHf/x system, first introduced in major-league stadiums in 2006 with data made public in 2007, was accurate to one inch. In analysis performed by Mike Fast in 2011, he found that the error on that system was actually better than that:

Just over half of the stadium-seasons have a measurement error below half an inch in each dimension. When both dimensions are combined, 29 percent of the stadium seasons are below half an inch of measurement error, 67 percent have less than one inch of error on average, and 98 percent are less than two inches.

Mike Fast – Spinning Yarn: How Accurate is PitchTrax?

There does not appear to be any publicly available strike zone accuracy tolerance or error rates for ball-tracking systems by manufacturers such as TrackMan and Flightscope. Are they accurate to within a tenth of an inch? One inch? Two inches? Four inches? It’s important to know.

Even if the companies did publish that data, MLB would still need to set allowable amounts, and then test these systems in order to verify and calibrate those machines on a constant basis. These set tolerances also need to include the amount of time the machine takes to process the data (i.e., the time that elapses between the pitch and the decision made by the machine), or the game will be slowed down.

With PITCHf/x there was a concerted effort from Sportvision to engage the public analytical audience in helping to refine and improve the product. This is most clearly exhibited through their hosting of PITCHf/x Summits, which were aimed at collaborating with the public analysts who made PITCHf/x so popular. This interest in collaboration not only improved PITCHf/x as an analytical tool but also built trust with the public. Max Marchi summed up the sentiment thusly:

Sportvision has been doing a terrific work in the past few years in tracking every major league pitch and we are really fortunate that it (and MLBAM) let us put our hands on that wealth of data. Thus, pointing at miscalibrations is not meant as criticizing their amazing work, but rather as a way to give something back.

Max Marchi – Fine tuning PITCHf/x location data

For current ball-tracking systems, some are known to have issues reporting correct spin rates on pitch types, such as certain types of sliders, low-spin pitches, gyro-spin pitches, and missing exit velocity and spin data on certain batted balls. Much of that likely is due to the way the Doppler effect is used to detect spin. But it’s also quite possible some of these errors could be due to occasional radio frequency (RF) interference, even though the estimated frequency band for TrackMan and Flightscope is likely the relatively little used 10.5GHz band. This is based on viewing their mandatory FCC certification filings.  Plus, RF interference could be other transmitters (accidental, or on purpose), self-reflections, or other effects from material (like cables or seats) near the transmitter. In fact, in these bands, rain of any kind can be a significant interference source that can be hard to eliminate, even with the best processing.

Those are the types of measurement errors and error rates that need to be monitored. It is widely known that both Doppler-based and camera-based machines are not accurately reporting 100 percent of their measured data 100 percent of the time. Even if they were accurate 99.9 percent of the time, what if that 0.1 percent is a strike but is called a ball on a 3-2 count in the bottom of the ninth inning with bases loaded and two outs in a tie game? It may seem like a silly thought exercise, but it’s important to consider these potential impacts on the game.

In the example below, you can see two views of the exact same pitch. From the mound view the pitch looks to be a ball, but a closeup shows it was a strike just barely on the edge of the zone. For a pitch this close to the strike zone, the accuracy tolerance of a machine calling balls and strikes is critical.

Rob Arthur of FiveThirtyEight showed that when it came to pitch tracking, the new TrackMan systems that made up Statcast had been experiencing some issues:

Errors in both horizontal and vertical movement have never been higher in the four years that Statcast has made some of its data publicly available. So it’s not just your imagination as you watch the game on TV: In-broadcast representations of the strike zone (like FoxTrax) take their data from Statcast, and Statcast’s errors, in turn, have bred anger with umpires and confusion over how pitches are being called.

The root cause of Statcast’s troubles is unknown. The problems could originate in the hardware, the computer code processing the resulting data, or any other part of a complex system. The hardware part of Statcast — the part that actually tracks pitches — is a radar system sold by a company called TrackMan.

Rob Arthur – Baseball’s New Pitch-Tracking System Is Just A Bit Outside

***

These ball-tracking systems all feature a combination of hardware and software that work together to track, analyze, and record each pitch thrown. The hardware—Doppler radar transceivers, camera systems, etc.—and software are each equal sources of potential issues. There can be software inaccuracies, bugs, or other issues that impact performance like any piece of software. Additionally, when software determines what measurement is reported, that opens the possibility for hacking. In theory it would be fairly trivial to modify software to report pitches closer or further away from the strike zone, toggling that to favor one team or another. While data is being sent from the machine to the reader, this can also be hijacked, or a spoofed measurement reported, again to favor one team or another. Securing the integrity of the software and the transmission link on which its sent to the user to read is critical.

Even if the data is securely transmitted and received, it is also uncertain how, exactly, the two sensor types (camera and radar) will be used to make the calls directly. Will one sensor make a primary call, with the other relegated to backup or confirmation? What if they disagree? If they’re to be used together, what algorithm will be used to combine the reports from each sensor? How are those validated? Once again, the measurement tolerances come into play, as the overlap between each sensor’s uncertainty region can mean the difference between a dead rally and a half-inning that won’t end.

You also have to prepare in the event the machine breaks. Machines will break, and under Murphyʼs Law, break at the worst possible times. If there were no umpire calling balls and strikes behind the plate, you’d have to account for the instances where the machine is broken. A broken machine could be caused by hardware issues or software crashes. There would have to be backup systems in place to immediately switch to or else there would be intolerable game delays. Would we need an umpire on standby to step in and make the calls in the event that the system goes down mid-game?

Even for calibration, measuring the accuracy of one machine with another machine is risky. One machine measuring another adds similar tolerance and accuracy issues within the machine being used to perform the calibration. This means that calibration would need to be done using more direct and indisputable physical methods, not unlike the foam board used by Sportvision to monitor the accuracy of the PITCHf/x system:

Sportvision’s high tech lab (parking lot) and several baseball diamonds have played host to their engineers on many occasions to test the system. Setting up the PFX cameras, and placing a foam board on the front-plane of home-plate, they have been able to establish the level of accuracy of the px/pz values. Within 1 inch. Sometimes better. In any case, you and I can trust those plate locations within a third of a baseball.

Harry PavlidisThe State of F/X: MLB.com Gameday and Baseball Analysis Improves

Verifying a machine’s accuracy is important[i], but it’s not the only potential source of issues. Currently, a stringer or operator is responsible for setting the top and bottom of the strike zone for each batter and each plate appearance. That process is subject to error as much as an umpire behind the plate calling strikes.

Even if you had optical detection and software to automatically set the top and bottom of the strike zone, we have the same issues with the accuracy and tolerance of that technology as we do with the radar systems, as well as potential for hacking or tampering to favor one team or another. That of course is if a machine setting of the zone were even feasible; it’s unlikely that a machine would be able to accurately track the top/bottom of the zone in real time as a batter moved in the box. There would also be the likelihood that batters would develop ways to game the optical top/bottom zone detection system much like catchers have developed techniques for pitch framing.

Rule 2.00: The Strike Zone

The STRIKE ZONE is that area over home plate the upper limit of which is a horizontal line at the midpoint between the top of the shoulders and the top of the uniform pants, and the lower level is a line at the hollow beneath the kneecap. The Strike Zone shall be determined from the batter’s stance as the batter is prepared to swing at a pitched ball.

Additionally, the rule book strike zone is somewhat subjective as well and would likely need a further refined definition if machine systems were to be implemented for calling strikes in a live game. Dr. David Kagan recently explored this aspect of the robo zone problem, noting that it remains one of the largest obstacles to implementation.

This would impact the analysis of the game as well–it’s unwieldy and unreasonable to account for the unique zone on every pitch when performing analysis. There were more than 720,000 pitches thrown in major-league games last year, and in theory each has a different zone depending on the batter’s setup in the box. Even if you limited it to one zone per batter, that’s still more than 1,300 different strike zones to analyze. Hence the rise of the “typical six-foot batter strike zone,” a relatively uniform instrument for post-game analysis.

The challenge is that even with this iteration of the zone for analysis, there isn’t consensus on the exact zone to use. This is an analysis challenge rather than a robo zone one, so we’ll save further discussion on this for another time.

If there were a Robo Zone, at some point there would need to be agreement on how to interpret the rule book strike zone using machine data. Dr. Kagan pointed out that MLB is using previous umpire calls to set the zone rather than stringers now:

PITCHf/x originally used poorly paid “stringers” to sit in a dark room under the stands and manually turn a dial to set the top and bottom of the zone on the video image of the batter. Saunders reports that Statcast uses the previous calls of major league umpires to build a database of the top and bottom of the strike zone for each hitter. 

Isn’t that ironic? Until MLB comes up with a machine-comprehensible definition of the top and bottom of the strike zone, machines will need the assistance of humans to define the strike zone for the machines.

Dr. David Kagan — The Physics of RoboUmp

We built a model that compared the umpire-called zone to the ones that stringers set to see what kind of impact the change Kagan notes would have. In reality, it would be minimal. Below is a chart showing the distribution of the bottom of the zone, on the y-axis by stringers and on the x-axis by umpires:

There’s a strong consensus that the bottom of the zone is roughly 1.75 feet off the ground, with variance for different batter heights and stances. We tested the variation in called strikes and found that using an umpire-set bottom of the zone rather than a stringer zone would result in a 0.0015 percent difference in calls at the bottom of the zone. This change may be a good one from a process standpoint, but it’s largely negligible from an impact standpoint.

Aside from the 2D dimensions of the strike zone as shown above, a machine is capable of measuring the strike zone as a 3D area volume above the plate, as opposed to a front only-facing 2D area. The image below illustrates what this 3D area volume looks like. It’s unlikely that umpires would call strikes in such a 3D space.

A machine could, however, so would pitches that enter the zone through anything other than the front facet of the zone be considered strikes by the robo zone? While it’s rare for a pitch to fall into this category now, it’s possible that pitchers could game the system by throwing more eephus pitches designed to drop into the top of the zone, or some other such strategy. Again, these implications may seem outlandish, but they must be reconciled before moving forward with something that could drastically change the game.

Impact

It’s important that we talk about the impact the robo zone would have on the game in real terms. Dr. Kagan did an excellent job outlining how random error would impact the called zone, so we won’t rehash that aspect here. What we will do, however, is examine real impacts to the game based on the available data today. How many pitches would change from ball to strike or vice versa?

The images above highlight the changes that would manifest should an automated system be implemented. The zones being used here correspond to a 24-inch-wide zone with the top and bottom set by stringers at each park. Pitches in red are strikes called by human umpires that would now be balls. Pitches in green are balls called by current umpires that the robo zone would call strikes. As you can see, the top and bottom of the zone would be significantly impacted, as would the outside corner for lefties (which is often left uncalled).

These charts also highlight the difference in opinion of the strike zone among stringers and umpires, as umpires tend to call pitches down below the bottom of the stringer zone strikes with some regularity. How might this change depending on count? We know that the zone expands and contracts in relation to whether the pitcher is ahead or behind.

There are changes here, too. Life becomes significantly better for the player with the advantage, as pitchers who fall behind no longer have a larger zone to pitch to, while pitchers who get ahead will get all those strikes on the edge that are often called balls.

Strike Rate Batter Ahead Pitcher Ahead
RoboZone (raw) 36.2% 16.6%
RoboZone (calibrated) 35.8% 16.7%
Human Umpires 37.7% 14.6%

Once calibrated, there’s roughly a two-percentage-point difference between the way that humans call the game and the way that the robo zone would be called. That may not seem like much, but it could result in a significant increase in walks or strikeouts.

It would likely impact pitcher strategy, as they’d be incentivized more than ever to get ahead early in the count–likely prompting early action from the batters as well. With human umpires the zone expands and contracts depending on the count and whether the batter or pitcher has the advantage. Knowing that, pitchers who fall behind know that an expanding zone can help them battle back. When the zone doesn’t change size based on count, pitchers will be incentivized to try to get ahead early so that hitters have to cover areas they haven’t traditionally had to defend when behind in the count (e.g., the top corners of the zone).

We looked at every major-league game in 2017 to see how many calls would change from one thing to another based on our model. The results ranged from just four changed calls to 50. Some games would be minimally impacted, likely in a way that was hardly noticeable. Some games, however, would see significant changes in how they’re called. This highlights the inherent issues with a system in which nobody seems to agree on what the strike zone ought to actually be.

Pitchgrader’s analysis across a significant amount of collegiate strike zone data suggests more of the same, with umpires generally missing calls in the same locations as their major-league counterparts (albeit in different magnitudes). That said, there can be significant strike zone changes from game to game and umpire to umpire, which suggests that implementing a robo zone in the collegiate game would see even greater impacts than in MLB.

(How PitchGrader handles missed calls.)

***

Conclusion

Even with all the potential issues, there are potential solutions for them. Technology could be employed to reduce subjective calls made in error with a redundant system. After all, everyone wants the calls made both quickly and correctly. Even so, we would never be in favor of a purely robo strike zone without an umpire.

Let’s assume we tried to create a reliable measurement system. It would need three elements as below:

  1. Doppler radar system
  2. Optical tracking system with video
  3. Umpire

Together, the three elements would be used to call balls and strikes in a majority rules fashion. If both machines call a strike, but the umpire calls a ball, the umpire is overruled, etc. Though this could work, it could also be an overly complex system that would be prone to more problems, breakdowns, delays, and errors than it potentially solves.

The various ball-tracking systems are an invaluable resource for live and postgame analysis, particularly in player development and scouting. The data they produce opens new ways of seeing details of the game like never before. It goes beyond sabermetrics into a new facet for process analysis rather than results analysis.

The data is a resource that needs careful interpretation, consideration, and understanding to be useful. If they’re accurate 99.9 percent of the time, that still makes the data they produce an amazing and invaluable resource for player development, scouting, and general fan interest. That’s the best application of the systems that exist today.

With all the potential problems and complications of implementing a robo strike zone it seems that, instead of a robo zone, an unbiased and skilled umpire is still our best choice.


Bio: Wayne Boyle is an electronics and software engineer with a career spanning 30 years. He is the founder of Pitchgrader, which makes advanced software used by top D1 & MLB teams for Pro level baseball 3D analysis/simulation/projection, and actionable meaning for pitched and batted ball tracking data for player development and scouting.


[i] Teams, especially colleges, need a fast and inexpensive way to perform tests on their own now and then. A good method to test strike zone accuracy would be as follows: Print a 1” grid pattern on a foam board, with printed marks for left and right side of the plate, and a height marking such as 10”. Then secure that foam board to a stand with a wood backing. Something simple like a 2’x4’ sheet of ¾” plywood, with 2×4’s attached to stand it up. Simply line up the foam board with the plate, and straight up vertically, and measure that the 10” mark is actually 10” from the ground. Then as each ball hits the target, mark the dent with a marker pen as #1, #2, etc. and confirm the foam board is aligned before the next pitch. Later you then compare the ball tracking system values with the ruler measured values on the foam board dents. A free calibration image that teams with ball tracking systems can print on a 24” x 36” foam board is available from Pitchgrader here.

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
tearecrules
1/29
I think the technical problem of "it does not register 100% of pitches" will be the biggest problem. It's one thing for the pitch tracker to be wrong; the umpires are not accurate either. It is another thing for the pitch tracker to go "what pitch?" and have it result in everyone standing there looking befuddled until the umpire does an Enrico Pallazzo and halfheartedly says "Strike?"
John Hicks
1/29
Excellent!
wayneboyle
1/30
Yes, even if the missing data error rate were 0.01% , that would be 72 pitches out of the roughly 720,000 pitches thrown in MLB in 2017. Doesn’t seem like a lot until you consider what would happen if 72 times a pitch was thrown and there was no call. Nothing... just silence... until the fans go nuts.
wayneboyle
1/30
Also, missing data error rate and data errors are 2 different things. I work with this data every day for the last 3 years. There are times where data is missing completely, and times when the data is just wrong. i.e. if a pitch registers as 121mph, it’s wrong.
Still, the data is absolutely awesome for the game. Mainly in player development and scouting. It’s also fun and interesting to see on TV in a live game, just have to remember it’s not absolute.
Robert Hacking
1/29
"Even if they were accurate 99.9 percent of the time, what if that 0.1 percent is a strike but is called a ball on a 3-2 count in the bottom of the ninth inning with bases loaded and two outs in a tie game? " ________ What an incredibly weak argument. The current accuracy rate of an umpire is less than 90%? Gory math details can explain how large the gap is here. Think Grand Cayon. After reading this article I'm even more convinced that the computer should be calling balls and strikes.
Byron Hauck
1/29
I'm glad I wasn't the only one who had trouble with this part.
jnossal
1/29
Huzzah.
Robert Hacking
1/29
If the computer was calling balls and strikes today and someone said: 'This isn't working, it's wrong .1% of the time. Why don't we get a human to stand back there and get it wrong 10% of the time?' that conversation would last 2 seconds.
wayneboyle
1/30
2 seconds unless you dig deeper into the question. Umpires are essentially judges. They mostly judge balls and strikes. So why wouldn’t we want Robo judges in the courts? Because we’d realize after 2 seconds of thought that the Robo judge could only be as fair as the information it recieves and the software that determines what to do with that information. The viable answer is for machines to assist the umpire, keeping the human element in the game. At the same time, we all want skilled and unbiased umpires. If they’re not skilled enough, send them down for more training. If they’re biased, remove them.
Byron Hauck
1/30
Come on, that's ridiculous. Umpires judge a spatial relationship. Judges judge a relationship between human behavior, events, and a complex set of laws. Umpires: a moving ball and an invisible box. Judges: all of human behavior and more laws and prior rulings than could fit in a warehouse. I'm not an expert on logical fallacies, is this some sort of unholy union between a strawman and a slippery slope?
wayneboyle
1/30
You’re not digging deep enough yet. How do you know the hardware gathered the information correctly, and how do you know the software intepreted the data correctly? Answer is, you don’t , you’re guessing. The argument isn’t against using machines, it’s that machines aren’t as perfect as people think. Also forgetting that the zone the machine is measuring against is set by a human. What if they set the zone 2 inches too high, and the machine then uses that? Plus hacking isn’t even addressed. Even if you had a mythical perfect machine, it can be hacked to favor one team or another. These machines are used in real time, you can’t go back and review if it was hacked after a game and expect to be able to fix it.
Jeff F
1/29
Maybe we can replace the players with actual robots too. And replace the fans in the stands with holograms. And then let fans at home make the on-field decisions via playstation
Robert Hacking
1/29
Yeah, because a common phrase you hear when walking out of the park is: 'Did you see that umpire! He was great. Can we come back tomorrow night Dad to see him again?'
jnossal
1/29
+1
Jeff F
1/30
Actually I've been to hundreds of baseball games. Not ONCE have I actually heard anyone say - what we need is robot umps. Crowds LOVE razzing the umps. And count me as one fan who will never attend another game if baseball ever heads down this path
Jeff F
1/30
And btw - there are life lessons embodied even in the disputes over umps calling strikes - and standing firm even when everyone in the park wants his head on a platter. boiled down to - He calls em like he sees em.
Christopher Schaffner
1/29
Umpires have always been part of the game. Sometimes they make a bad call. Sometimes they play favorites. So what? I much prefer human error and controversy to the endless replay reviews, and "phantom outs" that occur when an obviously safe runner disengages from the bag for a split second. Please lets go back to the old way, and never discuss this again.
John Hicks
1/29
Amen!
Robert Hacking
1/29
Agreed replays take too long, but that's irrelevant to this particular discussion. The computer is capable of making the calls both more accurately AND quicker than a human.
jnossal
1/29
Pretty sure Chris is being tongue-in-cheek.
majnun
1/29
Pretty sure they are not
Christopher Schaffner
1/30
Quite serious. Why are we messing with a good thing?
jnossal
1/31
Like Eric Gregg maybe?
wayneboyle
1/30
The idea is we don’t want Robo umps. Too many issues that haven’t been fully thought out and resolved. We have to talk about it so that Robo umps aren’t forced on the game. What we all want are more accurate and consistent calls. That can be done by better umpire training, weeding out biased umpires, and technology to assist the umpire in game. There should always be an umpire behind the plate.
Robert Hacking
1/31
"There should always be an umpire behind the plate." This one statement sums up your opinion and bias. You're trying to make the point that the technology isn't ready but final admit here that you don't ever want to see it happen.
wayneboyle
1/31
Contorted logic from a cherry picked sentence won’t fly. Anyone who has read the article, or read any of the comments I wrote here can clearly see I’m consistently for technology assisting umpires, but not replacing umpires. Not replacing umpires clearly means there would be an umpire behind the plate.
Loren
1/29
I've long thought that the best use for strike automation would be as an aid to the plate umpire. If the system could provide a very fast judgement to the umpire, but the umpire still had the final say, it could be a useful tool. There should probably be an interface designed such that only the plate umpire was informed to reduce lengthy arguments. An earpiece or light indicator inside the mask might work.
tearecrules
1/29
I thought that too, but it won't work. Once the umpire gets the indicator light they're going to default to following what it says. Couple reasons for that. 1) Simple laziness. Once the computer is taking the requirements to locate a ball traveling rapidly in space from the umpire they're going to start looking at others things. Is the pitching committing a balk? Is the batter staying in the box? Is the catcher where he's supposed to be? They're no longer going to be focusing exclusively on the flight of the ball.

2) The umpire is going to need some pretty good justification for why they overrode the pitch tracker. Half an hour after the game is over when they're explaining to the league why they thought a pitch which was clearly 4 inches off the plate was really a strike.
wayneboyle
1/30
Agreed. But as Tearecrules points out, there are technical challenges to implementing machine assisted umpires. I think it would be better to have augmented reality glasses or a mask that allows the umpire a clear view as normal, but overlays the machine traiectory so the umpire has some additional info to consider. Especially if it allows a faded view of the batter stance from the umpire’s view. Idea is to help the umpire make better calls, not replace.
jnossal
1/29
What a bunch of excuses. Are you going to tell human umpires are accurate at less than an inch? And are as consistent as a machine? Dissemble all you want, electronic umpires are coming and the game will be better for it.
wayneboyle
1/30
Fact is, no one knows for sure. There are no listed accuracy ranges for the machines. But let’s suppose for discussion that they’re within +/-1 inch. That means an umpire could see a strike, but the machine thinks its a ball 2 inches outside. Who’s right ? After all, they are giving an accuracy rating for umpires based on what the machine said. The ball could have actually landed in the zone, but the +/-1 inch machine tolerance says it’s outside.
The future will likely be umpires assisted by machines, likely some form of augmented reality glasses or face mask.
Byron Hauck
1/30
Wait, why do you get to give machines a hypothetical +/-1 inch variance and then assume the ump is perfect? What if I assume the machine has a +/-.01 inch variance and the ump has a +/-1.5 inch variance?
wayneboyle
1/30
Because the point is we don’t know. Everyone assumes the machine is infallable. How do you really know that?
Byron Hauck
1/30
But we don't really know how accurate the human umps are, either, and it makes even less sense to assume they're infallible. We need to study whether or not robo umps are better than what we have now, not whether they're perfect.
jnossal
1/31
Straw man. Machines don't have to be infallible, just better than humans, which is provable, now.
wayneboyle
1/29
It’s great to see the discussion sparked with this article. That was the intent of the article in the first place. Too many have blind faith in the machines, not realizing they’re not as perfect as you’re led to believe, and they provide no proof. As fallable as Umpires can be at times, I still prefer an unbiased and skilled Umpire.
But for post game analysis and player development, the machines are absolutely awesome tools, when used correctly. Just not suitable for live game calls yet.
Robert Hacking
1/30
Unbias? I guess you never saw Joe Brinkman call a game. Human beings can be biased without even being aware of it themselves.
wayneboyle
1/30
Unbiased and skilled is the ideal. The Umpires you don’t hear about are doing it right. Main point is that people assume the machines are 100% correct , when they aren’t. And on top of that, the machines still need a human to set up, calibrate, and operate them. More chances for human error. The machines are fantastic, but just not ready for prime time live game calls yet.
Robert Hacking
1/30
Thanks for the article, BTW. I hear you saying 'the machines simply aren't ready yet', those on the other side of the argument are simply pointing out that none of your points are convincing of this. This point once again missing is that if the machine is 98% accurate it's kickin' the cr*p out of the human behind the plate and you haven't addressed this in your article or any of your replies. It doesn't have to be perfect, it just needs to be miles better (and it is).
wayneboyle
1/30
That’s really great that there’s a discussion. Many sides to the issue. The reason I wanted to write the article was to bring out all the issues that usually aren’t discussed. With 30yrs designing hardware and software, RF included, I’m very aware of the potential problems. The fact that we really don’t even know how accurate the new Doppler radars are, or how often then make major errors, or hardware or software crashes, means we truly can’t say which is more accurate. Just because a machine, or even video, shows a strike, doesn’t mean it was really a strike. With video, you have the camera view perspective that can make a ball look like a strike. You also have to realize that the video zone box is set by a human, which is just as prone to error as an umpire.
And there’s software that controls the machine and does the calculations to turn the radio signals into information. What if software version 8 is being used, but there is a more accurate version 9? I can hear it now, someone complaining “that’s a strike all day with version 9”
I’m all for eventually getting to the point where machine’s can assist an umpire to make better calls. Just that all the machines and software required are prone to lots of problems that are intolerable in a live game.
jnossal
1/31
No such thing as an unbiased human umpire. There probably isn't such a thing as an unbiased robot either, but at least it will be consistently biased.
Shane Waggoner
1/29
If we were inventing baseball today, is there any way in the world we would chose to have the error rate of human umpires as opposed to the error rate of the robo strike zone? We would intentionally choose to have less accurate game outcomes?
wayneboyle
1/30
That’s an interesting thought. I’d guess that if baseball were invented today that the strike zone would be a fixed rectangle instead of different for every batter and how the batter stood that day. That would go a long way towards more consistent calls because umpires would train for years with one zone.
Robert Hacking
2/07
With respect, you're ignoring Shane's point and inserting a different one. If baseball was invented today a computer would call balls and strikes and THAT would make it more accurate.
tearecrules
1/29
Being slightly more serious about the issue I'm kind of surprised the league isn't pushing this as a safety issue for the umpires. I know why the umpires aren't doing it, taking away their ball and strike authority basically takes away the only task they perform which is exceptionally difficult. But I'd be surprised if half the games I watched last year (maybe about 50; I work and I have a kid, don't judge me for not watching more!) didn't feature the home plate umpire taking a foul ball off the mask. Automating the balls and strikes would let the umpire be able to do something else which didn't require them to be right behind the plate.

As for answering the question of "setting upper and lower bounds" at the start of any series the teams take some BP with sensors at the rule-book height locations. I'm thinking like the motion-capture rigs used for movie and game CGI but just where it's needed. Use that information to set the player's heights when they come to bat. Heck, if it's quick enough you might even be able to do it before each game. It won't be perfect, but it's probably more accurate, consistent, and reviewable than having someone dial in a guestimate.
wayneboyle
1/30
Yes, MLB already does that with preset batter zones. Harry shows in the article the difference in calls between relying on those stringers vs how the umpires called. That was very insightful because it gives an indication of what could happen if there were a Robo zone being used in games. He even compared what would happen in some real games in terms of how the game outcome would be affected.

But who set the zone for that player in the first place? If the machine called the balls and strikes, then players would adapt and learn to game their Robo zone.
Jim Maher
1/29
What I would like to see is a laser light at the edge of the plate on each side coming up so the Ump would have its assistance in determining inside and outside pitches. the ump would still be making the determination. If a light beam were broken one could tell more easily that a part of the ball crossed the plate. High and low would be more difficult but not impossible.
wayneboyle
1/30
I like where you’re headed with that. Assisting the umpire. There are better ways to do that, such as augmented reality glasses or mask, or something along those lines to assist an umpire is probably the best way to go.
David Yeager
1/30
Superb and fascinating analysis. Well done.
wayneboyle
1/30
Thanks David. We all worked very hard on that for a few months. The intent was to bring to light all the potential issues and spark a thoughtful discussion, especially in areas that haven’t been thought out fully. We’re definately pro machine for the player development, scouting, and post game analysis work we do. It’s invaluable and great for the game in that regard.
Mac Guyver
2/05
I prefer my mistakes to be made by humans. I can stomach that quite easily.
James Leahy
2/16
Scientists say that cars soon, if not already, will be able to be driven more safely by computers. If computers can do a complex task like driving a car better than humans, there is no doubt that a computer can call balls and strikes far more accurately. As for umpires always being a part of the game?Baseball is an athletic contests between humans where rules must be followed. Human umpires have been used to enforce those rules because that was the only way to do it. The enforcement of the rules is a necessary part of the game, but the human umpires are not.
Ryan Hammon
2/19
This is a terrific accounting of the ways in which robo umps are less than perfect. But as good as that accounting is, it doesn't support the conclusion. For all the ways robo umps may fail us, the ways in which human umps fail us are just as numerous and just as egregious. I understand the difficulty in citing the available data to "prove" human umps have a higher error rate than the robo umps, but it seems clear that the preponderance of evidence points to robo umps being supperior. The goal here shouldn't be perfection; it should be improvement on the status quo. With a relatively small investment of time and money, MLB could address most of shortcommings described here and ensure the robo umps would consitute an improvement in ball/strike accurancy. If folks want to argue that human umps are better because they like human error in the game, like having egg shaped strike zones, like having different strike zones for lefties/righties, like having a person to rage against rather than a machine, so be it. But to argue in favor human umps over robo umps based on error rates? Sorry - I just don't buy it.
lawheelock@gmail.com
5/31
Question. MLB Statcast documentation says that the top & bottom of the strike zone are determined by the operator when the ball is halfway to the plate

https://baseballsavant.mlb.com/csv-docs#sz_top

In reviewing statcast data, the sz_top & sz_bot values for the same batter, in the same plate appearance are different.

For example: MIN at TEX August 15, 2019 (statcast game_pk 566826) Elvis Andrus strikezone, as determined by the operator in his first inning AB varied 3.44 inches. In the 6 pitch AB, the sz_top value started at 3.31 then went 3.30, 3.20, 3.23, 3.03, 3.02. A robot called strike on the first pitch at the top of the zone, would have been called a ball in the same spot just 4 pitches later by the same 'robot'.

The most accurate thing said in this article is that this will just shift the disagreements. We won't solve the problem by getting rid of human umpires, we're just choosing a different problem.