August 10, 2004
Confessions of a Questec Operator
How the System Works, How It Can Be ImprovedLast year in a BP Q&A, Ivan Santucci discussed the basics of the Questec Umpire Information System. As a former Questec operator, allow me to take you behind the scenes.
Since its first inroads into major league ballparks a few years ago, Questec has exploded onto the baseball scene, exploded at the hands of Curt Schilling, and has since evaporated a bit from our consciousness. Still, Questec continues to grow its reach, with systems now functional in 23 of the 30 MLB parks. The company's promising technology and increasing reach into the game show Questec's potential. Technical glitches in the system and cash flow problems could stand in the way.
From April to July of this year, Questec played a big part of my life (I'm moving to Texas next week to start grad school). Rotating Miller Park home games with a working partner, I'd arrive about an hour before game time to complete pre-game procedures. Pre-game procedures include saving the previous game and clearing the system of the last game's data, all of which takes 10-15 minutes. Unless of course something goes wrong, which is the main reason for early arrival. As an operator, you need to be prepared to deal with bumped tracking devices, expired software licenses, lack of an audio feed, video feed or both, and a raft of other little problems.
While working games, my work setting was the loading dock at Miller Park. The room is called the Telecast Center, which is a fancy name for a cold, damp, cinder block storage room and a bathroom. My workstation was a tool cabinet with two computer monitors, a TV and the usual computer accessories.
The operator watches the game through a small TV that shows a network game feed. A computer on the left records and scores the game. The scoring system is simple; the operator tells the computer who the umpire is and who the combatants are, and scores the game pitch-by-pitch. Later, on the same computer, the operator will compile the data and turn it into a CD, which is then distributed to the Umpire, MLB, and Questec. While scoring the game, the operator also scores the game on paper, to make sure that all of the pitches are scored correctly. The operator must worry about the score sheet first, because it is easy to catch up on the scoring computer between innings.
The computer on the right is called the Tracking PC, which shows the tracking of each pitch. What the operator sees is a shallow arc with dots throughout denoting how many points the pitch was tracked on. More points equal a better track. In order for this process to run smoothly and as accurately as possible, the operator needs to "calibrate" the system prior to the game. The calibration involves setting the dots on certain parts of the field, with the back point of the plate being the most critical. This calibration gives the cameras a center point from which to track each pitch.
As the game goes on, the scoring PC automatically removes warm-up pitches and only includes pitches with a result (called ball, foul etc). After the game, the pitches are further filtered to only include pitches in which the umpire called ball or strike. The largest and most controversial procedure is setting the strike zone for each called pitch, a process Santucci described in his interview. For all the criticisms the UIS system gets, this process should not be of concern to critics. Setting the lines at the top of the belt and at the hollow of the back knee only takes about 10 pitches for a beginner to master; Santucci closely scrutinizes every operator's training. Following the setting of the strike zone the operation performs a series of mouse clicks that produce a data CD, which includes the following features:
CDs are passed to the umpire by the operators the next day, or distributed by Major League Baseball. This depends on whether or not the umpiring crew is in town the next day or if it's off to another city.
Relative ease of use is one of the Umpire Information System's major strengths. The system does ensure complete objectivity. It is impossible for an operator to get back at an umpire who made a bad call a few weeks back and cost his team a game. What's more, any small mistakes that are made have an equal chance of helping the umpire as they do of hurting him.
For example, if an operator sets the strike zone on a particular pitch an inch too low, he has no idea if it will help or hurt the umpire. This is because when setting the strike zone the pitch is just a number, so the operator has no idea where the pitch ended up or what call was made. Each operator merely supervises the system, a system that can be a valuable tool to umpires who actually do want to improve. The UIS can give umpires valuable insight as to the pitches they are consistently missing. It isn't very difficult or time-consuming for the umpire to study the data and make a mental note of where he is struggling. Implementing those changes from there is up to the umpire.
A core weakness of the system is that it consistently misses certain types of pitches, mostly sweeping pitches (sliders and curves), as well as late, hard, 'boring' pitches such as the cut fastball. These pitches can land all the way into the opposite batter's box, but because they caught a tiny slice of the front corner of the plate, the computer calls them strikes. No umpire who values his safety is going to call a pitch that lands in the opposite batter's box a strike.
Some pitches also have to be thrown out for various reasons. Operators often encounter a coach or bench player sitting in front of the camera, making it impossible to set the strike zone. Miller Park is particularly difficult because of the shadows that emerge between the mound and home plate. These shadows have often caused the pitch to be missed entirely. These missed pitches have to be thrown out of the data set and can greatly shrink the sample size, a crucial blow to any objective analysis. Santucci notes that Questec is working to correct this problem.
After working with the system for more than half a season, it is apparent why umpires hate the system and why Sandy Alderson and others in MLB offices love it. Umpires should be held accountable for their actions, to be sure. But some umpires are simply not very good behind the plate, yet are excellent in the field; the reverse is no doubt true for others. Would it be worth only allowing certain umpires to work behind the plate? Perhaps this is something MLB should consider. The system in some ways is much like the SAT: Those who score well don't have a problem with the system, those who don't tend to complain about the system. If you look back at the history of Questec, the umpire who is one of the most vocal critics of the system happens to be the worst umpire that I scored all season.
Several problems need to be addressed for Questec to continue its growth. First, the glitches in the system need to be fixed immediately. Though that will require some technical work, Questec will likely have to spend some money to address the problems as well.
Also, more needs to be done to allow the system to make a greater differentiation between umpires. The average game consists of about 150 called pitches. Roughly 80% of the umpires fall within a 5% accuracy range. What does this tell us--that the vast majority of MLB umpires are average? Indeed, the way the system is currently set up tells us very little about which umpires stand above the rest. Consider this: In a 150-pitch game, a 90% accuracy score (which is exceptional, and very rare) means that the umpire still missed 15 pitches. Fifteen!
Major League Baseball also needs to consider buying Questec. To become a better acquisition target, the company needs to get its finances in order. Cash flow was sometimes an issue, as some Questec operators have received their paychecks weeks late. Problems have also occurred in getting an adequate supply of blank CDs, as well as mailers to mail the CDs to their intended recipients. If MLB were in charge, its resources would likely enable the company to run more smoothly.
MLB can help reduce umpiring errors by implementing the Umpire Information System across the board. But it must first work with Questec to eliminate key problems, both operational and technical. If the system is accurate, then umpire's complaints are easily refutable and better decisions can be made on hiring and firing issues. The longer MLB waits in its perfection of the process, the more the problems will persist.
Jason Karegeannes is an intern for Baseball Prospectus. He will begin graduate school at the University of Texas-Austin this fall, where he'll study Sports Management. You can reach Jason at firstname.lastname@example.org.