keyboard_arrow_uptop

In a postseason featuring late-inning heroics and a few coming-out parties for talented players (like Carlos Gonzalez), the actual game play has taken a back seat to shoddy umpiring. Instead of fans marveling at the dominant pitching performances from CC Sabathia and Cliff Lee, they have been left wondering how a play in which two runners were tagged while not standing on the base could only result in one out. Aside from the occasional missed call present throughout the regular season, the umpiring crews have blatantly blown calls in crucial situations that simply cannot be chalked up as “the human element” of the game and subsequently be brushed aside. Even worse, the crews have shown an inability to rectify mistakes made by individual members, and have refused to face the music and admit to their errors.

In an attempt to eradicate this issue and shift the focus back to whether or not the Phillies can defend last year’s title, Major League Baseball has decided that only veteran umpires will call the Fall Classic, a change from the past wherein younger umpires were rewarded with a ticket to the grandest of stages to garner more experience. The six chosen men in blue-Joe West, Gerry Davis, Brian Gorman, Mike Everitt, Jeff Nelson, and Dana DeMuth-will certainly have their work cut out for them, but how will the selection of these umpires affect the potential World Series hurlers?

Using a PITCHf/x database, I enlarged the strike zone to extend beyond the standard horizontal parameters as well as the specific heights for the hitters. From there, the number of called strikes in that inflated zone was measured in relation to the total number of called pitches-balls and called strikes. The higher the percentages in a certain game, the larger the strike zone. For all umpires with at least 10 games behind the plate, the league-average rate of called strikes out of called pitches with these parameters was 63.1 percent. When we bring in the six men calling the shots for the World Series, an interesting trend emerges, in that all but Jeff Nelson had a zone tighter than average for MLB umpires. With Nelson, the sextet averaged 62.2 percent; without him, 61.8 percent, suggesting this selected subset of umpires is much stricter in their definition of the zone. In terms of zone size, Nelson was a pitcher’s best friend, with West not far behind, while Davis and Everitt proved incredibly stingy, especially when stacked up to the league.

Since I measured this zone percentage for each umpire-game, standard deviations can be calculated as a shorthand method for determining consistency at holding true to a zone. As a whole, this group of 75 umpires averaged a standard deviation of 5.8 percent. Our subgrouped sextet boasted a 6.2 percent deviation, suggesting they were slightly less consistent than the league. Most of that falls squarely on the padded shoulders of Dana DeMuth who, at 7.1 percent, was the fourth least consistent of all umpires this season. The most consistent of the group, Mike Everitt, registered a 5.6 percent deviation essentially at the league average, so while the chosen ones were stricter than the league, their zones were also more prone to variation.

In addition to this data, what about more standard numbers aggregated with umpires behind the plate? The table below features the aforementioned inflated zone rate, the walk and strikeout rate of pitchers with these umps behind the dish, and the league ranks:


Umpire         UBBr    Rank   ZONEr    Rank    SOr     Rank
Joe West      9.12%     12    63.54%    27    18.50%    23
Brian Gorman  7.33%     66    62.36%    43    18.14%    30
Mike Everitt  8.28%     37    60.16%    64    18.11%    33
Dana DeMuth   8.94%     14    62.23%    46    17.61%    50
Jeff Nelson   9.30%     7     64.16%    14    16.72%    69
Gerry Davis   8.37%     35    59.64%    66    16.50%    72
MLB Average   8.27%   75 Tot  63.11%  75 Tot  17.95%  75 Total

One very important aspect to note is that the walk or strikeout rates for these umpires do not, on their own, tell us about the tendencies of umpires. For instance, look at Jeff Nelson, who has the largest zone of the group, but also the highest unintentional walk rate. And Gerry Davis, who maintains one of the tightest zones in the game, is essentially league average in unintentional walk rate observed. Running a correlation between a few of these metrics, zone size and walk rate produced an r-value of -0.53, which makes intuitive sense as larger zones should theoretically result in fewer walks being issued. Then again, Jeff Nelson serves as an example of how this will not always ring true. The level of consistency and size of the zone correlated at a rather insignificant 0.12, while consistency and walk rate share virtually no relationship at all.

Reverting back to our ultimate question, how does this information affect those potentially toeing the rubber this week? From the Phillies’ side of things, Cliff Lee will be on the mound in Games One, Four and, potentially, Seven. The Yankees will match Lee with Sabathia every step of the way, utilizing Burnett against Martinez in the second game while combating Hamels in the third game with fellow southpaw Andy Pettitte. The most interesting of the matchups-the one most likely to be affected by the umpires-is Burnett-Martinez, since the former is prone to wild spells and might not benefit from the men in blue if either Davis or Everitt is behind the plate. Similarly, Martinez now relies on acumen and location and the Phillies could be in for a long day if he isn’t getting calls. On the flipside, if Pedro receives the aid of a larger strike zone, expect him to exploit every quarter-inch of the zone in an attempt to carve up the Yankee hitters.

Then again, this subset of umpires was a bit less consistent at holding true to their zones than the league as a whole, so we may very well see DeMuth, the fourth least consistent ump in the league, become strict, while the much more consistent Nelson becomes impossible to gauge. The fact that the selection of these six umpires has become so publicized speaks volumes to the ineptitude that has been on display, and Major League Baseball better hope that their names become less prominent in the minds of fans after this Fall Classic or else face intense scrutiny throughout the entire offseason.

A version of this story originally appeared on ESPN Insider Insider.

You need to be logged in to comment. Login or Subscribe
pbconnection
10/27
This article is so exciting. For my next big thrill, I'll do my taxes.
mattymatty2000
10/27
You, sir, are a jackass.
EJSeidman
10/27
Based on other comments, it seems he isn't a fan of my work. That's fine, I'm not going to win over every reader. The respect I get from all the other readers who understand how hard I work to produce consistent and quality work is more than enough for me.
Oleoay
10/27
You feed me, I pay you in kudos. :)
pbconnection
10/28
My criticism here isn't about the quality of your work. The topic (umpiring) is just something that is of little interest to most baseball fans because they tend to be watching and talking about team and players, not the people who facilitate the games. I'd be just as bored if the article was discussing the differences in grounds crews, equipment managers, and ball boys.
pbconnection
10/28
You can call me whatever you want, but considering that I pay for the privilege of reading these articles, I feel entitled to comment when I don't think articles provide interesting analysis. I suppose I was wrong, considering that so many comments have been left (although I suspect that has much to do with Eric's quick replies, which is certainly commendable) - my interest in baseball has to do with players, not umpiring.
mltepper
10/27
The thing that bothers me is that the MLB is saying it is only using the experienced umpires. Frankly I don't care about experience, I car about having good umpires. I know they are making the assumption that experience=quality, but that isn't a necessarily good assumption.
EJSeidman
10/27
No, no it isn't at all. The umps who blew the major calls were experienced veterans. Being proactive is always better than being reactive. Being reactive is okay as long as you react the right way. This is that third tier - wrong reaction.
joelefkowitz
10/27
>>>From there, the number of called strikes in that inflated zone was measured in relation to the total number of called pitches—balls and called strikes. The higher the percentages in a certain game, the larger the strike zone. For all umpires with at least 10 games behind the plate, the league-average rate of called strikes out of called pitches with these parameters was 63.1 percent. Maybe it's just me, but this was very confusing. At first, it sounds like you are saying you did: (called strikes within just the enlarged portion of the zone) (all called pitches) But that number clearly wouldn't be 63 percent. I guess what you did was: (called strikes within the entire enlarged strikezone) (all called pitches within the entire enlarged strikezone) Is that correct? That doesn't seem to necessarily indicated a larger strike zone...
EJSeidman
10/27
Yes, the latter. It was extended out to be in balls territory, with a bit of weighting to clear up any bias with some guys seeing more balls or in zone pitches than others as well as some umps being behind the plate more often than not. It's not a 100% perfect way to determine the size of a zone but it does the trick for now.
joelefkowitz
10/27
Gotcha, I was wondering about the bias you mention, but I guess you got that under control. You could probably do the inverse of this -- Called Balls within a smaller portion of the strike zone -- and then compare the 2 percentages. The larger the difference between the 2, the more inconsistent (and therefor lousier) the umpire.
EJSeidman
10/27
That's a good idea. I'd also like to explore how movement affects umps. How many times have you seen that two seam fastball tail over the plate but it isn't called? Or the sharp curve that fools both catcher and ump? I bet there is something there. It's just a fine line when evaluating umps that I'd like to cross slowly.
joelefkowitz
10/27
Just out of curiousity, how large (in inches) did you extend the strike zone? I did a couple of queries of my own and to get the same 64% for Jeff Nelson, I had to extend it 2.5 inches in all directions. So I was just wondering if this was close to your number. Here's a few additional notes: Within the standard strike zone, I get Nelson calling strikes on 77.54% of called pitches Within the 2.5 inch-larger-in-all-directions strike zone, I get Nelson at a 64.31% rate. So, a pitch within 2.5 inches of the strike zone has a 13.23% chance of being called a strike by Jeff Nelson. This is basically exactly in line with the league average (13.21%). This is because the rest of the league is better at calling pitches 2.5 inches or less outside of the strike zone a ball, but Nelson is above average at calling pitches in the zone a strike. So perhaps Nelson's zone isn't much bigger than average?
Oleoay
10/27
In the aggregate, how do AL umpires compare to NL umpires in terms of zone size, walk rate, strikeout rate, etc? Also, by pitch f/x pitch location, is it possible to determine the frequency that umpires miss calling their zone?
joelefkowitz
10/27
If you can get me a list of AL/NL umpires, I can try to answer this for you. For the 2nd part of your comment, what exactly do you mean by calling their zone?
EJSeidman
10/27
Yeah, Richard, you can certainly determine frequency of missed calls based on location, as we have the location parameters as well as the result of the pitch, but the definition of zone is less concrete. An umpire's zone is probably a lot like a park factor in that it's more meaningful over a rolling span... unfortunately PITCHf/x is still in its infant stages, so the real questions are a) what is the frequency of missed calls by zone, and b) WHY are they missed? The second part is one that people, for whatever reason, don't seem to care about. I, personally, want to know WHY certain calls are missed, not if they are or not, because if we know the reason we can either alter the training of umps or re-train current ones. If it's movement of certain pitches, where the catcher originally sets up, how the ump sets up, etc these are the questions to ask. Unfortunately we cannot answer them all yet.
doncoffin
10/27
It's a single pool of umpires now--no AL, no NL, just MLB.
Oleoay
10/27
Good point. I'm not sure then about how to aggregate umpires then. Maybe by tenure or umpiring school?
Oleoay
10/27
Defining zone would be tricky. I guess I picture an umpire's zone as the area where at least 50% of balls thrown in that area are called (non swinging) strikes, regardless of velocity, spin. etc. Then, to get into Eric's why, look at the common attributes of missed pitches that an umpire has a high (75%?) chance of calling strikes. I imagine such logic can be used to see how defined an umpire's zone is. However, the height of the strike zone can vary so maybe its best, for an initial test, to look at the width of an umpire's zone (since the width of the plate is fixed) and look at all pitches that are about three feet high (give or take three inches) which should be in the strike zone of any MLB player. Then, if that model holds, work more on defining the height of the zone.
EJSeidman
10/27
Yeah this is what should be discussed - the WHY - and finding out how to do that, which shouldn't be tough, per se, but very time consuming. I'll be honest, though, I know there was a bit of an unspoken hope that analysts wouldn't focus their time and efforts on umpire valuations. I don't know if that has changed a whole lot, but after being at the PITCHf/x Summit this summer I got the sense it is still the same. This line of research came up because of how atrocious they have been so far but I probably will not continue along these lines. Not to say I don't consider it valuable information, but I don't necessarily want to p*ss the wrong people off.
Oleoay
10/27
Kinda an odd statement. Analysts label people who have spent their whole lives playing baseball as replacement-level, chastize managers who spend all night studying matchups and videos for boneheaded moves, complain about dubious scorekeeping calls on errors and wins and saves, yet umpires are somehow immune? Shouldn't umpires be studied with the same rigor as park factors?
llewdor
10/27
I thikn the concern is that in the PitchF/X data is used in ways MLB does like, then MLB will take away the PitchF/X data. This has happened before. A Canadian think-tank decided to use the publicly available standardised test scores (among other things) to evaluate schools in Ontario (both public and private schools). The government on Ontario really didn't like that, so they stopped making the data available. MLB could do the same thing if they don't like how people use PitchF/X (though it blows my mind that anyone would think it wouldn't be used for this). I hope some analyst produces regular umpire stats.
EJSeidman
10/28
Exactly.
Oleoay
10/28
So, if you found strong statistical basis for something that could upset or embarass people, you wouldn't report it? Let's say you were able to use pitch F/X (in conjunction with some video clip of possible doctoring) to identify the difference between a pitcher's normal fastball and a spitball or scuffed ball. You wouldn't report it? If the topic is uninteresting or too tedious, I can see why you wouldn't research umpires. But to be afraid of upsetting one group suggests potential restraint in research or bias. I guess it'll just fall under the "not worth the effort/hassle" category?
EJSeidman
10/28
You're misunderstanding everything that has been said and putting words in my mouth. Not once did I say I would not report something interesting with the data, or anything involving pitchers. In fact, reading the comments, I don't see how any of this could be misconstrued as dealing with anything other than umps. Interesting findings, of course, but since there was some hesitance or suggested restraint with regards to the data being used by analysts to evaluate umpires, I, myself, am not going to walk that fine line.
Oleoay
10/28
I admit that I am misunderstanding you, but I wasn't trying to put words in your mouth. I think the reason for my misunderstanding is that I lacked context. I realize that Pitch F/X is a great, free resource that brings additional insight into the game. Yet, I did not know that the providers of pitch F/X had concerns and suggested restraint about how the data was applied to umpires. All I really know about pitch F/X is from BP. I haven't been to any SABR conventions or pitch F/X conventions (though I heard there was a Hit F/X one a few months back). So, I was unaware about the context (i.e. the hesitancy, the powers that be were touchy) in which pitch F/X was provided to the public, and thus, apologize for my misunderstanding and for making inaccurate guesses at your intentions and your integrity.
EJSeidman
10/28
Richard, no apology necessary, you just seemed like you were picking up steam ;-) and I wanted to slow that locomotive. The stance may have changed but it just seems to me like something with relatively little reward in relation to the risk. But the big thing I wanted to get across here in the thread is that we focus too much on the what and the who and not the why, which is more important.
EJSeidman
10/28
Essentially, the risk greatly outweighs the reward in this instance given that it's fantastic that PITCHf/x is free right now and I'm not going to exert energy into something that could potentially alter that.
Oleoay
10/28
Questec reports were released to the general public. Is this that different?
EJSeidman
10/28
PITCHf/x data should not be taken for granted. It's amazing that we have this for free. If it's something the powers that be were touchy about, count me in as someone who is not going to try and make them angry.
doncoffin
10/27
What I keep waiting for is for someone to leak the ratings of these umpires (as submitted to and maintained by MLB).
EJSeidman
10/27
And in addition to that, HOW they are grading them. I can't imagine--or at least I would hope--more goes into it then a straight, this is the zone, how many did you miss, etc.
doncoffin
10/27
I'm pretty sure the grades involve more than abll-strike calls. But just that would be interesting.
ssimon
10/27
Can we use this data and break it down by home/away and see if pitchers at home get slightly larger zones?
Oleoay
10/28
I'm kind of curious if right handed umpires (who tend to have right-eye dominance) tend to call strikes differently on right-handed batters than left-handed batters (and vice versa).
joelefkowitz
10/28
When the Home Team is Pitching: Pitches inside standard strike zone: 76.1% Called Strikes Pitches within 2.5 inches of the strike zone: 63.1% Incorrect Calls / All Calls Made: 14.95% When the Away Team is Pitching: Pitches inside standard strike zone: 76.1% Called Strikes Pitches within 2.5 inches of the strike zone: 62.7% Incorrect Calls / All Calls Made = 14.48% Inside the strike zone, umps are identical for home/away. Home teams get a small increase in called strikes for pitches near the strike zone.
ahemmer
10/28
Maybe I am looking at this wrong, but technically shouldn't 100% of pitches called within the standard strike zone be called strikes? Is this saying that umpires get about 25% of pitches within the strike zone wrong?
joelefkowitz
10/28
Yes. That's what this is saying. I'd suspect this has a lot to do with pitch f/x's approximations of the top and bottom of the strike zone. Keep in mind though, that the exact percentages aren't extremely important. We just wanted to see the differences between the home team and away team.
terryspen
10/28
A study I would like to see -- albeit time-consuming -- is to look at the umpires' bang-bang base calls to see what percentage they get right. Umpires say they get 98 percent of their calls correct (or something like that) but most of their calls are easy and could be made by a low-rung amateur. I want to know which ones are getting the tough ones right almost always and which ones aren't. Since every game is now televised and has an independent data collector monitoring it, my thought is that these people could mark the close plays and then someone could go back and look at the replays. On the ones that are too close to determine, give the ump the benefit of the doubt.
JeffZimmerman
10/28
I hope to keep this short, I doubt it since it is one of my favorite topics. When looking at home plate umpires we must remember the following points: 1. Umpires are humans 2. Humans produce inconsistent work and make mistakes. 3. For the foreseeable future, computers will not call balls and strikes. With this understanding, umpire's strike zone size and consistency should not generally (there are a few horrible cases) be judged outside of MLB. Instead, what should be looked at is how well pitchers and hitters adapt to each umpire's unique zone. First the strike zone that umpires call isn't the perfect 2 by 2 box. The zone they call is much flatter and is shifted inside depending on the batter's handness. I think umpires zones should be investigated, but not for evaluating the umpires. Instead, their strike zone tendencies should be known so it can be seen how well pitchers and hitters adapt to the different zones, which is actually a great part of the game baseball. Gamblers, who have more at stake than team pride, have been tracking umpire stats for years to see how they effect game scoring. I have looked at a few cases of suspect umpire calls (Milton Bradley, Shane Victorino and Zack Greinke arguing about the zone) and in almost every case the umpires did nothing different than what they do every night. They call there own unique game and the hitters and pitchers that adapt first usually will have an advantage.