CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

<< Previous Article
Premium Article World Series Prospectu... (10/27)
<< Previous Column
Premium Article World Series Prospectu... (10/27)
Next Column >>
Premium Article World Series Prospectu... (10/28)
Next Article >>
Premium Article Ahead in the Count: Th... (10/28)

October 27, 2009

World Series Prospectus

The Umpires

by Eric Seidman

the archives are now free.

All Baseball Prospectus Premium and Fantasy articles more than a year old are now free as a thank you to the entire Internet for making our work possible.

Not a subscriber? Get exclusive content like this delivered hot to your inbox every weekday. Click here for more information on Baseball Prospectus subscriptions or use the buttons to the right to subscribe and get instant access to the best baseball content on the web.

Subscribe for $4.95 per month
Recurring subscription - cancel anytime.


a 33% savings over the monthly price!

Purchase a $39.95 gift subscription
a 33% savings over the monthly price!

Already a subscriber? Click here and use the blue login bar to log in.

In a postseason featuring late-inning heroics and a few coming-out parties for talented players (like Carlos Gonzalez), the actual game play has taken a back seat to shoddy umpiring. Instead of fans marveling at the dominant pitching performances from CC Sabathia and Cliff Lee, they have been left wondering how a play in which two runners were tagged while not standing on the base could only result in one out. Aside from the occasional missed call present throughout the regular season, the umpiring crews have blatantly blown calls in crucial situations that simply cannot be chalked up as "the human element" of the game and subsequently be brushed aside. Even worse, the crews have shown an inability to rectify mistakes made by individual members, and have refused to face the music and admit to their errors.

In an attempt to eradicate this issue and shift the focus back to whether or not the Phillies can defend last year's title, Major League Baseball has decided that only veteran umpires will call the Fall Classic, a change from the past wherein younger umpires were rewarded with a ticket to the grandest of stages to garner more experience. The six chosen men in blue-Joe West, Gerry Davis, Brian Gorman, Mike Everitt, Jeff Nelson, and Dana DeMuth-will certainly have their work cut out for them, but how will the selection of these umpires affect the potential World Series hurlers?

Using a PITCHf/x database, I enlarged the strike zone to extend beyond the standard horizontal parameters as well as the specific heights for the hitters. From there, the number of called strikes in that inflated zone was measured in relation to the total number of called pitches-balls and called strikes. The higher the percentages in a certain game, the larger the strike zone. For all umpires with at least 10 games behind the plate, the league-average rate of called strikes out of called pitches with these parameters was 63.1 percent. When we bring in the six men calling the shots for the World Series, an interesting trend emerges, in that all but Jeff Nelson had a zone tighter than average for MLB umpires. With Nelson, the sextet averaged 62.2 percent; without him, 61.8 percent, suggesting this selected subset of umpires is much stricter in their definition of the zone. In terms of zone size, Nelson was a pitcher's best friend, with West not far behind, while Davis and Everitt proved incredibly stingy, especially when stacked up to the league.

Since I measured this zone percentage for each umpire-game, standard deviations can be calculated as a shorthand method for determining consistency at holding true to a zone. As a whole, this group of 75 umpires averaged a standard deviation of 5.8 percent. Our subgrouped sextet boasted a 6.2 percent deviation, suggesting they were slightly less consistent than the league. Most of that falls squarely on the padded shoulders of Dana DeMuth who, at 7.1 percent, was the fourth least consistent of all umpires this season. The most consistent of the group, Mike Everitt, registered a 5.6 percent deviation essentially at the league average, so while the chosen ones were stricter than the league, their zones were also more prone to variation.

In addition to this data, what about more standard numbers aggregated with umpires behind the plate? The table below features the aforementioned inflated zone rate, the walk and strikeout rate of pitchers with these umps behind the dish, and the league ranks:


Umpire         UBBr    Rank   ZONEr    Rank    SOr     Rank
Joe West      9.12%     12    63.54%    27    18.50%    23
Brian Gorman  7.33%     66    62.36%    43    18.14%    30
Mike Everitt  8.28%     37    60.16%    64    18.11%    33
Dana DeMuth   8.94%     14    62.23%    46    17.61%    50
Jeff Nelson   9.30%     7     64.16%    14    16.72%    69
Gerry Davis   8.37%     35    59.64%    66    16.50%    72
MLB Average   8.27%   75 Tot  63.11%  75 Tot  17.95%  75 Total

One very important aspect to note is that the walk or strikeout rates for these umpires do not, on their own, tell us about the tendencies of umpires. For instance, look at Jeff Nelson, who has the largest zone of the group, but also the highest unintentional walk rate. And Gerry Davis, who maintains one of the tightest zones in the game, is essentially league average in unintentional walk rate observed. Running a correlation between a few of these metrics, zone size and walk rate produced an r-value of -0.53, which makes intuitive sense as larger zones should theoretically result in fewer walks being issued. Then again, Jeff Nelson serves as an example of how this will not always ring true. The level of consistency and size of the zone correlated at a rather insignificant 0.12, while consistency and walk rate share virtually no relationship at all.

Reverting back to our ultimate question, how does this information affect those potentially toeing the rubber this week? From the Phillies' side of things, Cliff Lee will be on the mound in Games One, Four and, potentially, Seven. The Yankees will match Lee with Sabathia every step of the way, utilizing Burnett against Martinez in the second game while combating Hamels in the third game with fellow southpaw Andy Pettitte. The most interesting of the matchups-the one most likely to be affected by the umpires-is Burnett-Martinez, since the former is prone to wild spells and might not benefit from the men in blue if either Davis or Everitt is behind the plate. Similarly, Martinez now relies on acumen and location and the Phillies could be in for a long day if he isn't getting calls. On the flipside, if Pedro receives the aid of a larger strike zone, expect him to exploit every quarter-inch of the zone in an attempt to carve up the Yankee hitters.

Then again, this subset of umpires was a bit less consistent at holding true to their zones than the league as a whole, so we may very well see DeMuth, the fourth least consistent ump in the league, become strict, while the much more consistent Nelson becomes impossible to gauge. The fact that the selection of these six umpires has become so publicized speaks volumes to the ineptitude that has been on display, and Major League Baseball better hope that their names become less prominent in the minds of fans after this Fall Classic or else face intense scrutiny throughout the entire offseason.

A version of this story originally appeared on ESPN Insider Insider.

Eric Seidman is an author of Baseball Prospectus. 
Click here to see Eric's other articles. You can contact Eric by clicking here

Related Content:  The Who,  Umpires,  Jeff Nelson

40 comments have been left for this article. (Click to hide comments)

BP Comment Quick Links

pbconnection
Other readers have rated this comment below the viewing threshold. Click here to view anyway.

This article is so exciting. For my next big thrill, I'll do my taxes.

Oct 27, 2009 12:30 PM
rating: -21
 
Matt Kory

You, sir, are a jackass.

Oct 27, 2009 15:49 PM
rating: 2
 
BP staff member Eric Seidman
BP staff

Based on other comments, it seems he isn't a fan of my work. That's fine, I'm not going to win over every reader. The respect I get from all the other readers who understand how hard I work to produce consistent and quality work is more than enough for me.

Oct 27, 2009 15:54 PM
 
Richard Bergstrom

You feed me, I pay you in kudos. :)

Oct 27, 2009 16:09 PM
rating: 0
 
pbconnection
Other readers have rated this comment below the viewing threshold. Click here to view anyway.

My criticism here isn't about the quality of your work. The topic (umpiring) is just something that is of little interest to most baseball fans because they tend to be watching and talking about team and players, not the people who facilitate the games. I'd be just as bored if the article was discussing the differences in grounds crews, equipment managers, and ball boys.

Oct 27, 2009 20:18 PM
rating: -5
 
pbconnection
Other readers have rated this comment below the viewing threshold. Click here to view anyway.

You can call me whatever you want, but considering that I pay for the privilege of reading these articles, I feel entitled to comment when I don't think articles provide interesting analysis.

I suppose I was wrong, considering that so many comments have been left (although I suspect that has much to do with Eric's quick replies, which is certainly commendable) - my interest in baseball has to do with players, not umpiring.

Oct 27, 2009 20:13 PM
rating: -4
 
mltepper

The thing that bothers me is that the MLB is saying it is only using the experienced umpires. Frankly I don't care about experience, I car about having good umpires. I know they are making the assumption that experience=quality, but that isn't a necessarily good assumption.

Oct 27, 2009 12:40 PM
rating: 2
 
BP staff member Eric Seidman
BP staff

No, no it isn't at all. The umps who blew the major calls were experienced veterans. Being proactive is always better than being reactive. Being reactive is okay as long as you react the right way. This is that third tier - wrong reaction.

Oct 27, 2009 12:41 PM
 
jlefty

>>>From there, the number of called strikes in that inflated zone was measured in relation to the total number of called pitches—balls and called strikes. The higher the percentages in a certain game, the larger the strike zone. For all umpires with at least 10 games behind the plate, the league-average rate of called strikes out of called pitches with these parameters was 63.1 percent.

Maybe it's just me, but this was very confusing. At first, it sounds like you are saying you did:

(called strikes within just the enlarged portion of the zone)
(all called pitches)

But that number clearly wouldn't be 63 percent. I guess what you did was:

(called strikes within the entire enlarged strikezone)
(all called pitches within the entire enlarged strikezone)

Is that correct?

That doesn't seem to necessarily indicated a larger strike zone...

Oct 27, 2009 12:55 PM
rating: 0
 
BP staff member Eric Seidman
BP staff

Yes, the latter. It was extended out to be in balls territory, with a bit of weighting to clear up any bias with some guys seeing more balls or in zone pitches than others as well as some umps being behind the plate more often than not. It's not a 100% perfect way to determine the size of a zone but it does the trick for now.

Oct 27, 2009 13:09 PM
 
jlefty

Gotcha, I was wondering about the bias you mention, but I guess you got that under control.

You could probably do the inverse of this -- Called Balls within a smaller portion of the strike zone -- and then compare the 2 percentages. The larger the difference between the 2, the more inconsistent (and therefor lousier) the umpire.

Oct 27, 2009 14:12 PM
rating: 0
 
BP staff member Eric Seidman
BP staff

That's a good idea. I'd also like to explore how movement affects umps. How many times have you seen that two seam fastball tail over the plate but it isn't called? Or the sharp curve that fools both catcher and ump? I bet there is something there. It's just a fine line when evaluating umps that I'd like to cross slowly.

Oct 27, 2009 14:43 PM
 
jlefty

Just out of curiousity, how large (in inches) did you extend the strike zone? I did a couple of queries of my own and to get the same 64% for Jeff Nelson, I had to extend it 2.5 inches in all directions. So I was just wondering if this was close to your number.

Here's a few additional notes:

Within the standard strike zone, I get Nelson calling strikes on 77.54% of called pitches

Within the 2.5 inch-larger-in-all-directions strike zone, I get Nelson at a 64.31% rate.

So, a pitch within 2.5 inches of the strike zone has a 13.23% chance of being called a strike by Jeff Nelson. This is basically exactly in line with the league average (13.21%). This is because the rest of the league is better at calling pitches 2.5 inches or less outside of the strike zone a ball, but Nelson is above average at calling pitches in the zone a strike.

So perhaps Nelson's zone isn't much bigger than average?

Oct 27, 2009 15:18 PM
rating: 2
 
Richard Bergstrom

In the aggregate, how do AL umpires compare to NL umpires in terms of zone size, walk rate, strikeout rate, etc?

Also, by pitch f/x pitch location, is it possible to determine the frequency that umpires miss calling their zone?

Oct 27, 2009 15:23 PM
rating: 0
 
jlefty

If you can get me a list of AL/NL umpires, I can try to answer this for you.

For the 2nd part of your comment, what exactly do you mean by calling their zone?

Oct 27, 2009 15:29 PM
rating: 0
 
BP staff member Eric Seidman
BP staff

Yeah, Richard, you can certainly determine frequency of missed calls based on location, as we have the location parameters as well as the result of the pitch, but the definition of zone is less concrete. An umpire's zone is probably a lot like a park factor in that it's more meaningful over a rolling span... unfortunately PITCHf/x is still in its infant stages, so the real questions are a) what is the frequency of missed calls by zone, and b) WHY are they missed?

The second part is one that people, for whatever reason, don't seem to care about. I, personally, want to know WHY certain calls are missed, not if they are or not, because if we know the reason we can either alter the training of umps or re-train current ones.

If it's movement of certain pitches, where the catcher originally sets up, how the ump sets up, etc these are the questions to ask. Unfortunately we cannot answer them all yet.

Oct 27, 2009 15:38 PM
 
doncoffin
(422)

It's a single pool of umpires now--no AL, no NL, just MLB.

Oct 27, 2009 15:40 PM
rating: 0
 
Richard Bergstrom

Good point. I'm not sure then about how to aggregate umpires then. Maybe by tenure or umpiring school?

Oct 27, 2009 16:07 PM
rating: 0
 
Richard Bergstrom

Defining zone would be tricky. I guess I picture an umpire's zone as the area where at least 50% of balls thrown in that area are called (non swinging) strikes, regardless of velocity, spin. etc. Then, to get into Eric's why, look at the common attributes of missed pitches that an umpire has a high (75%?) chance of calling strikes. I imagine such logic can be used to see how defined an umpire's zone is.

However, the height of the strike zone can vary so maybe its best, for an initial test, to look at the width of an umpire's zone (since the width of the plate is fixed) and look at all pitches that are about three feet high (give or take three inches) which should be in the strike zone of any MLB player. Then, if that model holds, work more on defining the height of the zone.

Oct 27, 2009 16:05 PM
rating: 0
 
BP staff member Eric Seidman
BP staff

Yeah this is what should be discussed - the WHY - and finding out how to do that, which shouldn't be tough, per se, but very time consuming. I'll be honest, though, I know there was a bit of an unspoken hope that analysts wouldn't focus their time and efforts on umpire valuations. I don't know if that has changed a whole lot, but after being at the PITCHf/x Summit this summer I got the sense it is still the same. This line of research came up because of how atrocious they have been so far but I probably will not continue along these lines. Not to say I don't consider it valuable information, but I don't necessarily want to p*ss the wrong people off.

Oct 27, 2009 16:11 PM
 
Richard Bergstrom

Kinda an odd statement. Analysts label people who have spent their whole lives playing baseball as replacement-level, chastize managers who spend all night studying matchups and videos for boneheaded moves, complain about dubious scorekeeping calls on errors and wins and saves, yet umpires are somehow immune?

Shouldn't umpires be studied with the same rigor as park factors?

Oct 27, 2009 16:18 PM
rating: 3
 
Evan
(47)

I thikn the concern is that in the PitchF/X data is used in ways MLB does like, then MLB will take away the PitchF/X data.

This has happened before. A Canadian think-tank decided to use the publicly available standardised test scores (among other things) to evaluate schools in Ontario (both public and private schools). The government on Ontario really didn't like that, so they stopped making the data available.

MLB could do the same thing if they don't like how people use PitchF/X (though it blows my mind that anyone would think it wouldn't be used for this).

I hope some analyst produces regular umpire stats.

Oct 27, 2009 16:36 PM
rating: 3
 
BP staff member Eric Seidman
BP staff

Exactly.

Oct 27, 2009 17:17 PM
 
Richard Bergstrom

So, if you found strong statistical basis for something that could upset or embarass people, you wouldn't report it? Let's say you were able to use pitch F/X (in conjunction with some video clip of possible doctoring) to identify the difference between a pitcher's normal fastball and a spitball or scuffed ball. You wouldn't report it?

If the topic is uninteresting or too tedious, I can see why you wouldn't research umpires. But to be afraid of upsetting one group suggests potential restraint in research or bias.

I guess it'll just fall under the "not worth the effort/hassle" category?

Oct 27, 2009 17:36 PM
rating: -1
 
BP staff member Eric Seidman
BP staff

You're misunderstanding everything that has been said and putting words in my mouth. Not once did I say I would not report something interesting with the data, or anything involving pitchers. In fact, reading the comments, I don't see how any of this could be misconstrued as dealing with anything other than umps. Interesting findings, of course, but since there was some hesitance or suggested restraint with regards to the data being used by analysts to evaluate umpires, I, myself, am not going to walk that fine line.

Oct 27, 2009 17:39 PM
 
Richard Bergstrom

I admit that I am misunderstanding you, but I wasn't trying to put words in your mouth. I think the reason for my misunderstanding is that I lacked context. I realize that Pitch F/X is a great, free resource that brings additional insight into the game. Yet, I did not know that the providers of pitch F/X had concerns and suggested restraint about how the data was applied to umpires.

All I really know about pitch F/X is from BP. I haven't been to any SABR conventions or pitch F/X conventions (though I heard there was a Hit F/X one a few months back). So, I was unaware about the context (i.e. the hesitancy, the powers that be were touchy) in which pitch F/X was provided to the public, and thus, apologize for my misunderstanding and for making inaccurate guesses at your intentions and your integrity.

Oct 27, 2009 20:07 PM
rating: -2
 
BP staff member Eric Seidman
BP staff

Richard, no apology necessary, you just seemed like you were picking up steam ;-) and I wanted to slow that locomotive. The stance may have changed but it just seems to me like something with relatively little reward in relation to the risk. But the big thing I wanted to get across here in the thread is that we focus too much on the what and the who and not the why, which is more important.

Oct 27, 2009 20:41 PM
 
BP staff member Eric Seidman
BP staff

Essentially, the risk greatly outweighs the reward in this instance given that it's fantastic that PITCHf/x is free right now and I'm not going to exert energy into something that could potentially alter that.

Oct 27, 2009 17:47 PM
 
Richard Bergstrom

Questec reports were released to the general public. Is this that different?

Oct 27, 2009 17:24 PM
rating: 0
 
BP staff member Eric Seidman
BP staff

PITCHf/x data should not be taken for granted. It's amazing that we have this for free. If it's something the powers that be were touchy about, count me in as someone who is not going to try and make them angry.

Oct 27, 2009 17:37 PM
 
doncoffin
(422)

What I keep waiting for is for someone to leak the ratings of these umpires (as submitted to and maintained by MLB).

Oct 27, 2009 15:42 PM
rating: 0
 
BP staff member Eric Seidman
BP staff

And in addition to that, HOW they are grading them. I can't imagine--or at least I would hope--more goes into it then a straight, this is the zone, how many did you miss, etc.

Oct 27, 2009 15:55 PM
 
doncoffin
(422)

I'm pretty sure the grades involve more than abll-strike calls. But just that would be interesting.

Oct 27, 2009 16:13 PM
rating: 0
 
Scott D. Simon

Can we use this data and break it down by home/away and see if pitchers at home get slightly larger zones?

Oct 27, 2009 16:49 PM
rating: 1
 
Richard Bergstrom

I'm kind of curious if right handed umpires (who tend to have right-eye dominance) tend to call strikes differently on right-handed batters than left-handed batters (and vice versa).

Oct 27, 2009 17:23 PM
rating: 1
 
jlefty

When the Home Team is Pitching:

Pitches inside standard strike zone: 76.1% Called Strikes
Pitches within 2.5 inches of the strike zone: 63.1%
Incorrect Calls / All Calls Made: 14.95%


When the Away Team is Pitching:

Pitches inside standard strike zone: 76.1% Called Strikes
Pitches within 2.5 inches of the strike zone: 62.7%
Incorrect Calls / All Calls Made = 14.48%


Inside the strike zone, umps are identical for home/away. Home teams get a small increase in called strikes for pitches near the strike zone.

Oct 28, 2009 07:32 AM
rating: 1
 
ahemmer

Maybe I am looking at this wrong, but technically shouldn't 100% of pitches called within the standard strike zone be called strikes? Is this saying that umpires get about 25% of pitches within the strike zone wrong?

Oct 28, 2009 08:01 AM
rating: 0
 
jlefty

Yes. That's what this is saying. I'd suspect this has a lot to do with pitch f/x's approximations of the top and bottom of the strike zone. Keep in mind though, that the exact percentages aren't extremely important. We just wanted to see the differences between the home team and away team.

Oct 28, 2009 10:12 AM
rating: 0
 
terryspen

A study I would like to see -- albeit time-consuming -- is to look at the umpires' bang-bang base calls to see what percentage they get right. Umpires say they get 98 percent of their calls correct (or something like that) but most of their calls are easy and could be made by a low-rung amateur. I want to know which ones are getting the tough ones right almost always and which ones aren't.

Since every game is now televised and has an independent data collector monitoring it, my thought is that these people could mark the close plays and then someone could go back and look at the replays. On the ones that are too close to determine, give the ump the benefit of the doubt.

Oct 28, 2009 07:05 AM
rating: 0
 
JeffZimmerman

I hope to keep this short, I doubt it since it is one of my favorite topics.

When looking at home plate umpires we must remember the following points:

1. Umpires are humans
2. Humans produce inconsistent work and make mistakes.
3. For the foreseeable future, computers will not call balls and strikes.

With this understanding, umpire's strike zone size and consistency should not generally (there are a few horrible cases) be judged outside of MLB. Instead, what should be looked at is how well pitchers and hitters adapt to each umpire's unique zone.

First the strike zone that umpires call isn't the perfect 2 by 2 box. The zone they call is much flatter and is shifted inside depending on the batter's handness.

I think umpires zones should be investigated, but not for evaluating the umpires. Instead, their strike zone tendencies should be known so it can be seen how well pitchers and hitters adapt to the different zones, which is actually a great part of the game baseball. Gamblers, who have more at stake than team pride, have been tracking umpire stats for years to see how they effect game scoring.

I have looked at a few cases of suspect umpire calls (Milton Bradley, Shane Victorino and Zack Greinke arguing about the zone) and in almost every case the umpires did nothing different than what they do every night. They call there own unique game and the hitters and pitchers that adapt first usually will have an advantage.

Oct 28, 2009 08:32 AM
rating: 0
 
You must be a Premium subscriber to post a comment.
Not a subscriber? Sign up today!
<< Previous Article
Premium Article World Series Prospectu... (10/27)
<< Previous Column
Premium Article World Series Prospectu... (10/27)
Next Column >>
Premium Article World Series Prospectu... (10/28)
Next Article >>
Premium Article Ahead in the Count: Th... (10/28)

RECENTLY AT BASEBALL PROSPECTUS
Premium Article The Prospectus Hit List: Friday, April 24
Premium Article Daisy Cutter: Carlos Rodon and the White Sox...
Eyewitness Accounts: April 24, 2015
Premium Article Rubbing Mud: Of Signals and Sausages
Raising Aces: Debut Ante: Raisel Iglesias
Painting the Black: No D In Desmond?
BP Bronx

MORE FROM OCTOBER 27, 2009
Premium Article World Series Prospectus: Four Keys
Premium Article Kiss'Em Goodbye: Los Angeles Angels of Anahe...

MORE BY ERIC SEIDMAN
2009-11-12 - Premium Article Checking the Numbers: Extending the Discipli...
2009-11-06 - Premium Article Checking the Numbers: Detecting Discipline
2009-10-29 - Premium Article Checking the Numbers: Quick Change Artistry
2009-10-27 - Premium Article World Series Prospectus: The Umpires
2009-10-22 - Premium Article Checking the Numbers: Crossing Over
2009-10-16 - Premium Article Playoff Prospectus: Yankees versus Angels LC...
2009-10-07 - Premium Article Playoff Prospectus: Phillies versus Rockies ...
More...

MORE WORLD SERIES PROSPECTUS
2010-10-27 - Premium Article World Series Prospectus: Game One Pitching M...
2010-10-26 - World Series Prospectus: World Series Previe...
2009-10-28 - Premium Article World Series Prospectus: Yankees versus Phil...
2009-10-27 - Premium Article World Series Prospectus: The Umpires
2009-10-27 - Premium Article World Series Prospectus: Four Keys
2009-10-26 - Premium Article World Series Prospectus: The Weather
2008-10-21 - World Series Prospectus: Rays versus Phillie...
More...