May 30, 2013
Bill James, Base Umpires, and the Sabermetric Significance of Checked Swings
When I got home from the SABR Analytics Conference in mid-March, I spent a week or so writing and talking about some of the most interesting things I’d heard there. But there was one particularly intriguing topic that I wasn’t yet ready to write about.
That topic was brought up by Bill James, who's given us more than his fair share of interesting insights. Here’s all the backstory you need to know: James was on an “Analytics Super Panel” with Brian Kenny and Joe Posnanski, and he was asked by an audience member whether there’s any utility for teams in gathering information on home plate umpires. If I’ve embedded it right, this video should play only a short clip from the relevant portion of the panel (59:40-1:00:29). If I’ve embedded it wrong, you can either skip to that section yourself or read the transcript below.
We all know that home plate umpires have different strike zones, and we’re aware that teams pay attention to and try to exploit that information. (Here’s a recent picture of some of Inside Edge’s umpire charts taped up in the Dodgers’ dugout.) But before I heard James say it, I’d never considered that checked-swing strike rates might vary by base ump, or that pitchers might alter their approach accordingly. Of course, there’s no precise definition of what constitutes a swing in the official rulebook, so it makes sense that certain umpires would see swings differently. Some might rule based on whether the batter’s wrists “break,” or whether the barrel of the bat passes in front of the plate. Others might try to assess the batter’s “intent to swing.”
So I went in search of stats that could support what James said. They’re pretty tough to come by. MLBAM, Baseball Info Solutions, and STATS LLC don’t track checked swings. The only major data provider that does is Inside Edge, and even Inside Edge doesn’t track them quite the way you’d want. The company doesn’t record whether an appeal to a base umpire was made, except in rare cases where video replay revealed that an ump had made an obvious wrong call (in the opinion of the Inside Edge scout). It simply records whether there was a checked swing, regardless of outcome. Of course, that’s as subjective a call as the umpire’s is. Here’s how Inside Edge instructs its video scouts to consider checked swings, according to Product, Marketing, and Sales Director Kenny Kendrena:
Insidge Edge tracked check swings on a trial basis in 2008, but began tracking them for all games in 2009. The company recorded 11,807 checked swings in 2012 alone.
Baseball Prospectus acquired Inside Edge’s checked-swing data from 2009-12 and merged it with our pre-existing PITCHf/x database, which made all sorts of analysis possible. Max Marchi ran the numbers through a multi-level logistic regression with random effects—essentially, a “with or without you” model that works with more than two actors. (This is the same technique Max uses to isolate the impact of catcher receiving by correcting for all of the other factors that can influence a ball/strike call.) Max looked at checked swings that resulted in either a swinging strike or a ball, accounting for pitcher, batter, pitch type, count, home plate umpire, and relevant umpire on base (third base ump with a left-handed hitter at the plate, and first base ump with a righty up). I’ll report all of his findings below, but first I want to discuss whether the stats back up Bill James, whose comments inspired the study.
Max split the dataset into even and odd rows and ran a correlation on the extra strike calls by base umpire, weighting for the number of check swings that occurred with each umpire on base. The year-to-year correlation was 0.34—not strong, but tough to dismiss, given the number of data points and the fact that we aren’t even considering whether the home plate umpire appealed. (There’s a stronger correlation between a base ump’s rates at first and third.)
Here’s a scatterplot of checked-swing strike rate on odd-row checked swings vs. checked-swing strike rate on even-row checked swings. (“Swing rate” in the axis labels refers to how often the checked swing was ruled a real swing.) The darker the dot, the larger the sample.
So it does seem that there’s a persistent difference in the rates at which base umpires call checked swing strikes, as James suggested. However, Max found no difference in the percentage of sliders thrown when umpires who call more checked-swing strikes are on base. That means that we can’t confirm James’ contention that pitchers are altering their approaches to take advantage of the rates at which base umps call checked swings strikes. Nor can we disprove it. As Max put it, “I believe his claim might be true, but pitchers try to exploit the knowledge of the base ump tendency so rarely that it would not come out from analysis of the data.” If we tried to look for evidence of particular pitchers or teams tailoring their approaches to these tendencies, or studied only certain situations, we’d be slicing the base-ump samples so small that we’d lose statistical significance, regardless of what the results revealed.
That said, I’m a little skeptical that “almost every team is on top of” umpire checked-swing strike rates. James’ comments certainly suggest that the Red Sox have studied the issue, maybe with more granular information than we obtained from Inside Edge. But the difficulty of obtaining the necessary data, and the modest potential payoff, makes it unlikely that most of the other 29 teams have done the same studies. I asked statistical analysts from a few teams whether they have data on checked swings, and whether they’ve studied checked-swing strike rates. All of them said that they didn’t, and hadn’t. “I don’t know what every team in baseball is doing,” one of them told me. “I do know what a handful of them are doing, and they aren’t doing that.”
Front-office types are under no obligation to tell me the truth, but it makes sense that checked swings wouldn’t be anywhere close to a team’s top priority. Last season, 1.6 percent of pitches led to checked swings. Teams saw, on average, 145 pitches per game (290 for both clubs combined). That means there were an average of 2.3 checked swings per team, per game (4.6 for both clubs combined). A team could steal an extra strike from time to time if it knew the probabilities of checked-swing strikes by base umpire, but since the third parties that teams typically get data from aren’t providing that data, they’d have to collect it themselves or get Inside Edge’s info and then watch every checked swing to see whether there was an appeal. (There's no technology currently installed around the majors that tracks the path of the bat.) That would take some serious time, and few teams have a large enough army of interns to spare one for that sort of project.
Let’s take a look at the rest of what the checked-swing information has to offer:
Here's Meals on the left, calling a strike on Alex Liddi last May, and the more lenient Cuzzi on the right, letting Matt Holliday off easy last April. (Liddi did appear to go around farther, so this isn't the best example of the disparity between the two.)
According to Max’s model, the difference between the highest and lowest is only about 10 percent, but the true difference is likely more dramatic. These results are somewhat diluted, since we can’t focus only on checked swings that were appealed.
And here are the increases and declines broken down by home plate umpire:
Brian Runge is unlikely to call a checked-swing a strike wherever he’s stationed, while Joe West is much more likely to do so when he’s behind the plate than when he’s on the corners.
These differences are smaller, but it makes sense that there would be some distinction, since home plate umpires can decide to call checked-swing strikes themselves. Their ball calls can be appealed by the catcher, but their strike calls can’t by appealed by the batter.
Where checked swings happen
And here’s a GIF with both combined:
The highest concentration of checked-swing strikes on those plots is in the same region to hitters of both handedness: down and in to lefties and down and away to righties. As it turns out, pitcher handedness is a bigger factor than batter handedness. Here are the checked swings to righties, first against left-handed pitchers and then against right-handed pitchers:
And here are the lefty batters against pitchers of both handednesses:
Now we see the difference.
Finally, a breakdown of checked-swing rates by location inside or outside of the strike zone:
Pitches outside the typical called strike zone are about five times more likely to be checked swings than pitches inside the typical called strike zone. That makes sense: hitters hesitate less when deciding whether to swing at strikes.
This is pretty consistent with the “compassionate umpire” effect John Walsh discovered when he looked at the size of the called strike zone by count. Umpires are much less likely to call strikes on 0-2 than they are on 3-0; they’re also less likely to call checked swings swinging strikes.
And here’s a related table that tells us the change in the probability that a checked swing will be called a strike, broken down by pitch type instead of count, and with four-seam fastballs as our baseline:
These results also make sense. A checked swing at a cutter or slider is more likely to be called a strike; a checked swing at a fastball is less likely to be called a strike.
Also unsurprisingly, the batters with the lowest checked-swing rates are guys who don’t swing a lot and tend to make contact. The only surprise is that Jeff Keppinger couldn’t quite crack the top 10 list (he finished 14th).
Wes Helms was the only hitter to see at least 1000 pitches from 2009-2012—he saw 2658—without a single checked swing. Say what you will about Helms’ offensive abilities, but the man has no doubts about whether he wants to swing. (You might notice that that list is lousy with Marlins, which could potentially point to some sort of scorer bias.)
And here are the batters whose checked swings were called strikes more and less often than expected, after controlling for other factors:
And low-swing/chase/whiff guys don’t:
Oviedo threw 3187 pitches without seeing a single checked swing. And here are the pitchers whose induced swings were called strikes more and less often than expected, after controlling for other factors:
I’ll leave it to you to decide whether there are any perceptible patterns among the arms on those lists.
There was also one checked swing on a pitchout, courtesy of Nick Punto last June:
The hit-and-run was on, but Ryan Kalish stayed put, having either anticipated the pitchout or missed the sign. Punto looked silly, and home plate umpire Sam Holbrook pointed at him to make sure the whole ballpark saw.
The pitchout to Punto was also the farthest outside of any pitch that induced a checked swing. We’ll wrap up with some other extremes. Here’s the highest pitch that induced a checked swing, a 3-2 offering from Brandon Beachy to Brett Lawrie last June:
And here’s the lowest (well, the lowest viewable on video), an October 2010 swing by Ryan Howard that was ruled a strike by third-base umpire Jim Wolf:
As for the inside offerings: there were 96 checked swings on pitches that hit batters.
Thanks to Max Marchi and Ryan Lind for research assistance, and to Inside Edge for checked-swing data.