June 21, 2013
To Sleep, Perchance to Swing
Over the last few weeks, a press release has been making the rounds. It’s a persuasive press release that reports some interesting research, and wherever it goes, it produces a post. There’s just one problem: the research it reports is a little misleading.
The press release, which you can view here, was put out by the American Academy of Sleep Medicine. It summarizes the results of a recent study on the effect of fatigue on strike-zone judgment. The source of the study is a research abstract published in an online supplement of the journal SLEEP—you can access the abstract (PDF) on page A408 here—and the principal investigator behind it is Scott Kutscher, MD, an assistant professor of sleep and neurology at Vanderbilt University Medical Center.
The study’s conclusions are simple. Here’s one of the quotes from Kutscher in the press release:
To make things even more interesting, the study promises a PED connection. As the abstract notes, “This effect became more pronounced following a ban on stimulants in 2005.” Or as one of the articles about the study says:
So, to summarize the study’s claims (and strong suggestions): Plate discipline declines as the season goes on. This decline stems from increased fatigue. And the effect has become more pronounced since baseball banned amphetamines, which players used to take to counteract fatigue.
There’s a reason—actually, there are a few reasons—why this study has been spreading so fast. It’s about baseball, which means that it appeals to a wider audience than your average research abstract. It reports an intriguing discovery with real implications: if fatigue is making batters worse, then perhaps teams could improve their performance by taking steps to minimize that strain. It’s tied to PEDs, and it appears to isolate a performance-enhancing effect, which is notoriously difficult to do. It comes from a reputable source—a doctor/professor who specializes in sleep and neurology at a university you’ve heard of. And it feels like a fairly intuitive finding—we’ve all experienced the fatigue associated with travel and a busy schedule, and we know how lack of sleep can impair our performance. Why shouldn’t it impair baseball players’ performance, too?
Thanks to that potent confluence of factors, there’s a pretty good chance that you’ve seen something about this if you read about scientific research or baseball on the internet. On the science side, we have ScienceDaily, Medical Daily, PopSci , Medline Plus, and Medpage Today. On the baseball side, we have Grantland, Hardball Talk, Baseball Musings, Reddit, and The Boston Herald. And then there are the general news sources, like the Nashville Business Journal and US News & World Report. (And so on—I’ve been told that the New York Times will soon publish its own report on the research. *Update* Here it is.) Some of those posts merely regurgitate or link to the press release, while others interpret and editorialize. But none of them raises any serious doubts about the study’s methodology or conclusions.
So, is this study misleading? There are three aspects of the research that deserve some scrutiny. First, there’s the claim that plate discipline erodes throughout the season. Second, there’s the claim that the cause of that erosion is fatigue. And third, there’s the claim that baseball’s amphetamine ban has exacerbated the effect of this fatigue.
Let’s tackle each of those in turn.
Does plate discipline erode throughout the season?
We have two publicly available sources of plate discipline data: PITCHf/x, which powers plate discipline stats at both BP and FanGraphs, and Baseball Info Solutions, which licenses information to FanGraphs. Kutscher’s study draws on the BIS data, which is collected by stringers from video and extends a few years into the pre-PITCHf/x era. The abstract reports a straight April-to-September comparison, but the researchers also looked at data from each intervening month.
We were able to verify that the April and September O-swing rates reported in the abstract are accurate, according to the BIS information at FanGraphs. And the same trend appears in BP’s PITCHf/x-based plate discipline data, although the numbers aren’t the same: from 2008 to 12, league-wide O-swing rates were 27.6 percent in April and 29.0 percent in September.
But that “linear increase” breaks down when you look at the monthly splits:
O-swing rates were at their lowest in April. But (disregarding the smaller sample size of October) they peaked in July, then declined slightly and held steady in August and September. If fatigue were impairing plate discipline, we’d expect the rate to be higher in August than July, and higher in September than July and August. But that’s not what we see.
Moreover, reporting O-swing rates alone offers an incomplete picture of plate discipline. In an email exchange with Russell Carleton, Kutscher said that the researchers focused on O-swing because swinging at a ball is always a bad idea. (Pablo Sandoval begs to differ.) Judgment and reaction time are known to be affected by sleep deprivation, so O-swing rate would seem to be the plate discipline stat most susceptible to fatigue.
Except, well…what do we see when we look at not just O-swing rate, but also Z-swing rate (the percentage of pitches inside the strike zone that drew swings) and overall swing rate? Again from 2008-12:
It’s not just O-swing rate that’s higher in September than April. Z-swing rate follows the same trajectory. So hitters don’t just swing at pitches outside the zone less often in April—they swing at all pitches less often in April. And swinging at strikes is generally a good decision, or at least not a bad one. Why would fatigue cause batters to swing at more pitches inside the strike zone? When you examine the other swing rate stats, the narrative that batters make worse decisions about when to swing later in the season doesn’t hold up.
The correlation between monthly O-swing and Z-swing rates is 0.96. In other words, the two move in tandem. There’s no evidence that one could reduce fatigue-related swings at pitches outside the strike zone without also reducing swings at pitches inside the strike zone.
One last thing before we move on to the study’s second claim. Colin Wyers pulled the monthly BIS-based plate discipline split stats for April and September—the same data set that the researchers drew on—from FanGraphs, then took the difference in overall swing rate and O-swing rate for each season. Here’s what that looks like:
As Colin put it, “the reported effect essentially evaporates if you look at Swing% instead of O-Swing%.”
Is fatigue the cause of the erosion in plate discipline over the course of the season?
We can’t prove that it doesn’t, though you’d have to come up with a convoluted argument to suggest that it does. But there are factors other than fatigue that could influence swing rates. For one thing, the player pool in April isn’t the same as the player pool in September. Rosters expand in September, and teams give a higher percentage of their plate appearances to rookies who weren’t around earlier in the season. Those rookies might tend to be free swingers, which could skew the stats. The abstract doesn’t offer any evidence that the researchers made an attempt to control for this.
Colin Wyers did make an attempt to control for this, and as it happens, it doesn’t make much of a difference. Even after controlling for changes in the player pool, swing rates are still at their lowest in April, higher in September, and highest in July. (You can see the controlled results and a brief explanation of the methodology here.) Teams give a higher percentage of their innings in September to rookies, too, and perhaps those rookie pitchers are less capable of inducing swings outside the zone. So maybe it all evens out. (Which brings up another question: If hitters are fatigued later in the season, mightn't pitchers be fatigued, too?)
However, Colin’s attempt to control for changes in the player pool doesn’t control for changes in the players. And there’s one important change in player performance that could influence plate discipline: pitchers don’t throw as hard in April as they do in subsequent months. The average four-seam fastball reaches peak velocity in July, as temperatures rise and pitchers stretch out their arms. A graph of average fastball speed by month looks a lot like a graph of swing rate by month. When pitchers are throwing harder, hitters have to decide whether to swing earlier. Maybe they’re more likely to swing when they have less time to weigh their options.
Another possible cause: as Max Marchi’s research earlier this year suggested, hitters may well be “ahead” of pitchers earlier in the season, maybe because pitchers need more time to build up their arms and return to peak playing shape after a winter off. That could be a factor here, though it wouldn’t necessarily explain why hitters swing at fewer pitches inside the zone in April. Or maybe there’s something about the weather that makes hitters more or less likely to swing, aside from its effect on pitch speed. When the weather is warm, the ball travels farther, which gives hitters more incentive to swing.
Even if there were a clear, progressive decline in plate discipline over the course of a season, it would still require quite a leap to conclude that fatigue was the cause. To do that, you'd have to assert that something that did change (plate discipline) was linked to something that was only speculated to change (level of fatigue), and was also very difficult to verify. There might be dozens of other factors that one could pinpoint as the cause just as plausibly. Fatigue might be a sensible explanation, but to single it out as by far the most likely smacks of just-so story.
Has the amphetamine ban made players more fatigued?
Unfortunately, we can’t check to see whether the PITCHf/x data from 2002 to 2005 shows the same trend as the BIS data, because PITCHf/x data from 2002 to 2005 doesn’t exist. But as Colin Wyers has documented (here and here), the BIS plate discipline data from the pre-PITCHf/x era is somewhat inconsistent, and subject to certain biases. For instance, over the 2002-10 period that Colin studied, the BIS-recorded overall swing rates and take rates inside the strike zone remained pretty stable from season to season, but the rate of recorded swings at pitches inside the zone declined dramatically:
As Colin wrote,
Or, in chart form, with “delta” representing the difference between April and September:
In other words, the difference attributed to the amphetamine ban might simply be a difference in the way the data was recorded.
I sent a draft of this article to Dr. Kutscher, and he kindly responded via email to some of my criticisms. He acknowledged that he had considered the possibility of scoring bias in the BIS data, and he noted that our explanations aren't necessarily mutually exclusive. For instance, while increased pitch velocity and the resultant reduced time to make decisions could explain the rise in swing rate, fatigue could still be a factor, since it's known to cause reaction time to suffer. While this is true, it's also part of the problem: the "fatigue effect" explanation isn't incompatible with any interpretation of the data, because we don't have any of the information that could prove or disprove a tie to plate discipline. In the absence of hard data on player fatigue, all we have is a hypothesis. And if all we have is a hypothesis—and some data that might be biased—why put out a press release?
The principle investigator for that study, W. Christopher Winter, MD, of the Martha Jefferson Hospital Sleep Medicine Center, has consulted for teams, so maybe there will be a trend toward paying more attention to players’ sleep habits. That wouldn’t be a bad thing. As Kutscher put it to me, "Fatigue has been shown to have a profound impact on performance in a wide variety of fields, and sleep represents a way for athletes to potentially improve their performance. In what ways, and to what extent, remains in question, but in a game of inches (or, perhaps more accurately, tenths of seconds) even a small improvement could make a big difference." On that we can agree.
But while a sleepy player might suffer from impaired performance, it’s a stretch to say that the league as a whole has worse plate discipline due to fatigue later in the season, and an even greater stretch to suggest that the amphetamine ban has produced a marked uptick in player fatigue. Those things might be true, but this study hasn’t shown them to be true. (It’s important to note that since this study has only been published as an abstract in an online supplement to SLEEP, not as a full paper in the journal itself, it hasn’t been through a formal peer review process.)
It may be that flawed studies are more likely to produce interesting results, and we know that studies with interesting results are more likely to be published and posted about. So maybe the lesson is: approach research and studies with a skeptical eye (and that includes studies published at Baseball Prospectus). Not everyone has the time and access to data necessary to evaluate every researcher’s claims, and when those claims come from a reputable source, cite statistics, and don’t sound implausible, it’s easy to accept them (especially when you really need something to write about). But caution is always in order.
Thanks to Dan Brooks, Russell Carleton, and Colin Wyers for research assistance.