Baseball Therapy: Whatever Happened to Predictability?

In theory, spring training is the place to figure out what a team has become over the winter. The reality of baseball is that teams change, and the plan from last year might need some updating to reflect the new personnel. Got a couple of new faces in the bullpen this year? Well, we need to assign jobs to each of them. Otherwise, they’ll sit out there like little lost puppies, not sure what to do. But what if spring training wasn’t quite enough time to get everyone into place? What if your plans were suddenly derailed by an untimely injury?

In the early going this season, we find Angels manager Mike Scioscia talking about using a “closer by committee” and Red Sox manager John Farrell isn’t entirely sure who’s pitching the seventh and eighth innings for his team. (His job is made a little easier by having Craig Kimbrel to pencil into the ninth). For right now, things are up in the air. There’s a bit of conventional wisdom which says that this is a problem.

We live in an era of very strictly defined bullpen roles. Beyond the designated “closer” who pitches in the ninth inning, a guy who pitches in the eighth, and so on. Using innings, rather than situations, as a way to assign roles is justified in the name of relievers having “predictability” during a game. Whatever happened to predictability? And then, for some reason, this happened:

In the video, Mike Felger and Evan Drellich, both media folks who cover the Red Sox and the general Boston area, are discussing the Red Sox's bullpen. Drellich begins by making the case against a strict innings-based bullpen system, suggesting instead that the Red Sox (and by extension, most baseball teams) might consider being less rigid, and perhaps bringing in the closer in the eighth inning to face the 3-4-5 hitters and then asking the eighth-inning guy to tackle the bottom of the lineup. If both men are going to pitch anyway, why not have the better pitcher face the better hitters?

Felger countered that he didn’t think it would work, because the relievers themselves like having a set role, and that they can show up to the ballpark knowing when they will pitch. Felger states that in these situations, “That’s when you get the best performance out of relievers.”

It got airplay because both men used words like “dinosaur” and “stat nerd” to describe each other, eventually culminating in Felger saying that “nerds are ruining sports.” (That sound you heard was me rolling my eyes.) As journalists, I’m sure that the two men can appreciate the ever-present need for a good fact-check, and the hypothesis of which they took opposite sides is a testable one.

In fact …

Warning! Gory Mathematical Details Ahead!

We’re not completely in the dark on this one. We know, for example, that when a team takes a lead in the bottom of the eighth or the top of the ninth and the closer doesn’t have as much warning time to adjust to the idea that he will be called on to save the game, he does show some evidence of jitters in the form of a lot more variability in the velocity and movement that we see on his pitches. However, when we look at his actual performance in those outings, there doesn’t seem to be an effect.

The idea that managers could ever implement the fully leveraged bullpen, where relievers get clicked into the game when a situation pops up with runners at second and third with one out, up by one in the eighth inning, isn’t realistic. Relievers need time to warm up, and most of the time trying to predict when one of those situations will occur would require a manager to have psychic powers. Still, something more modest, such as Drellich's suggestion that the manager would have his best relievers ready for single innings, although for a particular reliever it wouldn’t always be the same inning all the time.

A manager might approach his closer to say that he may get the call in the eighth inning, rather than the ninth, if it looks like the matchups are more favorable. All else equal, that makes sense (although the actual effect of these sorts of card-shuffling moves is rather small, and that’s before we account for the fact that all else might not be equal. But hey, any improvement is still an improvement).

Let’s dive into the question, then. How important is consistency of role in pitcher performance? For this one, I’m going to use data from 2003-2016. I found all reliever appearances and coded them for a couple of key features, including the inning in which the pitcher entered the game (coded as 1-6, 7, 8, or 9+), whether the pitcher came into the game mid-inning or whether he started the inning, and whether it was a ninth-inning save situation or not.

Let’s start with the inning in which our fair reliever made his stage entrance. There are two questions worth asking here. One is whether a pitcher has an inning that is generally “his” or not. In this case, I looked to see whether a pitcher had entered into a specific inning in seven or more of his previous 10 relief appearances. I didn’t demand 10-for-10 because there are going to be some “seventh-inning guys” who occasionally pick up a ninth inning to close out a game where the team is up by five or six runs and they need to pitch that day.

This does technically mean that a reliever can move in and out of being “consistently” used in a specific inning, and I actually consider that a strength of this methodology. As pitchers are used in the same situation over and over again, their sense of stability is going to increase. The other relevant question when we look at performance is whether the pitcher is currently pitching in “his” inning. What happens when you get a guy conditioned to pitching in the seventh inning, but then you bring him into the eighth inning?

When we do analyses like these, we need to be careful. One reason that a pitcher gets assigned to the ninth inning, rather than the sixth, is that he’s a better pitcher. We need to control for this. Fortunately, we have a way to do that. I used the log-odds ratio method, which controls for both the quality of the pitcher overall and the quality of the hitters he faces. (Since our data set stretches back into 2003, we are also going to control for the league context, which has gotten much more pitching-dominated over that time period.) For fun, I also controlled for the platoon effect.

Using this method, we can look at periods where a pitcher had a consistent role (or at least a consistent inning) vs. when he did not. I looked at all sorts of outcomes, including the ones that we worry most about (K, BB, 1B, 2B/3B, HR, Outs on balls in play, BABIP, OBP, and batted ball types). What did the data say?

I started with the question of whether a pitcher had a “usual” inning (seven or more of his 10 previous appearances had started in that same inning). Do pitchers get better outcomes when they’ve recently been used more consistently? Nope. Well, for those pitchers who do have a “usual” inning, we could ask whether they have been brought into that inning today, or whether today is weird. If they’ve gotten used to the seventh, but find themselves in the eighth, does that make a difference? Nope.

The one (and only) finding that shook out was that we saw a small uptick in singles when a pitcher was being used consistently in one specific inning. Consistency made him a little worse. (Um, given how many tests I ran … let’s insert a disclaimer about building up Type I error.) It seems that our pitchers perform in line with what we might otherwise expect of them, no matter what inning they are brought into. Advantage nerds?

Let’s talk about save situations. Maybe the sixth inning and the seventh inning aren’t so different from each other, but perhaps the save situation is its own special animal. A pitcher not only has to go out and pitch in the ninth inning and defend a small lead, he has to do so knowing that if he messes up, there’s a limited chance for someone to bail him out. Similar to the above, I looked for pitchers who had been used in a ninth-inning save situation in seven or more of his last 10 appearances. I then looked to see whether that day’s appearance was a save situation or not. Did closers pitch differently when they were outside the comforts of the save rule? Nope.

Turning that around, what happens when pitchers not normally used in save situations get thrust into the ninth inning with a small lead? Does that have any impact on them? Yes, it does. Non-closers trying to nail down a save issue more walks and doubles/triples. The effect size isn’t giant, but it’s there.

What about pitchers who are used to starting an inning? Does it make a difference if they are brought in, mid-inning? It turns out that this one makes a difference as well, with the pitcher giving up more walks, though fewer singles, and getting fewer ground balls. Again, all of this is compared to what we might otherwise expect based on his seasonal stats. It all about washes out.

So, our overall verdict: There is room to move pitchers around. The idea that pitchers show any difference (for good or ill) based on having a consistent inning that “belongs” to them is not supported by the evidence. The one exception is a save situation. This isn’t to say that no one other than the closer can save the game, it’s just that the setup guy probably isn’t as good as the closer (otherwise, he’d probably be the closer) and his performance is going to suffer a tiny bit when he gets out there in the ninth. We also see that pitchers suffer a bit when they are inserted into the middle of an inning (and are not used to that). Again, it’s not a huge effect, but there’s a price to be paid.

I’m Not a Dinosaur! You’re the Dinosaur!

I’m sure that relief pitchers prefer to have a set job. Predictability in life is nice. However, what relievers would like to happen is different from whether it actually affects their performance. In the debate between Messrs. Felger and Drellich, Felger presented the idea of the mix-and-match bullpen where a reliever might come in with two outs in the seventh one night and to close out a game the next and then into the fifth inning a few days later. That’s unworkable for a number of reasons.

However, Drellich’s more measured approach—that maybe sometimes a guy could pitch the seventh rather than the eighth because the game unfolded in a way that makes sense—is a reasonable approach to take, and the pitchers will be just fine. You want to be careful doing that sticking someone into the ninth inning, but there’s room for moving guys around. It’s even possible that pitchers can be reasoned with and, if you explain to them ahead of time that they might see action in the eighth or ninth, and here are the parameters for both possibilities, they might be able to figure it out.

It’s worth saying again that redefining roles doesn’t actually clear a lot of value if all it’s really doing is changing who protects which lead. The real value is in putting better relievers into tie situations. However, if managers want to experiment a little bit with who throws where and what makes the most sense, the evidence suggests that the relievers will be OK.