BP360 now on sale! A yearly subscription, '23 Annual & Futures Guide and t-shirt for one low price

Your team has a bit of a problem. Namely, it’s the eighth inning and you are behind by two runs as you take the field to play defense. Worse, your starter is tired and you need to make a call to the bullpen. The question now is whom you should summon. After all, bringing a pitcher in now affects his availability for tomorrow. Should you bring in your ace set-up reliever, try to keep the deficit at two, and hope your offense can come back? Should you bring in the lesser reliever, figuring there’s no use wasting such a valuable resource on a game that, more likely than not, you will lose? Decisions, decisions.

This is the anti-save situation: It’s late, it’s close, but, unlike the actual save situation, you are losing. If this were a regular save situation, there would be a predictable rhythm to how it would unfold. If it’s the eighth inning, you bring in the "set-up guy." If it’s the ninth, you employ your "closer." In other words, it’s a rote formula that anyone could use. What to do, though, when it’s the other way around and you’re not holding a lead, but chasing one? It says a lot about a manager’s style as to how he handles this situation. Is he the sort who likes to go for broke, or does he prefer to save his resources for a rainy day? No one bothers to think about this sort of situation, as they don’t hand out medallions for keeping the score close so that the offense at least has a chance. As a reliever, if you go get a win, then you are derided as a vulture!

Figuring out how a manager handles this situation turned out to be harder than I expected, because, once again, a manager is limited to what resources he has been given. Some managers are blessed with a bevy of outstanding relievers who can handle any situation. Some have to make do with bullpens that are a threat to explode on a nightly basis. Furthermore, reliever performance is a fickle thing due to the small sample sizes inherent in pitching to only a few batters every couple of nights.

Warning: Gory Methodological Details

The first part, defining anti-save situations, was easy enough. I looked for situations in the seventh and eighth innings in which a team was pitching and down by either one or two runs at the start of the inning. (My database stretched from 2003-2009.) The second part, figuring out how the manager was playing the situation, turned out to be tougher.

At first, I figured that I would look at the raw run averages (RA) of the pitchers brought into these situations. Leaving aside for a moment the problem of statistical reliability, a manager who brought in a guy with a 3.00 RA clearly values the situations more than the manager who brings in the guy with the 5.00 RA. But what happens if a manager doesn’t have a 3.00 RA guy to bring in, and, unlikely, but possible, the 6.00 RA guy is actually his best bet?

Instead, I went for an ordinal ranking method. For each team-year, I ranked their relievers from No. 1 to whatever based on seasonal FIP (that is, the rankings for 2009 were based on what the relievers’ FIPs were at the end of the 2009 season). This way, if a manager brings in his best guy in this situation, he’s not penalized if that the guy is a bum, and he gets some recognition for at least trying. The problem is that I’ve taken data that are roughly interval and turned them into a much less-useful ordinal variable. Worse, I’m going to do something I often told my stats classes not to do and that’s take the average of ordinal data! It’s not ideal, but this is a tough nut to crack.

In anti-save innings in which more than one reliever appeared, I took the best-rated reliever of the bunch. Because I looked at both the seventh and eighth innings, a manager could have two "anti-save" innings per game. Most had around 50-60. I looked at the average ranking, again relative to the other bullpen options available, that the manager employed in these anti-save innings.

The Results

The manager who most jealously guarded an anti-save lead during the 2009 season? Clint Hurdle (average ranking = 1.75). Before he, you know, got fired. Lest you think that was the reason, though, in second place was his successor, Jim Tracy (2.11), who took the Rockies to the playoffs and won National League Manager of the Year. Then again, in third place was Bob Melvin (2.23), prior to his untimely dismissal from the Diamondbacks. Maybe there’s something to this theory, as now-former Indians manager Eric Wedge came in sixth. For the record, Manny Acta, who also got the axe in Washington, finished in the middle of the list.

The manager who was most lackadaisical about trying to keep the deficit close was Dave Trembley, followed by Cecil Cooper and Ken Macha. I’m not sure what the message is there with those three. But for the morbidly curious, here’s the list, complete with the average ranking of the reliever brought in, relative to his own bullpen:

Manager		Average Reliever Ranking  
Clint Hurdle		1.75
Jim Tracy		2.11
Bob Melvin		2.23
Ron Washington	        2.33
Jim Riggleman		2.34
Eric Wedge		2.41
Lou Piniella		2.47
Jim Leyland		2.58
Charlie Manuel		2.60
Jerry Manuel		2.63
Ozzie Guillen		2.65
Joe Maddon		2.71
Bob Geren		2.82
Bobby Cox		3.21
John Russell		3.22
Manny Acta		3.22
Cito Gaston		3.22
Bud Black		3.35
Freddi Gonzalez	        3.35
Joe Torre		3.35
Dusty Baker		3.38
Joe Girardi		3.39
A.J. Hinch		3.53
Tony LaRussa		3.54
Mike Scioscia		3.56
Don Wakamatsu	        3.69
Ron Gardenhire		3.82
Terry Francona		3.94
Trey Hillman		4.00
Bruce Bochy		4.04
Ken Macha		4.15
Cecil Cooper		4.17
Dave Trembley		4.28

As in my last article, I looked for the reliability of this new toy stat using intra-class correlation (AR(1) rho). It came up as a disappointing .253 over four years (2006-2009). That’s high enough that it makes me think that there’s something there, but that we’d need a few more years’ worth of data to get a good read on it.

Then again, the variability may be due to the fact that often, roles within a bullpen seem to be based on incumbency, rather than performance, or at least that there is a bit of lag between diminishing performance and being politely asked to leave a set-up role. There’s also the small sample size problem with relievers, which can lead to large variations in performance metrics. Relievers face roughly 200-300 batters during a season. Sometimes weird things happen, and the guy who in reality is the second-best pitcher in that pen looks like the fifth-best. Sometimes, it works the other way around. So, the manager, in his own mind, may think that he is calling for his second-best reliever, but my model would say that the man he’s called for is one of the scrubs. This is the greatest difficulty of psychological research: I can’t look inside the manager’s head. I can only look at the results and make a reasonable guess as to what he was thinking. Still, I think this situation is an overlooked window into how a manager approaches the game.

Does the Anti-Save Explain Home-Field Advantage?

One interesting side note that occurred to me: Could these anti-save situations help to explain the presence of home-field advantage in baseball? Consider that the home team has a small structural advantage here. If it’s the eighth inning, the home team pitches in the top of the eighth, meaning that if they are behind, their offense still has two at-bats in which to gather the needed runs to tie or go ahead. For the visitors, it’s the bottom of the eighth, meaning that they have only one more time to bat.

Using a lesser reliever might actually be a sensible strategy in the long run, especially on the road. There is a price to be paid for bringing in a good reliever, which is his possible unavailability the next day. Even if the good reliever holds the other team scoreless, his efforts might be in vain if his team doesn’t score any runs. So, it makes more sense for the manager to take this risk when his team has a greater number of times at bat. On the road, it might make more sense to hold the good reliever back and give up today for a better chance at tomorrow.

I looked to see by inning and by home/visitor status what the average ranking of the pitcher brought in was. There was almost no difference, and what difference existed was in the direction of the visitors bringing in a slightly better pitcher. In the seventh inning, the visitors, on average, brought in their 3.28th-best reliever, while the home team brought in their 3.31th-best reliever. In the eighth inning, the visitors went with their 3.11th-best reliever, while the home team went with their 3.13th. Road managers appear no more likely to want to give up than home managers.

Given the rationale above, I’m left to wonder if this represents inefficiencies in general bullpen management. Certainly, there’s a cultural taboo against "giving up," especially within pro sports, and sending out a less effective reliever may be seen as giving up. But if the goal is to win as many games as possible, it is at least conceivable that there might be a set of circumstances that would leave giving up today and conserving resources for tomorrow as the preferable strategy. (Perhaps a team with a bad offense?) Given the structural advantage that the home team enjoys in batting last, this set of circumstances would be more likely to happen on the road. There should be a difference in those scores.

The near identical rankings observed make me believe that managers, as a whole, are not making these kinds of rational cost-benefit calculations, either explicitly or implicitly. Instead, they appear to be responding to some sort of cultural expectation around the issue, probably around some sort of idea of dishonor at having given up on a game. The eighth inning with a deficit of X runs calls for the nth-best pitcher, whether it’s the top or the bottom of the inning. It’s not about winning or losing, but saving face. That cultural expectation is certainly strong, but if it’s getting in the way of winning as many games as possible, is it not a better idea to buck the culture?

Russell A. Carleton, the writer formerly known as 'Pizza Cutter,' is a contributor to Baseball Prospectus. He can be reached here.

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
I think we fans should jump in now and explain how our team's manager handled this -- the details are a lot more interesting than a single number indicates!

Atlanta Braves: Bobby Cox
Cox was somewhat limited, especially early in the season in the fact that his top three bullpen options were all coming off surgery. His solution was to use Soriano and Gonzalez pretty much in only save situations and Moylan (#3) as the seventh inning guy in save situations. Only Moylan (who Bobby apparently sees as the reincarnation of Mark Eichorn) went on consecutive days early in the year.

As to the, "Anti-save," Early on, Manny Acosta (ordinal rank of about #50 in the braves pen) got the bulk of the 7th inning work while Moylan usually pitched the 8th. Later in the year, Chris Medlen took over and often pitched both innings, although Moylan was still used as well.

To sum up, Bobby had a good thing going by the end of the year but wasted the first half opportunities (on last year's Braves, go figure!)
Don't know that I'd trust your data set much prior to seeing it run on a multi-year basis. If managers bounced up and down alot on it from year to year, that would suggest something else is determining things, like availability.
Agreed. Being an Angel fan, I'd be interested in seeing the delta with 2008 with the Mets and Angels in light of losing Frankie.

I think its too simplistic wthout considering the starters and relievers available to a manager too year over year. If your 'pen is an arson squad for instance, the best choice is to leave your starters in.

Jay - you write some of the best-ver pieces I read, but this one is premature i think.
sorry, thought this was from jaffe. oops.
I'm not nearly that cool. Or good looking.
Here could be one possible explaination for the home and road teams bringing in close to the same ranking pitcher in the 8th inning. If down by two in the top of the 8th as the home team, the some home managers could be thinking that they have two innings to score runs so one more run isn't that big of a deal, while others think how you did, in that there are two more innings to score two runs so the third one would have large negative effects on the outcome. Then with the road managers some think that two runs is too big of a deficit already so they give up, while others believe that two runs can be had very easily so they bring in their best. So both home and road managers have clear ways of thinking to be both conservative and aggressive thus making the rankings of both the home and road relievers the same.
Trey Hillman runs Farnsworth out there early in the season to give up a lead and then noting that Soria could only go one inning. Follow that comment up with a late season run of 2 inning saves for Joakim and I'm thinking that earns him his spot in the bottom-5.
Is there any correlation between this statistic and winning baseball games? I'm a little slow maybe, but I don't see one. Who's to say Dave Trembley isn't right, that those games are most likely lost, and it's better to save your best relievers for games where you're tied or ahead?
Just looking at the list, it doesn't look like there's a correlation one way or the other, although a lot of the guys who got fired were at the top.
I'm not so sure that using FIP to rank bullpens is the best bet, considering most managers probably use silly things like ERA, and of course our most favoritest statistic, GUT. Of course FIP is a better true indication of a pitcher's ability, but likely not the one the managers are looking at in the bullpen when they make their decisions.
I agree. Also, ordinal rankings don't make much sense. If I have guys with FIP of 3.11, 3.12, 3.13, and 5.00 what matters is that I don't use the 5.00 in close and late situations. It doesn't mater so much which of the 3.1x guys I use. I think making it more a ratio of how close to the best guy you use might make a more meaningful comparison.
Two thumbs up. This could partially explain the relatively low ranking of Terry Francona -- for a good portion of the season, the top 3 righty setup guys in the Sox pen (Ramirez, Delcarmen, Bard) were nearly fungible. I'm not looking at FIP right now, but it's conceivable they finish something like 3-4-5 after Okajima and Papelbon, and Francona's 3.94 is nearly identical to the average of those guys' ordinal ranks.

My first thought was that the two Rockies managers being at the top might mean volatility of runs by inning is correlated with desire to keep the game close. A lot of the powerful offenses (and I think that also implies high volatility, if I remember past research correctly?) appear towards the bottom of the pack, though.
Several years ago Nate or Jay did a GREAT article on new statistics for relief pitchers based on when they were used. In additions to saves and holds, they included categories for "keeping game close down 1-2 runs" "successful mop-ups (3+ innings in blowouts)" etc. I searched the archhives for the article but couldn't find it. Anyone remember this? It seems relevant for this article
If we're trying to get at how aggressively a guy is managing to win today at tomorrow's expense, we're conflating it with a bunch of other stuff here:
- What about platoon splits? A LOOGY might be the most effective guy for a given situation, even if his aggregate season numbers (which presumably reflect some low-leverage PAs by right-handed hitters) aren't up to par. Since some teams have such a guy and others don't, it seems like a potential source of bias.
- This confuses the actual effectiveness of a reliever with the manager's (possibly justified) view of his ability. Your 2nd-best reliever coming into the year might flame out and be ranked 5th, but that's a problem with forecasting the guy's ability, not bad usage patterns.
You're right that there's a lot of other gunk in there. It was the most difficult piece of the work. I was pleased that I was able to get some sort of a reliable measure, but without mind reading abilities, this one would be really hard to do properly.
Is there a correlation between Average Reliever Ranking and Park Factor? It would make sense: if managers in a certain park (say Coors, or Chase) were only down a couple of runs late, then holding the deficit where it is would yield a better chance of winning then it would in a park like Petco or Dodger Stadium. Of course Bud Black and Joe Torre are in the middle of the list, so it's probably nothing, but the 3 at the top made me suspicious.
Or what about using a season-long leverage index (LI) to determine what role each pitcher played in the bullpen? That calculates directly how important the manager finds each pitcher throughout the year, which i think is a better indication of who their bullpen ace is, and who's second, and c. I fear you're finding a lot of variability here just because managers don't know who their best pitchers are, especially given the types of pitchers that are usually brought out in these situations (seems to me young guys with promise are usually tested out in roles like this).

As is, you're punishing teams with strong closers and set-up men, bullpen roles that have less turnover throughout the year.
I think there might actually be a lower cost to using a good reliever on the road, because you don't have to use a reliever in the bottom of the 9th. Using your 2nd and 3rd best relievers when you are losing in the bottom of the 7th and 8th innings isn't wasting resources as much as using your 2nd and 3rd best relievers when you are losing in the top of the 7th and 8th innings, because in the event that you are still losing afterwards, the top of the 9th inning also requires another reliever to be sent to the mound by home managers, but not by road managers if they are losing in the bottom of the 9th. Even if managers have a better chance of winning at home and want to hold back less, they have a higher probability of needing to use a reliever in the 9th so they need to hold back more.

Really interesting article, all in all. I'm not sure how I would have approached a lot of this stuff, and I think you probably used the best methodology given the circumstances. Looking at managers qualities piecemeal is undoubtedly the way to actually evaluate their decision-making performance.
Russell - lots of food for thought, but one factor in your gory details may be holding you back from reaping tellable results; By looking at seasonal FIP numbers in determining the proper chaining of a bullpen, you may be seriously skewing your perception of what the manager was doing earlier in the season, when some of those year-end numbers would seem far-fetched, or at the very-least, unknowable.
For example, let's look at Cito Gaston. When the year started, he thought BJ Ryan would return to the closer role and effectiveness, if not dominance. So when he returned from his surgery, he was made the top reliever (top-3 at least). He struggled at first, which may have been attributed to rust. Eventually, though, he was deemed to be useless and first demoted to a lesser spot on the chain, and then outright released from the organization. So, while Cito had good reason to presume his usefulness when he first returned, accumulated results make him look bad for having used him in high leverage spots early on.
This type of thinking hurts any manager for having used a guy in high leverage spots when he was going well, and then reacted by demoting him when he began to struggle more consistently.
In the second to last paragraph you touched on something I'm interested in (but too lazy to do the work myself): what is the correlation with this data and the team's offensive capabilities?

It would seem that a smart manager with a weak offense may recognize his low probability of a comeback to dictate how to apply his resources. And the converse for the manager with the better offense. So some application of win probabiliy and offensive capability would be a nice addition.

I am not sure about your data structure. One thing that bothers me is that you are using data at the end of a period to analyze the decision process during the period. Maybe Mr. X was going strong in the spring and brought in early but he fell off during the year. Now you go back and say he was reliver #5 but when he was used in the 8th inning in May he was reliever #2 (or vice versa).

As to the specifics, Charlie Manuel virtually never gives up on a close game. With the Phillies line-up why should he? Nice to see come from behind wins calculated against your questionable stat.
I think this is an interesting first stab, but it definitely needs further work and development. I'm particularly concerned about the use of season-long data in this situation. I imagine Charlie Manuel's ranking is considerably skewed by the fact that he viewed Brad Lidge as his No. 1 reliever throughout the season whereas he probably ranks about No. 10 on your system. Thus, while he's relying on his No. 1, 2, or 3 guys according to your system, in his mind he's using No. 2, 3, or 4.

It would seem that managers who use a clearly defined closer that does not rank No. 1 on their team's ordinal system according to FIP will wind up with significantly higher scores as a result.
Interesting article, looking forward to the next in the series. Maybe one wrinkle you could have run would be to give the team's "closer" the number 1 ranking, regardless of FIP; also, adjust the rankings if the closer were to lose his job [a la BJ Ryan].