The next time you’re sitting on your sofa contemplating the parade of busted moves by the assembled general managers of your favorite franchise-say, the Mets-and are tempted to proclaim that you could do a better job than that clown in the executive suite, think twice. Anyone can make the easy calls and buy or trade at the top of the market, fishing for the Pujols and Sabathias when they’re available. The aspect of the job that earns GMs their end-of-day bourbon and Maalox cocktail is putting together a bullpen. Even the abstemious Branch Rickey might have been driven to drink by relievers had he been forced to give them more than cursory attention; they’re just too unpredictable for comfort.

A T-shirt recently spotted at Yankee Stadium read, “Anyone but Farnsworth.” As this holdover garment from 2008 testifies, reliever acquisition is the part of the roster where GMs are most likely to spend millions and get burned. Brian Cashman bids $17 million over three years for the 2005 version of Farnsworth, and gets one of the few pitchers capable of personally inflating the league home-run rate instead. Given Farnsworth’s career, Cashman probably should have known better, but the Farnsworth Dilemma extends across the class of relievers as a whole. In truth, there are only a handful of relievers who are solid bets on more than a year by year basis: Mariano Rivera, Joe Nathan, Francisco Rodriguez, perhaps Scot Shields, and Chad Qualls. Once past that quintet, a team takes its chances.

In 2006, Baseball Prospectus took a look at reliever consistency over the previous 30 years. We used Fair Run Average to rank the top 50 relievers in baseball in each season from 1975 to 2005 who had a minimum of 50 innings pitched. Fair RA is a way of compensating for the distorted nature of reliever RA by adjusting it for expected runs allowed on inherited and bequeathed baserunners. What we found was massive turnover in the top 50. In any given year, an average of 30 relievers drop off the list, and the changes grow with each added year. For example, of the top 50 relievers in 2005, 37 of them did not rank in the top 50 in 2004, 38 of them didn’t rank there in 2003, and 45 of them did not rank there in 2002. Overall, the average change was 76 percent every three years, 70 percent every two years, and 60 percent every year. Even if a GM spent each winter shopping exclusively among the top 50 relievers in baseball, he would still have a six in ten chance of choosing incorrectly.

Since 2005, the turnover of the top 50 has only accelerated, making the GM’s job even harder. Here are the figures for reliever reliability from 2006 through 2008:

Year  Year-1  Year-2  Year-3
2006    20      10       5
2007    15       8       4
2008    14       4       4  

AVG     16.3     7.3     4.3

Using 2006 as an example, only 20 of the top 50 relievers from 2006 were still on the list in 2005, just 10 of them were on the list in 2004, and only five of them remained on the list from 2003 through 2006. The four who made it from 2005-2008: the aforementioned Rivera, Nathan, Rodriguez, and Qualls.

The lack of reliever reliability persists regardless of the standard you use. Ranking the top 50 according to BP’s reliever statistic WXRL, or expected wins added by each reliever above replacement-lineup adjusted, a more useful stat than saves and holds, shows a similarly steep turnover, with only Shields and Huston Street joining the list of 2005-2008 hangers on. Taking a broader view also reveals the instability of the relief corps. Here are the top relievers for 2003 through 2005, again ranked by Fair RA, followed by their 2006-2008 performances. There is a dramatic change for the worse in Fair RA:

Pitcher                 03-05                06-08
                      IP     FRA   RANK    IP     FRA   CHANGE
Eric Gagne           178.0  1.73     1    100.3  4.15    +2.42
Mariano Rivera       228.7  2.04     2    217.0  2.12    +0.08
Billy Wagner         212.0  2.10     3    188.7  3.09    +0.99
Joe Nathan           221.3  2.22     4    208.7  1.83    -0.39
Scott Linebrink      221.0  2.52     5    192.3  3.61    +1.09
Brad Lidge           250.3  2.55     6    211.3  4.03    +1.48
Jason Isringhausen   176.3  2.59     7    166.3  4.28    +1.69
B.J. Ryan            208.7  2.62     8    135.7  2.28    -0.34
Brendan Donnelly     181.3  2.64     9     98.3  4.77    +2.13
Tom Gordon           244.3  2.72    10    129.0  4.38    +1.66
Francisco Rodriguez  237.3  2.74    11    209.7  2.82    +0.08
Chad Cordero         168.0  2.79    12    153.7  3.41    +0.62
Armando Benitez      173.7  2.91    13     95.0  5.64    +2.73
Keith Foulke         215.7  2.93    14     81.7  4.39    +1.46
Rheal Cormier        213.0  3.03    15     51.0  4.95    +1.92
David Riske          225.7  3.03    15    156.0  4.11    +1.08
Scott Eyre           178.0  3.11    17    139.3  4.34    +1.23
Ryan Madson          165.3  3.13    18    183.7  3.69    +0.56
Scot Shields         267.7  3.14    19    228.0  3.45    +0.31
Juan Rincon          245.7  3.16    20    189.3  5.48    +2.32
LaTroy Hawkins       216.7  3.17    21    178.7  3.79    +0.62
Eddie Guardado       167.0  3.22    22    107.0  4.87    +1.65
Octavio Dotel        188.7  3.24    23    108.7  5.27    +2.03
Shigetoshi Hasegawa  208.7  3.27    24      N/A   N/A      N/A
Luis Ayala           232.3  3.31    25    118.0  5.56    +2.25
Damaso Marte         199.7  3.33    26    169.7  3.37    +0.04
Guillermo Mota       269.7  3.43    27    172.0  5.09    +1.66
Francisco Cordero    223.3  3.45    28    209.0  3.85    +0.40
Justin Duchscherer   184.0  3.45    28     72.0  3.08    -0.37
Salomon Torres       226.0  3.46    30    226.0  4.22    +0.76

So what is a general manager to do if he wants to end up with the Phillies ’08 bullpen and not the Mets ’08 bullpen? Prayer might help-that, and a gambler’s mentality. For though the instability at the top of the reliever corps is great, the pitchers that move onto the list have to come from somewhere. Quite often, they come from the minor leagues, be they prospects or journeymen. This means that a GM’s best option is often also his cheapest option. When we examined the top reliever ranks in 2006, we found that from 1975 through 2005 on average the top 50 contained only 20 pitchers who had thrown more than 10 innings in the majors the previous year. This particular form of turnover seems to have slowed in recent years, with only one-fifth of the list yielding to pitchers who, through injury, inexperience, or ineffectiveness, were largely absent from the major leagues the year before.

Standard reliever statistics contain almost as much noise as signal. Their ERAs indicate little, their saves totals not much more, as any Indians fan who saw Joe Borowski lead the AL in saves in 2007 can tell you. Even more indicative statistics, like hits- and strikeouts-to-innings pitched ratios, are subject to enormous fluctuation due to the increased bearing of luck on pitchers who throw only 50 to 90 innings per year. No team has ever bought a great bullpen. Those come about through luck-one or two front-line arms are backed by a series of second-line, bargain-bin refugees who happen to click behind them. Alternatively, you can hope that the Yankees let you have Mariano Rivera, or failing that, some hapless GM deals you Chad Qualls for Jose Valverde. It’s not as glamorous a solution, but that’s the whole point.

A version of this story originally appeared on ESPN Insider Insider.

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
"[T]here are only a handful of relievers who are solid bets on more than a year by year basis."

Not Joakim Soria?
Soria makes the FRA list for 2007-2008, but of course his MLB career doesn't go back any further than that. If Hillman ever remembers to use him, we can check back in the future and perhaps we'll see him here. He's also a great example of the kind of "found" reliever that we're talking about.
Okay, I guess I see why Steven's list only had those five guys. But today I would take Soria over all of them except Mariano.
Trey Hillman wouldn't. He'd rather keep Soria permanently glued to the bullpen bench.
Well yeah. Soria's a Closer with a capital C. Having him pitch in a hold situation would be like buying non-alcoholic beer: what's the point?

When the A's fired Macha, one of the rumors I heard was that Trey Hillman was a finalist for the manager's position. After seeing his tenure in KC, I'm sort of happy we dodged that bullet.
And I'm sure in 2004 quite a few would have taken Eric Gagne over most of them as well. That wouldn't have turned out all the great. Which I think is the point of the article, relief excellence today very rarely leads to relief excellence tomorrow.
I would say Soria is #2 to Nathan.
"No team has ever bought a great bullpen."

I'm not sure of that.

Not going too far back, the 2006 New York Mets led MLB with a 17.787 WXRL. They bought almost their entire bullpen. Here are their top seven relief pitchers, along with how they were acquired:

1. Billy Wagner 5.953 WXRL Free Agent
2. Aaron Heilman 3.282 WXRL Drafted*
3. Duaner Sanchez 2.795 WXRL Trade
4. Chad Bradford 1.834 WXRL Free Agent
5. Darren Oliver 1.334 WXRL Free Agent
6. Pedro Feliciano 1.263 WXRL Free Agent
7. Guillermo Mota 0.820 WXRL Free Agent

* Heilman was drafted by the Yankees and the Twins before being drafted by the Mets. He didn't choose to sign for the money offered by those teams. He signed with the Mets for a $1,508,705 bonus, pretty reasonable money for an 18th pick in 2001. While he wasn't acquired as a free agent, money was, to some degree, a factor.

I'd suggest that a team can possibly buy a bullpen, but I'd acknowledge that it's not easy.

"Bought" means spending alotta $$$ on the new guys. Your chart isn't useful without salaries. If they got all the guys other than Wagner and Heilmann cheaply, then they lucked into their good bullpen rather than bought it.
For 2006:

Billy Wagner $10.5M
Aaron Heilman $0.359M
Duaner Sanchez $0.3995M
Chad Bradford $1.4M
Darren Oliver $0.6M
Pedro Feliciano $0.359M
Guillermo Mota $3M

Total $16,617,500

By the way, tossing in a couple of other names they acquired, guys who weren't among the eventual seven best:

Jorge Julio $2,525,000
Roberto Hernandez $2,750,000 (shared with Pittsburgh)

There are a few other Mets bullpen pitchers from that year...The Mets invested on the order of $20 million in their MLB-best bullpen.

Given the number of free agents and the extraordinarily high total bullpen salary, I'd call it buying a bullpen. Others' standards and mileage may, of course, vary.
And by 2007, that bullpen wasn't the best anymore. It's hard to tell if a bullpen is consistently good because the sample size for each given reliever is so small.

Sometimes I wonder why organizations just don't develop utility player/pitchers. I mean, if overall reliever performance can fluctuate, and there are so many token lefties and token mop-up men, why not teach the organizational soldiers a different position? In addition, there are quite a number of pitchers that are converted catchers/infielders/outfielders. Why not keep working them at both positions? Even if they can't hit, if a player was able to play a passable CF/SS/2B/3B or catcher and serve as a mop-up reliever, wouldn't that be better than using that roster slot on a player who is used only as a mop-up reliever? The biggest problem I see with this idea, though, is there are only so many innings to distribute to a minor league roster of players, and it makes little sense to bump a prospect to a bench so that a pitcher-hybrid can work on his fielding.

Seems like Papelbon belongs on the consistently reliable closers list too.
I think the main argument would be that he's only pitched since 2005. But he certainly looks to be joining the ranks of consistently excellent relievers, similar to Joakim Soria.
"or failing that, some hapless GM deals you Chad Qualls for Jose Valverde."

Maybe I'm missing something here, but I don't see anything suggesting that Qualls is a significantly better pitcher than Valverde:

2004: 11.53
2005: 10.18
2006: 12.59
2007: 10.91
2008: 10.38

2004: 6.55
2005: 6.78
2006: 5.68
2007: 8.49
2008: 8.67

2004: 2.24
2005: 3.75
2006: 3.14
2007: 3.00
2008: 3.61

2004: 3.00
2005: 2.61
2006: 2.00
2007: 3.12
2008: 3.94

2004: 5.37
2005: 2.73
2006: 3.39
2007: 3.58
2008: 3.67

2004: 3.87
2005: 3.75
2006: 4.50
2007: 3.94
2008: 2.77

Valverde misses more bats while Qualls has better control. Both hover around a 3.5 FIP. If the argument is that Qualls is a better value I would definitely agree (he has made less than $3 million since '04 while Valverde has been paid almost $8M) but I don't see any evidence suggesting Qualls is materially better than Valverde. If I'm missing something, please let me know.
"Even more indicative statistics, like hits- and strikeouts-to-innings pitched ratios, are subject to enormous fluctuation due to the increased bearing of luck on pitchers who throw only 50 to 90 innings per year."

... which is why I would rather have seen this study using those more indicative statistics - far less volitile.