There is no group of players whose performances are more volatile than relievers’. Because of their small workload and variable handling, all but the most reliable of them vary wildly in effectiveness from season to season. They are also sensitive to the way they are deployed by managers. Two relievers with identical numbers are going to make differing contributions to the winning effort if one is used in close games and the other only when the team is ahead or behind by a large margin.

Despite the almost random nature of the subject group, PECOTA forecasts relievers each season in the form of Win Expectation above Replacement, Lineup-adjusted, or WXRL. Put simply, WXRL is a measure of wins added by a reliever. As history has long shown, reliever ERA isn’t a helpful guide to their performances because of the brevity of their appearances and the way that inherited runners that they allow to score are accorded to other pitchers. Nor are saves a proper measure, as not all saves are created equal-the textbook save, getting three outs with a three-run lead, isn’t as difficult or important a task as getting three outs with a one-run lead or keeping a tied game in stasis. Therefore, we use WXRL as a measure of a reliever’s contribution to the winning effort.

Since we’re at the season’s half-way point, a quick and dirty way to compare PECOTA‘s WXRL forecast to actual performances from the 2009 season is to take each reliever’s WXRL and double it. In order to screen out small samples, we’ll focus on those relievers who have thrown at least 30 innings this season. Because the group of relievers is vast, we’ll also take a sampling of those at the top of the rankings, those at the bottom, those in the middle, and last year’s leaders.

Pitcher           Team        G   WXRL   WXRLx2  PECOTA   Diff.
David Aardsma     Mariners   38    4.1    8.3     0.5     +7.8
Jonathan Broxton  Dodgers    37    3.4    6.8     3.2     +3.6
Heath Bell        Padres     34    3.4    6.8     1.8     +5.0
Ramon Troncoso    Dodgers    38    3.2    6.4     0.4     +5.9
Jonathan Papelbon Red Sox    37    3.1    6.2     3.4     +2.8
Ryan Franklin     Cardinals  32    3.1    6.1     0.6     +5.5
Rafael Soriano    Braves     38    3.0    6.0     0.9     +5.1
Francisco Cordero Reds       36    2.8    5.7     1.4     +4.3
Mariano Rivera    Yankees    34    2.8    5.6     3.7     +1.9

There are many relievers having great seasons right now-last year, only six relievers exceeded a WXRL of 5.0. PECOTA was a bit surprised by this, as well as the composition of the group. It figured that Mariano Rivera would have a good year, but as he was entering into his age-39 season, it anticipated a slight decline from his pace of a year ago. The miss here, the narrowest in the group, is unsurprising; PECOTA is a bit flummoxed by Rivera, whom it rates as a historically unique pitcher. The system generates its forecasts by looking for comparable pitchers, and it ranks these based on a scoring system; a player with a score a score of 50 or higher has a very common typology, while a player with a score of 20 or lower is historically unusual. Rivera’s score is eight.

The other misses in this group are attributable to changes in usage. The rise of David Aardsma from journeyman first-round bust to closer was unanticipated, even by the Mariners. Ryan Franklin was looked upon as a middle reliever. PECOTA saw Rafael Soriano as pitching effectively, but did not project him as the closer and worried about his durability. With Francisco Cordero’s age and so-so 2008, PECOTA didn’t foresee a dominant performance.

We turn to the bottom of the list, this year’s least effective relievers with a substantial number of innings.

Pitcher         Team        G   WXRL   WXRLx2  PECOTA   Diff.
Jensen Lewis    Indians    28   -0.4   -0.8     0.7     -1.5
Peter Moylan    Braves     45   -0.4   -0.8     0.6     -1.4
Joel Zumaya     Tigers     27   -0.4   -0.9     0.5     -1.4
Manuel Corpas   Rockies    33   -0.5   -0.9     1.1     -2.0
Luis Ayala      Twins      28   -0.5   -1.1     0.3     -1.4
Joel Hanrahan   Nationals  34   -0.6   -1.2     0.6     -1.8
Brandon League  Blue Jays  37   -0.7   -1.4     0.8     -2.2
Cla Meredith    Padres     33   -0.7   -1.5     1.0     -2.5
Miguel Batista  Mariners   32   -0.8   -1.5     0.6     -2.1
Brad Lidge      Phillies   33   -1.5   -3.0     1.9     -4.9

Here the volatility of the reliever population is even more pronounced. While PECOTA didn’t anticipate great things from any of these pitchers, it did generally figure that they would make mildly positive contributions. The most dramatic miss of the group is with Phillies closer Brad Lidge. The system foresaw a steep decline in effectiveness for Lidge-perfection doesn’t last-but didn’t anticipate the degree of his fall. Otherwise, we have a collection of mostly non-closers, middle relievers and set-up men; these pitchers are the most inconsistent of performers within an inconsistent class. Note the sensitivity to context of WXRL; most of these relievers have not been pounded. Miguel Batista, for example, looks to be superficially strong with a 3.22 ERA and 8.9 H/9. It’s when we look at his performance with runners on base that Batista’s WXRL starts to make sense: Batista has inherited seven baserunners from other pitchers this year, and five of them have scored, a rate more than twice as high as the major league average-relievers allow about a third of inherited runners to score, but Batista has sent 71 percent of them home.

A sample around the median WXRL marks:

Pitcher          Team        G   WXRL   WXRLx2  PECOTA   Diff.
Manny Delcarmen  Red Sox    32    0.7    1.3      1.1    +0.2
Evan Meek        Pirates    26    0.6    1.3     -0.1    +1.2
Brandon Medders  Giants     32    0.6    1.2      0.2    +1.0
Brad Ziegler     Athletics  34    0.6    1.2      0.6    +0.6
Burke Badenhop   Marlins    20    0.6    1.2      0.2    +1.0
Ryan Madson      Phillies   43    0.6    1.1      1.0    +0.1
Jason Jennings   Rangers    29    0.5    1.1      0.4    +0.7
Shawn Camp       Blue Jays  29    0.5    1.1      0.6    +0.5
Justin Miller    Giants     27    0.5    1.0      0.4    +0.6
Mark Lowe        Mariners   37    0.5    1.0      0.4    +0.6

On the whole, PECOTA was right on this group, expecting them to be somewhere in the middle of the reliever pack, and getting what it called for. The biggest gap here is that of Pittsburgh’s Evan Meek-PECOTA looked at Meek’s meek performance last year and didn’t see much to love about the former Rule 5 pick. Given Meek’s rate of 6.6 walks per nine innings, it seems likely that Meek will be catching up to his projection in the course of time.

Finally, the aforementioned group of relievers that were over 5.0 wins added in 2008. We’ve already accounted for the top two, Brad Lidge and Mariano Rivera, which leaves four others:

Pitcher              Team     G   WXRL   WXRLx2  PECOTA    Diff.
Joe Nathan           Twins   35    2.7     5.4     3.1     +2.3
Francisco Rodriguez  Mets    38    2.1     4.3     3.3     +0.9
Carlos Marmol        Cubs    44    1.8     3.6     2.3     +1.3
Joakim Soria         Royals  21    1.7     3.4     2.6     +0.9

Speaking broadly, we’ve seen consistency from Nathan and K-Rod, while Marmol and Soria have taken a step back, the former due to infection with the wildness bug, the latter due to injury, which has truncated his playing time. The broadest miss, Nathan’s forecast, is caused by a similar effect to what happened with Rivera: he’s been so good for so long that PECOTA has difficulty finding pitchers to compare him to. Indeed, Rivera is seventh on Nathan’s list of comparables, though the system doesn’t seem them as being close relatives in its taxonomy of pitchers. Nathan’s similarity score is 13, so the system casts a wide net and takes its best guess.

Taken in the large view, WXRL is a terrifically sensitive statistic, and at the end of the season its range tends to be fairly narrow. In more than 50 years, there have only been 49 individual pitcher seasons in which a reliever exceeded a WXRL of five and change, and most years the league leaders can be found in the 5.50-7.50 range. This means that as the season proceeds it wouldn’t take much of an improvement by pitchers underperforming their PECOTAs, or much of a regression by those over-performing them, to bring the projections and performances into alignment. Relievers have little margin for error, and each success or failure swings the statistics dramatically. PECOTA would seem to have done a good job so far in aiming darts at a board as big as the universe.

A version of this story originally appeared on ESPN Insider Insider.

You need to be logged in to comment. Login or Subscribe
This has been a great series of articles. Thanks BP.
It has been a good series of articles, but I still don't see what good WXRL is. It doesn't tell me anything about the pitcher; it tells me about the combination of the pitcher and the pitcher's luck, and I already have dreck like ERA for that. A high or low strand rate isn't sustainable, for example, so I don't see why we bother measuring it. WXRL rewards pitchers for good defense behind them, and for lousy strategies executed by the opposing manager: those are luck; the pitcher didn't cause those.
I'm not sure why you say it doesn't tell you anything about the pitcher. The pitcher does CAUSE his strand rate at any given time he pitches, we just make note that over the long run that strand rate should even out. Since wxrl measures how the pitcher has changed win expectancy, it tells us how effective a reliever has been. If a reliever comes in with the bases loaded and no outs, and then strikes out the side, shouldn't he get credit for that effectiveness? We shouldn't say, "His strand rate for that inning was 100%, so he just got lucky." And if we can measure his past effectiveness, that gives us a stat with which we can project future effectiveness, no? And if we project future wxrl, we can take into account the luck factor you are mentioning.
"If a reliever comes in with the bases loaded and no outs, and then strikes out the side, shouldn't he get credit for that effectiveness?" It depends on why you're looking at him. If you want to know, in hindsight, how much he contributed to team wins and losses, then yes. If you want to predict how useful he's going to be in the future, then not much. If you want to know whether to draft him for your Diamond Mind or Strat-o-Matic team, then not at all. I'll echo the original poster's plaint, which I have aired here myself in the past: BP doesn't publish any pitcher metrics that aren't either context-dependent (WXRL) or outcome-based (VORP, ERA, SNWL, etc.). There's nothing that is based solely on the outcomes of individual plate appearances.
Thanks for the article. One quibble. Every time I see someone say "sampling" here I feel like I'm going to see Yankees and Athletics and no Rangers or Royals. Please be systematic in your cutoffs, even if it makes the article harder to write. Its what we would expect from one of our BP Idol contestants, isn't it?
"In order to screen out small samples, we’ll focus on those relievers who have thrown at least 30 innings this season. Because the group of relievers is vast, we’ll also take a sampling of those at the top of the rankings, those at the bottom, those in the middle, and last year’s leaders." Those aren't systematic cutoffs?
"Indeed, Rivera is seventh on Nathan’s list of comparables, though the system doesn’t seem them as being close relatives in its taxonomy of pitchers." I would love to see some form of a phylogenetic tree based on PECOTA's similarity indices. It'd indicate nothing, as we already know that pitchers didn't develop their skills through natural selection across the generations, but it would be fun. I don't suppose you have any bioinformaticians on the staff?