Last time out, we discussed reliever evaluation tools that used the run-expectancy and win-expectancy frameworks as the basis for judging relief pitchers’ performances, based on the change in game situation between the beginning and end of their outings. This time around, we’re going to introduce the Support-Neutral method of evaluating starting pitcher performance, and then dip a toe into some reader mail.
Starting Pitchers and Expected Wins
The family of Support-Neutral pitching statistics is a set of advanced tools that look at a starting pitcher’s performance using a variant of the win expectancy framework. These tools were initially developed by Michael Wolverton, and were subsequently revised by Keith Woolner. At their core, Support-Neutral statistics track expected wins per start based on the runs allowed, innings pitched and base-out situation when the starter leaves the game. BP’s Support-Neutral stats get their name from the fact that they look at a starter’s performance independent of the support the player receives from his teammates on offense and from the bullpen. In both cases, average, or “neutral” support is assessed to the player, and his numbers adjusted respectively.
Think about the usual way we assign wins and losses to pitchers in baseball: a win is given to any starter who pitches five or more innings, and leaves the game with a lead that his team does not relinquish. It is possible for a pitcher to pitch poorly and get the win. Take Todd Wellemeyer on Sunday; he surrendered five runs (four earned) in 5 1/3 innings against the Angels, but he was bailed out by outstanding offensive support (including two homers by Albert Pujols, and nine runs scored by the Cards overall) and fine bullpen support (two and two-third scoreless innings from Randy Flores and Ryan Franklin) to get the win.
It’s also possible for a starter to pitch very well but lose. Consider the plight of Matt Cain, who on Sunday threw eight good innings spoiled only by a Marco Scutaro homer. It was a wasted effort because the Giants were shut out by the dream team of Lenny DiNardo, Santiago Casilla and Alan Embree, and made worse by weak support by his bulllpen when Brad Hennessey gave up an insurance run to Oakland in the ninth. The more traditional viewpoint sees nothing wrong with this picture–Wellemeyer pitched well enough to win, Cain only well enough to lose, and that’s that. A support-neutral approach points out that any system that treats Wellemeyer’s start as better than Cain’s is seriously screwed up.
The most common metric in this approach is SNLVAR, a support-neutral statistic (hence the “SN”) which is lineup-adjusted (that’s the “L”), and which measures value added over replacement (the “VAR”). Accordingly, SNLVAR could be considered a starting pitcher’s counterpart to WXRL, in that both stats are measured in wins, and both feature lineup and replacement-level adjustments. The scale for SNLVAR runs from the high of 13.5 set by Sandy Koufax in 1966 to a low of -2.2 posted by Steve Blass in 1973. A double-digit SNLVAR is relatively rare-fewer than 30 pitchers have done it since 1959, and the last one to do so was Pedro Martinez (11.5) back in 2000.
The features and limitations of SNLVAR are as follows:
- SNLVAR is a counting stat, measured in wins contributed to the team.
- SNLVAR values are measured above what a replacement player would do, with below-replacement performances indicated as negative numbers.
- Values are adjusted for ballpark and for the strength of the opposing lineup.
- SNLVAR accounts for the pitcher’s offensive run support and bullpen support, “neutralizing” both to the league average.
- SNLVAR is a results-oriented rather than component-oriented statistic.
- Limited in that it does not account for (or “neutralize”) a pitcher’s defensive support from his teammates.
- Limited to seasons after and including 1959.
- SNLVAR can be found in the Pitcher’s Expected Win-Loss Records report, and the Pitcher Season, Pitcher Team Year, and Team Pitching customizable reports.
Here is a quick list of the current leaderboard in SNLVAR:
Pitcher TEAM W L G IP ERA FRA SNLVAR Dan Haren OAK 7 2 14 97.0 1.58 2.09 4.3 Jake Peavy SDN 7 1 13 87.0 1.97 2.23 3.6 Brad Penny LAN 7 1 13 83.2 2.26 2.47 3.4 James Shields TBA 6 0 13 97.2 3.04 3.08 3.0 John Lackey ANA 9 4 13 86.2 2.60 2.97 2.9 Chad Gaudin OAK 6 1 13 77.2 2.43 3.04 2.8 Tom Gorzelanny PIT 6 3 13 84.2 2.76 2.96 2.7 Roy Oswalt HOU 6 4 14 96.0 3.38 3.51 2.7 Tim Hudson ATL 6 4 14 92.1 3.51 3.67 2.7 C.C. Sabathia CLE 9 1 14 99.0 3.09 3.40 2.6
The Support-Neutral framework gives us a few more tools that we can use to evaluate starting pitching. One is Fair RA (or FRA), which is simply runs allowed per nine innings, adjusted to omit any above- or below-average bullpen support the starter receives. The values for Fair RA track, roughly, what we expect from ERA. As you can see from the figures above, they tend to run a little higher than ERAs, because unearned runs are included in the calculation.
Another product of the Support-Neutral framework is the ability to calculate an individual pitcher’s expected wins and losses, as distinguished from the value added to team that’s measured in stats like SNLVAR. For example, looking at the Pitcher’s Expected Win-Loss Records report, C.C. Sabathia’s won-loss record stands at 9-1, but the support-neutral data suggests it should be closer to 7-4 (technically, 6.7-3.9). From the difference between a pitcher’s expected and actual win-loss records, we derive a measure of how lucky or unlucky they’ve been, which brings us to a question that I spotted in a recent Will Carroll chat:
Andrew (Millburn, NJ): Is anyone having more of an unlucky dominant season than Andy Pettitte?
Using the measure I was just describing, which we succinctly call LUCK, we can see that Pettitte, while not having a particularly lucky season, isn’t the most unfortunate starter in all of baseball:
Pitcher W L G IP ERA SNLVAR LUCK Matt Cain 2 6 13 84.1 3.31 2.5 -6.30 Anthony Reyes 0 8 9 50.1 6.08 -0.3 -5.47 Paul Maholm 2 9 13 77.2 5.33 0.1 -5.17 Joe Kennedy 2 4 12 69.2 3.23 2.5 -4.92 Mike Pelfrey 0 5 6 30.1 6.53 -0.2 -4.78
The top five unluckiest features Cain (who we mentioned earlier) and Joe Kennedy, both of whom have been at least as “dominant” as Pettitte this season. With a -2.13 LUCK score, Pettitte ranks a comparatively fortunate 25th on the list.
Combining VORP Two Ways
We’ll finish off this week with a reader tip from reader P.S., on interesting ways to use VORP:
1) Since VORP is a counting stat, would it be accurate to evaluate the quality of a team’s receiving corps by adding Catcher + Backup Catcher VORPs, to determine if Jason Varitek + Doug Mirabelli is more productive than, say, Jorge Posada + Wil Nieves?
2) Also, can we add a pitcher’s hitting VORP to his VORP on the mound? Did Carlos Zambrano score a 59.1 VORP last year (53.8 pitching, 5.3 hitting) or will we have to find another way to measure how much extra value pitchers create with their at-bats? I’m very curious to see if a guy like Jason Marquis creates enough runs with the bat that it pushes him a little bit closer to average in terms of overall value.
The answer to both questions is yes. The main limitation of the first approach is that it would work best if the team’s catchers’ playing time did not overlap at all. For that reason, the examples given aren’t the best ones, since both Posada and Varitek have been given a few starts in the designated hitter slot, sharing time in the lineup with their respective backups. So adding together starter and backup VORPs in that situation would not accurately represent the team’s production from the catcher position–there are a few DH plate appearances “contaminating” the sample.
The second idea, about adding up a pitcher’s pitching and hitting VORP figures to get a more rounded idea of his performance, is truly inspired. A few other measures–our own WARP, or King Kaufman’s PRV stat–incorporate a moundsman’s accomplishments in the batter’s box into his overall performance. Here’s what the VORP leaderboards (courtesy of the indefatigable William Burke) look like if we do the same:
2007 Pitcher Hitter Total VORP Rank VORP VORP Dan Haren 38.2 1 -0.6 37.7 Jake Peavy 33.1 2 3.2 36.3 Brad Penny 30.9 3 2.1 33.0 James Shields 27.8 4 -0.4 27.4 John Lackey 26.4 6 0.0 26.4 C.C. Sabathia 25.0 8 1.3 26.4 Ian Snell 26.1 7 -0.3 25.9 Tom Gorzelanny 26.8 5 -1.7 25.1 Chad Gaudin 24.2 12 0.1 24.3 Matt Morris 21.4 17 2.6 24.0
2006 Pitcher Hitter Total VORP Rank VORP VORP Johan Santana 78.6 1 -0.3 78.3 Roy Oswalt 73.2 2 2.3 75.5 Brandon Webb 69.8 3 0.2 70.0 Roy Halladay 67.1 5 -0.4 66.7 Bronson Arroyo 65.9 6 0.1 66.0 Chris Carpenter 68.7 4 -2.8 65.9 John Smoltz 62.8 7 0.4 63.2 Carlos Zambrano 54.7 8 5.3 60.0 Chien-Ming Wang 53.6 9 -0.5 53.1 Francisco Liriano 50.4 12 0.7 51.1
The gap between Zambrano and Smoltz on the mound last year was so great that even Zambrano’s 5.3 VORP hitting performance couldn’t close the gap. Still, in this year’s rankings, positive hitting performances are enough to drag Chad Gaudin and Matt Morris into the top ten, at least for the time being.
Michael Wolverton, “‘Support Neutral’ Statistics-A Method for Evaluating the True Quality of a Pitcher’s Start“: This article, adapted from a 1993 article for SABR’s By The Numbers newsletter, sets out the original concept for deriving Support-Neutral Statistics.
Michael Wolverton, About SNWL: A summary of the Support-Neutral Win-Loss statistics.
Michael Wolverton, Top Pitchers of the 1990s: A Support-Neutral Approach: A 1999 article showing the earlier support-neutral methodology in action.
Michael Wolverton, Baseball Prospectus Basics: The Support-Neutral Stats: Notes on support-neutral statistics from Baseball Prospectus’s Basics series.
Keith Woolner, Support-Neutral Pitching Reports, Revamped: This article describes the current formulation of the support-neutral statistics. Woolner’s articles in Baseball Prospectus 2005 and Baseball Between the Numbers (see last week’s “Further Reading” section) set the methodology out in greater detail.
Michael Wolverton, Banking on the Bullpen-How Much Do a Starter’s Numbers Depend on His Relievers?: This article discusses Fair RA, and the interplay between starters’ performances and bullpen support.