BP Comment Quick Links


February 12, 2010 Introducing SIERAPart 5A helpful comment in one of the previous articles alerted us that our parkadjustment method was not quite correct, so we have updated the formula accordingly for this article and in the glossary. The formula only changes slightly and the tests from the previous article are very close to the same as well. However, for the sake of transparency, we are highlighting this at the beginning of the article. The new formula: SIERA = 6.145  16.986*(SO/PA) + 11.434*(BB/PA)  1.858((GBFBPU)/PA) + 7.653*((SO/PA)^2) +/ 6.664*(((GBFBPU)/PA)^2) + 10.130*(SO/PA)*((GBFBPU)/PA)  5.195*(BB/PA)*((GBFBPU)/PA) Over the last several days, we have covered plenty of introductory ground with regards to the ERA estimator SIERA, from explaining its evolution to demonstrating the creation of its formula. In Part 4, we tested the metric in several different areas against other estimators and found that the goals set forth at the outset of this process were attained: SIERA proved better than any other estimator that appropriately treats HR/FB rates as luck at predicting parkadjusted ERA in the same season, and it proved better in its predictive nature of parkadjusted ERA in the subsequent season than any estimator, regardless of its HR/FB treatment. The final piece of this puzzle involves a bit of statistical exploratory surgery in order to explain why it fared better than its competition. While some may suggest that SIERA represents a mere fivepercent improvement on QERA, the reality of the matter is that the metric vastly improves our understanding of certain types of pitchers consistently undervalued or overvalued by the current crop of estimators. This concept may feel largely abstract, so our time will be spent on exemplifying what we mean through archetypal personifications. Recall that this method of estimation corrects a few problems inherent elsewhere, such as inconsistent denominators or the inclusion of insignificant regression terms. SIERA uses (GBFBPU)/PA to keep groundball rate normalized by PA like walk and strikeout rates are, so pitchers who allow an abnormal percentage of balls to be put into play can be better estimated. Similarly, we can also better estimate the ERAs of pitchers capable of inducing a surplus of worm beaters, who are therefore able to erase the inevitable groundball singles with double plays. SIERA also better estimates ERA for pitchers who are vulnerable to home runs but above average in the preventing base runners department, limiting dinger damage. In the coming paragraphs, you will be visited by three pitchers, representative of the archetypes that other estimators consistently miss, but are now captured correctly viewed through SIERA. Pineiro had a 3.56 parkadjusted ERA last season to go along with a 3.1 percent BBrate (1.14 BB/9), 12.1 percent Krate (4.42 K/9), and somewhere around a 61.3 GB% depending on the source. Consider the various ERA estimators and their attempts at approximating his runs allowed per nine frames:
ERA = 3.56 What should stand out initially is that Pineiro is precisely the kind of pitcher that exacerbates the denominator problem discussed earlier. He allows 84 percent of hitters to put the ball into fair territory, compared to the league average of 73 percent. Therefore, having a high groundball rate is particularly important and his well, well aboveaverage rate is very helpful. However, QERA only considers the average effect of ground balls on ERA and underestimates the importance for someone in Pineiro’s situation. For instance, if Pineiro maintained his 3.1% BBrate and 12.1% Krate, but had a 40%/20%/40% breakdown of GB/LD/(FB+PU), then his SIERA would be 4.56 and his QERA would be 4.55. Instead, going from 40 percent to 61 percent ground balls gives Pineiro an extra boost, as that 21percent difference is part of a much larger set of balls in play. Using xFIP leads to another problem in that Pineiro walks so few batters that he has fewer runners on base in the rare occurrences that he does give up home runs. FIP and xFIP multiply home runs and expected home runs, respectively, by 13 and divide by innings pitched. That treats the effect of blasts as constant for all pitchers, but the damage for someone like Pineiro is largely counteracted by his lack of ducks on the pond. Pineiro does allow plenty of ground balls, though, which can lead to singles, but he allows enough of them to the point that he frequently doubles runners off, as he induced 29 double plays last year. This is the kind of adjustment that SIERA makes, as it allows for a quadratic term on ground balls. Both QERA and xFIP are clearly high for the reasons above, but FIP is nearly as close as SIERA with Pineiro and is on the low side, for the reason that Pineiro only allowed 6.5 percent home runs per fly ball, below the league average. This explains why tRA does a little better than SIERA for sameyear ERA predictions, as it attributes this performance aspect, which is largely luckladen due to its inconsistency, to skill in addition to crediting him for his low 15.8 percent linedrive rate, which we know is also largely luckdriven. Along the same line of reasoning, this is why SIERA bests all others at predicting parkadjusted ERA in the following year, as it expects such lucky marks to regress. Santana is another example of a pitcher who is much more accurately estimated through SIERA. From 200409, his average ERA (not weighted, just a quick average of his six ERAs) was very low, but estimators perpetually overestimated what his ERA should have been. The estimators produced the following in that same span:
ERA = 2.93 The real issue is that FIP and xFIP are too bearish on home runs, neglecting to realize that his prowess when it comes to whiffing and walking mitigate the results even if he lacks the groundballing tendencies as other star pitchers; as with Pineiro, fewer baserunners lead to fewer multirun blasts. QERA is not specific enough about the interactions to properly nail down the ERA estimation, and by correcting for the shortcomings discussed in this paragraph, Santana exemplifies the impact on estimation SIERA brings to the table. To see this, let’s look in more detail at Santana’s actual home runs from 200409. He gave up 146 blasts, but 72 percent of them were solo shots, while the league average was 57 percent. Santana allowed 1.33 runs on the average home run, 17 percent lower than the league’s 1.59 runs on the average homer! Had the coefficient on home runs in FIP been 17 percent lower, his FIP would have drop by 0.24, making up nearly half of the difference. Webb is known for his extreme groundballing ways, and since SIERA’s groundball percent terms have a negative quadratic on ERA, the more grounders induced, the greater the impact on run prevention. This is unlike others, which treat grounders as constant or offer diminishing returns as the rates get higher. The rationale is that ground balls lead to a lot of singles, but those singles can be erased by double plays. Consider the following chart, showing Webb’s 200408 numbers:
ERA = 3.01 Webb’s SIERA is far closer to his ERA during 20042008. Those double plays were a major reason why this happened. Webb got 0.96 DP/9 IP, while the league averaged 0.81. This is more impressive, considering his WHIP was 1.24 to the league’s 1.40. His ratio of double playstobaserunners was .0860, well above the league’s .0646. Webb was somewhat vulnerable to singles due to his groundballing ways. He allowed singles on 78.2 percent of hits in play compared to the league average of 74.7 percent, but he was less vulnerable to extrabase hits and was able to erase those runners back off the basepaths with double plays. This last point is important, because SIERA adjusts for a common criticism of DIPS metrics, since it is based on a regression output. Many times you'll hear about DIPS for the first time and wonder if certain pitchers are more prone to doubles and triples on balls in play, even if they do not allow any more hits. As the pitchers who fit this criteria would presumably not be groundball pitchers (since ground balls have lower ISO on balls in play than fly balls do), a regression analysis will credit groundball pitchers for this double and tripleprevention skill by giving them lower ERAs if fellow groundball pitchers have had lower ERAs for this reason, too. Conclusion Pineiro, Santana, and Webb are obvious examples of where the benefits are for using SIERA to estimate ERA. Each of them saw other estimators inaccurately estimate their ERAs. Pitchers like these demonstrate the reason why SIERA predicted nextyear ERA better than any of the other available estimators. Developing better baseball statistics is not an academic exercise. It is a way to better value what happens on the field, allowing analysts to better understand and predict performance and front offices to build better teams.
Matt Swartz is an author of Baseball Prospectus. 26 comments have been left for this article. (Click to hide comments) BP Comment Quick Links seanpotter (50976) I'd like to calculate some pitcher SIERAs myself using FanGraphs data. They include popups in their fly balls number. Should I negate that to avoid redundancy? Feb 12, 2010 10:15 AM MHaywood1025 (46036) What is the probability a ground ball goes for a hit, given there is a man on first? It makes sense that it is different than the probability it goes for a hit with nobody on, but is it really significant enough to consider it a good thing for run prevention? Feb 12, 2010 10:39 AM sunpar (38553) The probability of a single doesn't change significantly with a man on. Obviously if you have a guy on first who attempts a ton of steals, you may increase ground ball BABIP a bit since the SS or 2B may be forced to cover, but this difference is probably not significant, and will likely be caught by the regression anyway. Feb 12, 2010 10:55 AM My browser isn't letting me reply to individual comments, so I will reply in bulk to the first three I see Feb 12, 2010 11:05 AM lopkhan00 (1994) Frankly, what this shows me is that (assuming that SIERA is the new gold standard, and I'm willing to accept that) ERA, adjusted for park influence, has always been a damn good stat and was closer to reality than all the previous attempts to improve upon it. Feb 12, 2010 12:57 PM sunpar (38553) That's not quite what is being shown here. The ultimate goal is to figure out how much a pitcher's performance is his alone and how much of it is defense/luck. ERA will always measure both, but we want to separate the two. Over the long run, defensive and luckbased contributions should be zero and thus pitchers' ERA should be close to their fielder and luck independent ERA. Feb 12, 2010 13:18 PM Juris (1283) The "presumably" in your final sentence could be tested explicitly by looking at the "next year" predictions for these pitchers. While the article compares the average "same year" predictions for the different metrics, I'm not it shows us that Siera is significantly better in predicting the "next year" for these pitchers, given the previous year or "base years" (typically this involves a weighted average of previous three years). Feb 12, 2010 14:50 PM One Flap Down (30321) Yeah, I'd like to see SIERA applied to 2008 numbers to see how well it predicted 2009 visavis FIP, QERA, etc. Feb 12, 2010 17:49 PM I put that in the comments of the last article. With the updated formula though, it's basically a tie between FIP and SIERA (1.108 for FIP and 1.107 for SIERA, with QERA at 1.185, and xFIP at 1.226 and TRA at 1.307. Feb 12, 2010 21:41 PM lopkhan00: With all due respect, I have absolutely no idea how you could come to this conclusion from this article. The 4th article showed that ERA doesn't do nearly as well predict future ERA as any of the estimators, and in this article, ERA was the baseline to check the other estimators against. I can't follow your thought process at all, but please expand if I'm missing something. Feb 12, 2010 21:37 PM jinaz (28427) Matt, Feb 12, 2010 18:29 PM Justin, what we've been doing is finding out individual pitchers' SIERAs and weighting them by the IP, so it would be like getting SkillInteractive Earned Runs for each pitcher, adding them up, and divided them by team innings (and multiplying by 9). You want to get the interactions to be relevant for the relevant pitchers. Feb 12, 2010 21:43 PM dREaDS Fan (51622) shanecris (48268) Can we get a Siera projections for all pitchers? For Fantasy purposes that would be terrific.. Feb 14, 2010 11:10 AM GoodKingJohn (21105) from the equation, we have 7.653*((SO/PA)^2). however, as k's go up, this value would go up, and result in a LARGER SIERA value. does this make sense? Apr 25, 2010 06:31 AM There is also a negative linear term in there for regular strikeouts that will always dominate. You would need to have over 100% strikeouts to have the positive squared term have a bigger effect than the negative linear term. May 09, 2010 17:15 PM sdgeiger (6577) They are all valuable in different ways, and there are also three versions of WARWARP. Aug 30, 2010 15:54 PM GoodKingJohn (21105) revisting siera part 5. Mar 26, 2011 08:07 AM Not a subscriber? Sign up today!

So... the obvious question after seeing a list of "betterthantherestat"... are there particular pitcher types that SIERA has trouble with?