Last week, Matt and I introduced and explained the derivation of SIERA, Skill Interactive Earned Run Average, a stat designed to pick up on the interactions between metrics within a pitcher’s control and let us know his park-adjusted ERA based on a set of sustainable skills. One of the introductory articles focused on testing the metric against its peers, and while it stacked up well against the competition, some concerns were raised over the the disconnect between FIP and xFIP in our testing, as the latter should theoretically perform better than the former when it comes to predicting ERA in the following season.
It turns out these concerns were well-founded as a trip back to my coding revealed a flaw in how xFIP and its inputs were being computed. I shouldn’t have made this mistake, and I should have caught it earlier on, but now it is time to rectify the error, as should be the case in any metric-creation process; tests are run, flaws revealed, corrections made, and improvements constantly applied. When the xFIP calculations are corrected, it does beat FIP, as it should. However, it is also in a literal dead heat with SIERA, with SIERA still coming out ahead in RMSE testing, but barely (1.159 for SIERA vs. 1.162 for xFIP), which does not invalidate anything, but rather invites some food for thought as far as the similarities and differences for the two stats.
So what does this tell us? Well, firstly it says that I need to get my head out of my (expletive deleted) and be much more careful in my coding and reviewing, but it also says that SIERA and xFIP are the best estimators around right now, and a thorough picture of run prevention at this stage should entail looking at both metrics. It also tells us that we should continue to work on refining SIERA, factoring in a lot of the excellent suggestions posted by readers on this site and others. Though they are closely linked, SIERA and xFIP are calculated from such different angles that it should be clear each will have its strengths and weaknesses in certain areas.
SIERA remains particularly strong for pitchers with very high and very low ground ball rates, and is very strong for pitchers with relatively average strikeout rates. Additionally, it works very well for the above average pitchers in terms of overall quality, making it a worthwhile tool for fantasy competitors. The differences are still pretty slight, so it should be repeated that SIERA and xFIP should be used in conjunction to one another right now in order to paint the most accurate portrait, but some of the ideas discussed in our threads were fantastic and will be potentially applied as we continue to develop the metric.
A final thought before signing off for now: There has been plenty of discussion of how we introduced the metric, and ways to improve in that forum. Our goal is to be as transparent as possible, and so as BP moves forward and introduces or refines metrics, is there anything specific that did or didn’t work in terms of really breaking down, in an in-depth fashion, how a stat is derived?
Thanks for the update. I don't mind the errors so long as they are handled in this transparent and respectful manner. I think this says a lot for the great "peer review" within the sabr community. (As well as existing work.) Keep up the good work and I'm excited to see how SIERA can be improved. In the mean time I'd love to see a SIERA vs FIP comparison for specific types of pitchers. Considering the great quality of work thus far, I expect that will be up within the week.
In article four we did these comparisons -- not sure what else you're looking for, or if you're thinking of something differently, but click the link in the article.
I meant to write xFIP not FIP so I'd be looking for those same charts but with the corrected xFIP. Also is there any hope of adding tRA* to the testing?
I agree with the importance of peer review. In the case of new metrics, or improvements on old ones, it's a good idea to post the articles "outside the pay wall" so as to maximize the transparency and feedback that you get -- including commentary on other websites that may involve non-subscribers to BP.
Dan Fox's rollout of his baserunning metric was a good precedent to follow (though IIRC he didn't make every aspect of it open, and some of it was published simultaneousl on his own website, while the articles remained beyind the paywall on BP).
I hope the development and discussion of the new defensive metric will also remain "open" on the BP site.
There's a standard that is sometimes insisted upon in academia, and this is that researchers make "replication data sets" available to readers and critics. Making available the actual data set (not just a reference to the sources) upon which the metric was devised, or the hypothesis was tested, will open up the process even more.
While applauding the correction, I will make one recommendation for "a way to improve in [how we introduced the metric]":
Changing your claim from "we beat everyone" to "we're as good as xFIP, possibly very slightly better" deserves more than an Unfiltered (and thus unarchived) post under the heading "SIERA Update" (as opposed to "SIERA Retraction" or "Reduced claims for SIERA" or similar).
Unfiltered posts are in the archive, but not indexed. In fact there's a lot that's in the archive that's not indexed in the search engine, because for some reason BP deletes the links to "former authors" from the search engine. They should include ALL articles in the search. I wrote to the new "editor" to point this problem out some months ago and got not even an acknowedgment in return.
As far as publishing in Unfiltered vs. the main article space is concerned, I think a lot of non-subscribers read Unfiltered, and this wasn't a bad way to get the new result -- the correction -- out. But publishing a full article in future would be a good idea, laying out some of the comparisons again, including some that were made only in response to comments to the earlier articles.
Bingo. As we continue to improve the metric and develop new tests and such, this will be referenced as will the results. This isn't a one-off or anything, but a way to get it across for everyone.
It seems that there should be a more analytical expression of where each system performs best than in your 4th paragraph. If you can correlate the relative performance of each to ranges of the input parameters, you ought to be able to find an intelligent way to combine the two metrics to create an ubermetric. Maybe you can vary a single input parameter at a time and group your input data into standard deviation ranges as far out from the mean as you have significant sample size and compare performance?
As an aside, as you continue to tune your performance metrics, you might want to think about over which range you really want to optimize. I think it's less important to be accurate for poor performance. I don't care if someone is 5 or 10 wins below replacement level -- I still don't want that person on my team.
That's a bit of an extreme example, but if we could tune the predictive performance tools to be optimized in the region of the greatest marginal value, wouldn't that add value?
well, first of all, i'm not entirely sure why the wheel needed to be reinvented here, but for the sake of argument let's just say it did. why couldn't you have waited a year before giving it such a prominent place in the new BP2010 book--iron out the rough edges? maybe put an essay in the back or something? i mean, the whole D-Backs section of the book is about how they performed compared to SIERA, and i couldn't figure out if it was that they underachieved, overachieved or neither. in any case, clearly, this stat still needs some tweaking before it can be really described as the top predictor on the market.
Maybe I'm reading the post incorrectly, but I don't see any mention of tweaks or refinements in SIERA itself. The error was in coding xFIP for comparison.
Exactly -- the post was to rectify the mistake on my part that showed xFIP testing so poorly. SIERA is still SIERA, though we're going to continuously refine the metric.
thanks Eric. I guess i'm just confused about what the heck SIERA is. is it a prediction of ERA for next year? is it an indication of what the ERA should have been last year in a neutral world?
SIERA is a new ERA estimator that is designed to eliminate the defense, ballpark effects, and luck from a pitchers actual ERA. If a pitcher had a lower ERA than SIERA, it was due to one of these three factors. This is useful in analyzing past performance as well as predicting future performance.
Eric's post in Unfiltered, above, shows that SIERA is about as accurate as xFIP as a predictive model.
If xFIP and SIERA come at very different angles and have similar accuracy, perhaps there is an opportunity to average the two and improve accuracy over either individually. If the errors don't tend to be in the same direction or for the same players, could be a large improvement.
I want to echo others who have said that if you really want to test SIERA, you are going to need to test it against tRA which is the gold standard in overall pitching metrics. tRA is clearly better than FIP and xFIP. If you want to see how good SIERA is, put it up against the big dog.
@Flynnbot
It's an indication of what ERA should have been last year in a neutral world that is evidenced by its ability to predict the next year's ERA. Luck isn't persistent, but strikeout, walk, and ground ball rates are, so the combination of these three skills will give you a good estimate of ERA. xFIP is very strong as well.
@royalsnightly:
Please read the linked article. We DID put it up against tRA and tRA did no better than FIP and distinctly worse than xFIP and SIERA, well modeled metrics of pitcher skill. It is a very flawed metric that fluctuates largely with line drive rate-- which has an intra-class correlation of 0.007, maybe the lowest ICC I've ever seen. tRA is not a big dog at all. Use xFIP and SIERA if you want to measure skill. It's silly to name tRA as a gold standard when it was not tested anywhere but here and did not outperform even FIP with more inputs.
To those suggesting an ubermetric:
That's definitely a good idea. Maybe we will incorporate that into future testing, but for now, I think that it would be great to use both.
To those asking for more detailed tables of subsets of pitchers:
The differences were small enough that we thought it was best to just publish more general ideas. Due to the construction of SIERA, it made sense that it did better with extreme ground ball rates in either direction, and moderate strikeout rates, but statistically these differences would not have been significant.
We look forward to continuing to improve SIERA. It's really just SIERA 1.0 right now, and it's already up there with xFIP as the clear best luck-/defense-neutral ERA estimators. We understand the frustration with this, but we want to stress that any good metric like this should be continually tested, and would have been regardless of this finding about xFIP's comparable strength.
How much better does SIERA need to be than xFIP to outweigh its increased complexity? If its accuracy doesn't improve with future iterations and it's not significantly better at predicting performance of specific types of pitchers, why would one use it over the more parsimonious option? I'm not saying it can't be improved, but if it isn't then it's DOA right?
"The differences were small enough that we thought it was best to just publish more general ideas."
"but statistically these differences would not have been significant."
Seem to be quite a bit smaller than what was in the original series, no? I seem to recall many mentions of large differences.
Here's just a quick quote from Part 4 that Eric linked to:
"it is apparent not only that SIERA is the best ERA estimator currently available, but specifically that it is exceptionally strong at measuring the skill level of specialized kinds of pitchers."
Obviously, it's not currently the best ERA estimator available anymore, but I'm really confused about the second part. Aren't the "exceptionally strong" from this quote and "not statistically significant" from the previous quite referring to the same - or at least similar - subsets of pitchers?
"It's an indication of what ERA should have been last year in a neutral world that is evidenced by its ability to predict the next year's ERA. Luck isn't persistent, but strikeout, walk, and ground ball rates are, so the combination of these three skills will give you a good estimate of ERA. xFIP is very strong as well."
I get this, but would like to note that selling SIERA as a predictor of next year's ERA in the same book where you're selling a differing PECOTA prediction of next year's ERA is a bit of a mixed message.
Again, I have a feel for both methodologies, and the differing intents. But if the book's editors felt it necessary to spell out stuff like 'DL means "Disabled List", and OPS means "On Base Percentage Plus Slugging Percentage,"' it seems a little more ink could've been spent shedding additional light on exactly how to use the new tool.
All in all, both The Book and the new tool appear to be pretty good products.
One suggestion - maybe once you get your SIERA legs under you, maybe you can keep it updated during the regular season. This would avoid the temptation to think of SIERA as "Component ERA, only a little better and 4 months later."
Really? I don't agree that SIERA is more complex than xFIP-- remember that this whole problem came about because Eric and I were unable to properly code xFIP. Yes, the coefficients are round numbers, but given that no one calculates either one in their head, I don't think Markov chains (xFIP) are necessarily easier to understand than regression (SIERA) at all. Both involve strong assumptions that aren't always true. xFIP assumes that individual events are of the same value and not more likely to occur in one context than another, and SIERA assumes that correlation between other variables like BABIP and DIPS variables are going to be persistent and that measurement error in batted ball rates aren't too large as to affect the regressors.
The two estimators come at things from very different angles. If they are both equally good at predicting ERA, I don't think it would be smart to throw either out.
Put it this way-- if I told you that a pitcher's xFIP was 4.00, your best guess is that his ERA in a neutral environment would be 4.00. If I then told you that his SIERA was 3.50, you should bump that down. If I had told you that has SIERA was 4.50, you should bump that up. Similarly, if I told you that a pitcher's SIERA was 4.00, then you should predict his ERA would be 4.00 in a luck neutral environment. If I then tell you his xFIP is 4.50, bump your guess up, and if I tell you instead that his xFIP is 3.50, bump your guess down. They both contain information that is different, and it would be dangerous to throw one out because the coefficients aren't round numbers.
with a RMSE of 1.1259, so not much else is added there. Regressing on correlated things like this doesn't get much of anywhere in some cases.
Brian Cartwright apparently found below that SIERA does a little better at next-next-year and next-next-next-year ERA too, though I can't verify that other than to say Brian has nothing at stake and is very smart, and that I like his research supporting SIERA :)
OOC, is the difference in the RMS error between SIERA and xFIP statistically significant? I'd expect it to be based on how large I assume the sample is, but it would be nice to provide that information.
I doubt the 1.159 vs. 1.162 is statistically significant, but I think it's enough to think that as SIERA gets more data, it should be able to get there we think. It also seems to be slightly winning in various tests, and doing better in a linear regression to estimate next-year ERA rather than just doing it directly. But the point here is that we've got something that should be at least as good from a totally different angle-- meaning the combination should be great-- and we have clear room to improve once pitchers get up on the mound and throw us a few more frames to put in the data, and once we incorporate all the great suggestions we got through this process.
Yes, I'd say a first degree polynomial is simpler than a second degree polynomial - rounding of the coefficients doesn't have anything to do with it. If you don't gain any accuracy with the squared term it shouldn't be there. I think the concept behind SIERA makes a lot of sense and I'm hoping future iterations prove to be more accurate, but version 1.0 appears to be a more complex and no more useful predictor than xFIP.
evo34 beat me to my second thought. The assertion that the average of the two is more accurate than either one used separately is one that can and should be tested. If the errors of the two models are correlated averaging them isn't going to do much.
Keep working on it though. I like the concept and it seems to at least have the potential to better predict certain unique cohorts, which would be quite valuable.
I mean, is a squared term really something that is all that complicated? Figuring out how to establish the run value to cook up the 3.20-ish moving constant or which is the right HR/FB number to make xHR seems less straightforward than multiplying a number by itself to me, but that's probably about the most subjective difference I can think of.
I appreciate the encouragement, though. Thanks. If you look at my RMSE posts above, you might be encouraged a bit more too.
The confusion is that xFIP is apparently much better than we thought because it is tricky to code and we did it wrong the first time. The difference between SIERA and FIP/tRA/QERA is still very large, as it was before, but xFIP and SIERA are both close in a lot of categories, enough that for this quick correction we didn't expand on the minor differences.
What is the best method to use the SIERA numbers in the BP2010 book for predicting ERA for this season? I am thinking about this from a fantasy baseball persepctive where unfortunately I can't convince any other owners in my league to switch to SIERA over ERA.
The most important thing to do is probably look for pitchers who have big discrepancies between their SIERA and their ERA, and then see if projections reflect that gap. Obviously, SIERA is going to regress to the mean like any statistic, but it won't be as volatile as ERA. Pretty soon we will have the SIERA numbers all up online, and the first thing you want to do as a fantasy player is look for gaps in SIERA and ERA. Some of the biggest ones that I remember seeing among people good enough to draft as starting pitchers were Ricky Nolasco and Cole Hamels. I don't really do fantasy baseball myself much, but I would guess that Nolasco is probably one of those guys you could nab late in the draft and then have him put up an ERA under 4 when no one expected it. Hamels is one of those players who might slip back in the draft but probably is someone you want to get before he's being drafted. I think two players that are great according to SIERA and maybe just very good according to other metrics are Justin Verlander and Javier Vazquez. This will be a more fun conversation when we get the numbers all up, I think! I remember only a handful of guys off the top of my head who's SIERAs taught me something, but there's probably more mixed in where the 2009 ERA and projected 2010 ERA won't look as much like the 2009 SIERA as it should, and that will be your green light/red light.
Whether one is a fantasy baseball fan or not, this is the sort of carry away any fan is going to want to see. A Phillies fan, for example would like to know this about Hamels.
Also, does SIERA have any uses during the season? If on July 1st, Brad Penny has a 2.85 ERA for the Cardinals, and a 4.56 SIERA, would that have implications for his second half (I'd assume so). Whether I was a fantasy owner of Penny or a Cardinals fan, that is the sort of commentary I'd like to see.
Definitely a great in-season estimator, especially early on when ERA is so noisy. One of the things that Nate Silver said about QERA was that it stabilized quickly-- SIERA will have the exact same benefit. The point of SIERA is to talk about how a collection of skills will produce an ERA. A July 1st ERA is full of noise, but SIERA will have skills only in there with less noise.
One of the articles I have in my queue is to repeat something I did last year with ERA and FIP, but with SIERA and ERA. Basically, that article looked at guys with big gaps in their ERA and FIP halfway through the year, and how they fared the rest of the year.
I plan on using SIERA in fantasy analysis this year. I used QERA often because it stabilized quickly, and could be easily adjusted in a spreadsheet for hypotheticals, such as, "What would happen if Cliff Lee's walk rate went back to pre-2008 levels?"
You'll get your commentary from a fantasy perspective, but the beauty of using stats like this for fantasy purposes is that it also works for real-life coverage.
Thanks Matt...I was thinking it could be used along the same lines as QERA and tRA as a good in-season ERA predictor when there's a disparity. Last year this little exercise helped me snag Blanton & de la Rosa around the All-Star break.
Eric provided me with the raw pitching data, and I ran my own tests which I posted at 'The Book' blog.
I confirmed that SIERA gave the lowest total errors in y1 (next year), although not by that much ahead of xFIP and FIP. I also looked at y2 and y3, figuring if this was really measuring the persistent skills it should still be good two and three years out, and again SIERA was slightly ahead.
Should be right after the PECOTA cards, I think. It'll be on the Statistics page, and we might be able to put together a nice hypothetical SIERA Calculator where you can just plug things in.
-Did you guys test tRA*, the regressed version of tRA? I don't know if the guts of tRA are out there, but I doubt that Graham MacAree would say "no" if you promised not to disclose it.
-Why not try something other than a season-to-season calculation for these metrics, such as an in-season correlation between a random selection of half a pitchers batters faced and the other half of his batters faced. As a fantasy baseballer, what I'm most interested in is a stat that can tell me what to expect in July from what I saw in April, June and May; I have PECOTA and CHONE and Marcels and every other projection system to tell me what to expect next season. (And I hope they all do better than plain old xFIP!)
I don't think we were able to get the formula for tRA* but the point is that tRA is largely dependent on line drive rate, which has a 0.007 intra-class correlation. So unless tRA* reflects that, it's bound to have some problems too. I heard Graham say once that (LD)/(LD+FB+PU) had a 0.4 high correlation, but that's because FB+PU has a high correlation. The point is that LD/(LD+FB+PU+GB) has a 0.0 correlation, so trying to credit pitchers with line drive rates different than their team line drive rates is going to be a problem. If you want to make a regressed projection rather than a formula based on actual statistics, though, you might as well use PECOTA and other projection systems.
Matt and Eric figured out the factors that are most relevant to predicting future ERA, and SIERA takes all those factors and predicts future ERA more accurately than any other system.
No, it's a skill based indication of a pitcher's talent (at creating ERA) in 2009. This is going to be a better indicator of 2010 ERA than 2009 ERA. A prediction of 2010 ERA is something along the lines of PECOTA. I imagine SIERA informs PECOTA a lot.
SIERA normalizes past ERA for things outside a pitcher's control and is therefore a better indicator of pitcher skill than ERA.
There's your 20 word explanation. If you want more:
It is not explicitly a predictive model because it does not incorporate aging, but it is as good as or better than any other metric that does not incorporate aging at predicting future ERA.
Yeah, if people were more familiar with that scale, that would have been better. Even Fangraphs converted tRA into tERA recently though, just because of that problem. There is also FIP with an RA adjustment to the constant out there somewhere too, but people prefer FIP as ERA-scaled.
I can see why FIP is scaled to ERA, because it imagines all pitchers have the same batted ball profile. And says HRa is a skill. I see xFIP as reclassifying that HRa skill.
Anyway, this is why I think using one number (.92 or whatever) to go from ERA to RA is fine. When we start looking at a pitcher's GB profile should we start to incorporate a changing ERA to RA factor? How different is the ERA to RA conversion for extreme fly ball pitchers and extreme ground ball pitchers? Isn't this a reason to check against RA?
But perhaps a test against RA would include a teams defense. Does GB profile effect ERA to RA difference? Thoughts?
Yes, GB profile makes the unearned run portion of RA go up naturally. But xFIP and FIP both talk about fly ball percentage, which is going to be directly related to ground ball percentage, so the bias applies to all of them. We did run some tests of what "SIRA" might look like and it was pretty similar with small diminished value of ground balls. Since this was a smaller issue, we left it out here, but we very well might make a "SIRA" later, just like xFIP should have varying constants based on fly ball rate and FIP should have varying constants based on home run rate. Note that xFIP and FIP do NOT at all assume constant batted ball profiles, just constant 1B/2B/3B/out distributions.
Does .92 ERA to RA have enough specificity considering that the difference in metrics like Defensive efficiency between the best teams and the worst teams are probably about .010 to .020? I'm not a stat guy, but if the best teams in the league have a defensive efficiency of around .990 and the worst teams are at .975 for defensive efficiency, there's a lot of room there that is attributed to skill that a .92 ERA to RA multiplier might not take into account.
I think of .92 as a league wide average. In reality the conversion rate will change depending on a team's defense but if we are just trying to consider the pitcher's talent we want an average defense. We use .92 to go from ERA to RA when tRA or FIP gives us a pitcher's talent (independent of defense) in ERA and we want it in RA.
xFIP looks at FB, BB, IBB, HP and SO per IP. Each FB is multiplied by the league average rate of HRs, so it's only the FB rate that really matters. Numbers of batters per IP varies from pitcher to pitcher.
SIERA looks at BB, SO and GB-FB-PU per PA. AS the batted ball terms are divided into PA and not the total number of batted balls, their percent is effected by the number of BB & SO. Perhaps it can add HP and IBB, I'd have to reread the articles to see if Matt & Eric already considered nd then decided to exclude them as insignificant.
They are about equally accurate at predicting a pitcher's ERA next year, with a slight edge to SIERA that widens in y2 and y3.
I have not yet had the time to compare this to my method of first generating a projection based on hits, homers, walks, strikeouts, etc, then estimating the pitcher's ERA from his projected wOBA allowed. In this method, BB and SO are regressed somewhat, HRs more, and base hits heavily.
To be fair in comparison, the tests of different formulas have to be conducted on the same data. Run SIERA on my data, or run my formula on the SIERA raw data set that Eric provided. My projections don't have the batted ball data in the final report. I'll either have to carry them them through to the end, or run the projections based only on the SIERA raw data (which is what I'm leaning to.)
But right now I have to finish coding playing time estimates.
64 comments have been left for this post.
Thanks for the update. I don't mind the errors so long as they are handled in this transparent and respectful manner. I think this says a lot for the great "peer review" within the sabr community. (As well as existing work.) Keep up the good work and I'm excited to see how SIERA can be improved. In the mean time I'd love to see a SIERA vs FIP comparison for specific types of pitchers. Considering the great quality of work thus far, I expect that will be up within the week.
In article four we did these comparisons -- not sure what else you're looking for, or if you're thinking of something differently, but click the link in the article.
I meant to write xFIP not FIP so I'd be looking for those same charts but with the corrected xFIP. Also is there any hope of adding tRA* to the testing?
I agree with the importance of peer review. In the case of new metrics, or improvements on old ones, it's a good idea to post the articles "outside the pay wall" so as to maximize the transparency and feedback that you get -- including commentary on other websites that may involve non-subscribers to BP.
Dan Fox's rollout of his baserunning metric was a good precedent to follow (though IIRC he didn't make every aspect of it open, and some of it was published simultaneousl on his own website, while the articles remained beyind the paywall on BP).
I hope the development and discussion of the new defensive metric will also remain "open" on the BP site.
There's a standard that is sometimes insisted upon in academia, and this is that researchers make "replication data sets" available to readers and critics. Making available the actual data set (not just a reference to the sources) upon which the metric was devised, or the hypothesis was tested, will open up the process even more.
I'll just second everything NLBB15 said.