Earlier this week we introduced the run estimator SIERA, providing a general summary of its purpose as well as the evolution of its development. Today, in Part 3, our focus will shift to the quantitative side of the metric, offering a detailed look at the data used to derive the formula as well as specifics pertaining to the regression analysis techniques used. The transparency should provide a better understanding of the integrity of such a process as well as a few insights into the SIERA-laden approach towards pitcher valuations.
All data used throughout this process, be it the calculation of SIERA or the various other comparative estimators, came from Retrosheet, a monumental achievement in the world of data without which several advancements in the field would not exist. The first step involved extracting seasonal tallies from the main events table, with statistics being grouped by pitcher, team, and year. This way, a pitcher with stints on various clubs throughout the same season would carry a different entry for each; Cliff Lee as both a Phillies and Indians pitcher last season. Next, using the Lahman database, the pitching park factor was added to each row in the table.
Park-adjusted ERA was then calculated, though only half of the park factor was applied to the individual pitchers given that only half of a team’s games are played at the home stadium. If a pitcher ended up with a PPF of 105, instead of taking 95 percent of his ERA, 97.5 percent was taken, equating to one-half of the difference between the actual park he called home and one considered to be neutral. With the adjustment applied to raw ERA, the next issue to bypass involved batted ball reliability.
While Retrosheet provides a fantastic wealth of information, batted ball data is realistically only usable from 2003-present. The major reason for this involves how balls put in play were scored, as the processes implemented have not been consistent. Before 2003, batted balls were only recorded on outs, meaning that a ground ball single through the third base hole counted as a single while a ground out to the second baseman went down as a grounder. Both are ground balls, but this rather vast issue precludes the usage of batted balls prior to that season.
Only data from 2003-09 moved onto the next round given this restriction. With that table in place, the QERA formula was unfoiled and the nine emerging terms were calculated for each row in the database table. The data was then ready for further processing and rigorous study.
SIERA was first estimated with 10 parameters: an intercept and the nine aforementioned terms that surface once QERA is unfoiled, which involved regressing park-adjusted ERA on all nine terms. The results can be seen below:
VARIABLE COEF. T-STAT P-STAT Constant 6.368 16.97 0.000 SO/PA -18.341 -7.10 0.000 BB/PA 9.471 2.00 0.046 (GB-FB-PU)/PA -1.807 -1.60 0.110 (SO/PA)^2 10.254 1.98 0.048 (BB/PA)^2 6.833 0.33 0.742 ((GB-FB-PU)/PA)^2 -7.063 -3.93 0.000 ((GB-FB-PU)/PA)*(SO/PA) 9.661 2.38 0.017 ((GB-FB-PU)/PA)*(BB/PA) -3.208 -0.44 0.661 (BB/PA)*(SO/PA) 2.828 0.18 0.857
Before getting into what the data originally said, a description of the columns is in order. The first column lists the variable in question while the coefficients were estimated by the regression. The t-statistic describes how many standard deviations from zero the coefficient strayed and the p-statistic tells us that, if the effect of the variable on park-adjusted ERA were actually zero, what the probability is a coefficient that far from zero would surface.
It is commonly accepted that p-stats less than .05 or .10 are probably different from zero. Unfortunately, reliable data for balls in play only exists from 2003-09, which means that we are unable to get many coefficients that make sense to be significant. Our intuition helped to build this model, with an understanding that as pitchers get back on the mound and throw some more games even more accurate results can be had. Note that the above table does not show the final formula for SIERA, but rather the original estimation using the entire formula for QERA regressed on park-adjusted ERA. Also note that the data used to build the table above originally came from 2003-08, not 2009; the latter year was excluded for the purpose of eventually testing a regression on an outside element. However, to contrast it with the table below, the table above includes 2009 data as well even though our original tests left out 2009 data for honest testing procedures.
What immediately stands out is that the quadratic term for walks is not significant, the .74 p-stat indicates that there is a 74-percent chance that you would get a value further from zero than 6.833 if the true quadratic effect of walks on ERA was zero. The conclusion: the effect of walks on ERA is linear but perhaps with interactions with strikeouts or ground balls. It is also evident that the effect of strikeouts and walks is non-existent as well. This seems plausible, seeing as there is no reason to assume walks increase ERA more for high strikeout pitchers as opposed to those with low whiff totals.
Two quadratic terms are significant as is an interaction term. The interaction between walks and ground balls could have been dropped, but intuition chimed in and kept it afloat because the significance of the interaction of strikeouts and ground balls forces honesty and requires the presence of the former interaction. The reason this interaction is believed to be clinically significant is that pitchers who strike more batters out allow fewer singles and need fewer double plays. This is true for walks as well.
Removing the other two insignificant terms sends the walk and ground ball interaction term closer to significant, but still far from it. It is our belief that including this interaction gives a more accurate prediction of a pitcher’s skill level and that the reason that the coefficient is insignificant is that the sample size is too small. Some of the other effects are even crisper when the regression is analyzed with the two insignificant terms removed:
VARIABLE COEF. T-STAT P-STAT Constant 6.262 28.07 0.000 SO/PA -18.055 -8.39 0.000 BB/PA 11.292 12.81 0.000 (GB-FB-PU)/PA -1.721 -1.57 0.116 (SO/PA)^2 10.169 1.97 0.049 ((GB-FB-PU)/PA)^2 -7.069 -3.94 0.000 ((GB-FB-PU)/PA)*(SO/PA) 9.561 2.38 0.017 ((GB-FB-PU)/PA)*(BB/PA) -4.027 -0.58 0.563
Four terms are worthy of further explanation as they are significant, or close enough to significant, like in the case of the linear term in (GB-FB-PU)/PA since its square proved to be significant. Each will be explained separately:
(SO/PA)^2 has a significant and positive coefficient, even though the linear SO/PA has a significant and negative coefficient. Essentially, this means that although increasing strikeout rate lowers ERA, whiffing more hitters has a diminishing effect on run prevention. If you take the derivative of SIERA with respect to SO/PA (the amount that SIERA changes relative to the change in SO/PA at a given level), you get the following:
-18.054 + 20.337*SO/PA + 9.561*(GB-FB-PU)/PA
Ignoring the third term for now, but for a given (GB-FB-PU)/PA of zero (to simplify calculations), an increase of strikeouts from 0-1 percent decreases ERA by about 0.179; an increase in strikeouts from 10 percent to 11 percent decreases ERA by about 0.138; and an increase in strikeouts from 20 percent to 21 percent decreases ERA by .097. Basically, strikeouts are more useful with runners on base, as the more whiffs one tallies, the fewer runners that reach base. Strikeouts have a gradually diminishing effect on run prevention as someone who strikes out 90 percent of hitters he faces is not doing much harm by allowing a few more balls in play.
((GB-FB-PU)/PA)^2 has a significant negative coefficient, adding to the negative coefficient on (GB-FB-PU)/PA. The more ground balls a pitcher allows, the more he will benefit from even more worm beaters, due to the fact that ground balls are often singles. Due to this, there will be more runners to double up. The derivative of (GB-FB-PU)/PA can be seen below:
-1.721 – 14.138*(GB-FB-PU)/PA + 9.561*SO/PA – 4.027*BB/PA
Using league average strikeout and walk rates, a (GB-FB-PU)/PA from 0-0.05 would drop ERA by 0.104; from 0.05-0.10 would drop ERA by 0.139; and from .10-.15 would drop ERA by 0.174. The more ground balls a pitcher gets, the more he will benefit by getting more of them.
(BB/PA)*((GB-FB-PU)/PA) has an insignificant but negative coefficient. This suggests that pitchers with higher walk rates will prevent more runs by generating ground balls than pitchers with low walk rates, but it does not offer statistical proof. However, for reasons suggested above, the significance of the interaction between strikeouts and ground balls implies that this is probably true as well, and that only sample size is holding us back. There are two reasons toexplain the negative effect.
First, pitchers who put more runners on first will get more double plays from generating ground balls than pitchers who do not put many hitters on first base.
Second, pitchers who allow more fly balls allow more home runs. These are more likely to be solo home runs if the pitcher does not give out many free bases on balls so they will not be as damaging. Getting ground balls is even more important for pitchers with high walk rates since they can avoid multi-run dingers. It seems particularly inaccurate that FIP puts a coefficient of 13 on HR/IP for all pitchers, regardless of their walking exploits. Solo shots do something different to ERA than grand slams.
(SO/PA)*((GB-FB-PU)/PA) has a significant positive coefficient. Pitchers who strike out a lot of hitters benefit from ground balls less than pitchers who allow many hitters to put the ball in play.
This follows the same logic as the walk/ground-ball interaction, since pitchers who strike out fewer hitters allow more balls in play. This leads to more runners reaching base who will conceivably be doubled up ground balls and allowed to score on multi-run fly ball blasts.
Thus, these four points have shown us that strikeouts have a diminishing return as you accrue more of them, ground balls have an increasing return the higher your tally, and ground balls are more beneficial to pitchers who allow more walks or balls in play, especially because fly balls are more detrimental to pitchers who allow more runners on base.
How beneficial are these results? In Part 4 of our introductory series on SIERA, the estimator will be put to the test at both predicting same year ERA better than other estimators that use similar statistics and at predicting future year ERA than any other estimator out there.