CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

<< Previous Article
Premium Article Future Shock: With Bor... (06/02)
<< Previous Column
Premium Article Checking the Numbers: ... (05/26)
Next Column >>
Premium Article Checking the Numbers: ... (06/04)
Next Article >>
Prospectus Q&A: Chris ... (06/03)

June 2, 2010

Checking the Numbers

Sneaky SIERA

by Eric Seidman and Matt Swartz

the archives are now free.

All Baseball Prospectus Premium and Fantasy articles more than a year old are now free as a thank you to the entire Internet for making our work possible.

Not a subscriber? Get exclusive content like this delivered hot to your inbox every weekday. Click here for more information on Baseball Prospectus subscriptions or use the buttons to the right to subscribe and get instant access to the best baseball content on the web.

Subscribe for $4.95 per month
Recurring subscription - cancel anytime.


a 33% savings over the monthly price!

Purchase a $39.95 gift subscription
a 33% savings over the monthly price!

Already a subscriber? Click here and use the blue login bar to log in.

Earlier in the year we introduced SIERA, an ERA-based estimator designed to more accurately predict pitcher ERA given more stable and significant—both clinical and statistical—inputs. With the season two months in, we are more closely monitoring the returns, and in doing so observed that the Arizona Diamondbacks have a vast disconnect between their ERA as it actually stands and what it should be given what are referred to as controllable and stable skills. Interestingly enough, this was a bugaboo for the desert dwellers last season, as several pitchers finished the year with a similar statistical archetype: very solid K/BB ratios, more grounders than flyballs, poor ERA.

When immersed in a field as much as we are, seeing those former two components usually translates into “good numbers” but the latter doesn’t always agree. This issue plagued the Diamondbacks last season, and while they were unlikely to play meaningful games into October this year or last, the problem has once again surfaced.  While some may wonder whether defense is playing a role, the Diamondbacks’ defensive efficiency in 2008 and 2009 was .686 and .687. They have fallen to .673 this year with mostly the same cast of defenders, and defense certainly does not explain the 18.7% HR/FB, dwarfing the 12.3% in 2008 and the 13.0% of 2009.  With the sample sizes increasing with each start we decided to place three different Snakes starters under the microscope to explain both how their SIERA either does or doesn’t match up with their ERA, as well as assuage some common misconceptions about estimators.

Dan Haren: ERA 5.35, SIERA 3.08—difference +2.27

YEAR

ERA

SIERA

UBB%

SO%

GB%

LD%

FB%

PU%

HR/FB

BABIP

BABIP w/ men on

OF in air/BIP

2007

3.07

3.59

5.8

20.6

45.4

21.2

26.9

6.6

13.0

.288

.272

45.1

2008

3.33

3.09

4.1

23.0

44.7

21.2

26.1

7.9

11.5

.307

.326

46.1

2009

3.14

2.92

4.0

24.6

44.2

19.8

29.8

6.2

14.1

.271

.286

47.3

2010

5.35

3.08

4.1

23.8

44.8

17.8

30.9

6.5

22.5

.335

.400

49.3

Haren’s ugly ERA for the first two months of 2010 represents exactly why statistics like SIERA are designed.  His repeatable skill-based stats (strikeout, walk, and ground ball rates) have held steady, while his luck-based stats—HR/FB and BABIP (especially with runner on base)—have shot through the roof.  While some pitchers are better than others at keeping batters from hitting fly balls to the outfield in general, there is no distinct group of pitchers that specializes in keeping 300-foot fly balls from going 400 feet.  On top of that, even though Haren has allowed fewer line drives and fewer balls to reach the outfield in the air (both indicators that batters have primarily made weak contact), his BABIP has shot up to .335, and is an even more unlucky .400 with men on base. 

His lack of luck is more determinable because his ugliest run totals came in games in which he performed rather well.  On April 10, Haren recorded nine punchouts and issued two walks against 32 Pirates hitters. He allowed just five outfield fly balls, but two landed on the wrong side of the wall.  Ten days later, he struck out eight Cardinals, while walking just two, but three of his four outfield flies left the park.  On May 11, the Dodgers whiffed 10 times in their 31 plate appearances against the ace, walking just once. Of the other 20 hitters, 10 recorded hits on balls in play, leading to four earned runs and only 6 1/3 innings.

Haren’s only legitimately bad start—meaning he pitched poorly and the box score backed that assertion up—was May 16 against the Braves, when he struck out two hitters and walked three, allowing two home runs out of 10 outfield flies, adding up to six runs in just 4 1/3 innings.  Last Thursday's performance against the Rockies was even more typical of his season, as four of his 15 outfield flies landed on the wrong side of the Coors Field walls.  All in all, the Diamondbacks are just 5-6 in his starts, and should have another two wins if not for some bad luck in his games. 

Edwin Jackson: ERA 6.03, SIERA 3.84—difference of + 2.19

YEAR

ERA

SIERA

UBB%

SO%

GB%

LD%

FB%

PU%

HR/FB

BABIP

BABIP w/ men on

OF in air/BIP

2007

5.76

4.66

11.3

17.0

46.0

23.6

24.1

6.4

14.7

.344

.329

43.4

2008

4.42

5.07

9.6

13.7

39.5

22.1

28.6

9.8

13.3

.304

.282

47.4

2009

3.62

4.22

7.5

18.2

40.8

19.5

29.6

10.1

13.9

.278

.284

47.4

2010

6.03

3.84

8.2

20.4

49.0

19.7

28.4

2.9

16.9

.330

.297

45.4

A quick glance at Jackson would yield the same claim of “bad luck,” where we play a little “sabermetric pepper,” and note the bad BABIP, the bad HR/FB, the solid walk, strikeout, and ground ball numbers, and quickly conclude that they all indicate Jackson is likely to turn it around and be an above average pitcher.  That’s all true. Jackson is likely to rebound, because skills like strikeouts and walks are more likely to repeat than balls in play falling for hits.  That’s why SIERA is an “ERA predictor.” However, nobody is going to argue that Haren is on a different plane than Jackson, and while the former is likely to finish the season with a sub-4.00 ERA as he does every year, the latter’s definition of “rebound” might not be as substantial.

Jackson was chosen for this study because he reeks of confirmation bias and represents the type of pitcher for whom extra research is needed to avoid inaccurate assessments. Jackson has always been one of those loads-of-potential pitchers who had never turned the corner. And then the first half of last year happened. Instead of asking if that was fluky, most assumed he had finally turned the corner. After all, the same people had expected Jackson to perform this way for several years. In reality, it is perfectly valid to think of his first half with the Tigers as the outlier.

Jackson’s BABIP is so high primarily due to two consecutive terrible starts: April 27 and May 2.  In those two games, he only struck out two hitters and walked three, while giving up 18 runs in 6 1/3 innings.  His BABIP in those games was .541 (20 for 37), as compared to .280 otherwise.  From a PITCHf/x standpoint, Jackson doesn’t appear to be doing too much differently. His fastballs and sliders have dropped about 3-5 percent with increases in curveballs and changeups making up the difference. This isn’t anywhere close to the realm of extreme. On top of that, his velocities have remained intact, and the only number not in line with years past is fastball movement.

While his pitches moved 4.9 inches horizontally and 10.3 inches vertically last year, the movement components are currently 4.0 and 9.4. The overall effect of less vertical and horizontal movement could certainly be the difference between a pitch on the outside corner versus one hovering over the plate. This might not be the definitive reason for Jackson’s struggles—after all, he wasn’t that great in 2007 or 2008 with similar movement numbers but little else sticks out from this granular standpoint.  With nothing else sticking out to explain the two incredibly poor starts, and little difference in the movement of his pitches, chances are his SIERA will rise as his ERA falls towards it, giving him a smaller gap to close than it may seem.

Ian Kennedy: 3.38 ERA, 3.98 SIERA—difference of -0.60

YEAR

ERA

SIERA

UBB%

SO%

GB%

LD%

FB%

PU%

HR/FB

BABIP

BABIP w/ men on

OF in air/BIP

2010

3.38

3.98

7.4

20.2

39.6

16.3

36.6

7.4

14.9

.253

.218

53.0

Short of a brief stint early in 2008 and a couple starts in 2007 and 2009, Kennedy had not gotten any real major-league experience before this season.  In 2010, he certainly has had some good luck (.247 BABIP) that has had a larger effect than some of his bad luck (16.4% HR/FB), but Kennedy has still just been a solid pitcher this year, finally living up to some of the hype that surrounded him at USC and in the Yankees organization.  When Haren’s luck turns around and Jackson’s numbers look a little bit less like they were torpedoed by two ugly starts a month ago, Kennedy will fit snugly into the Diamondbacks’ otherwise solid rotation, striking out nearly three times as many hitters as he walks, keeping the damage from his fly ball tendencies to a minimum.

But Kennedy is not assured of keeping his pretty numbers all year long. While it makes sense that he should given his propensity for K/BB glory, the BABIP numbers are very low, well below even what one might expect from aces like Roy Halladay, CC Sabathia or Zack Greinke, who might be the extreme pitchers with the skills to lower their ERA by about 0.15 below their team’s average, while Kennedy fits better in the category of someone perceived as a prospect that didn’t pan out. This is another important concept of estimators: just because one is close to an ERA does not mean the pitcher will sustain both marks all year long. With a pitcher like Haren, it is acceptable to expect a big turnaround due to both his track record and very solid controllable rates this year. With one like Jackson, it’s just as acceptable to expect not as substantial of a turnaround, with the ERA perhaps settling around 4.75 barring something unforeseen.

And with someone like Kennedy, it makes sense to expect nothing big to change, but that should not be treated as a guarantee. For all we know, his numbers could be 4.31 ERA/4.52 SIERA a month from now, with little disconnect between the two but a downturn in performance across both. The Diamondbacks are unlikely to make any headway in the National League West even with these three meeting expectations, but the lack of agreement across the metrics right now does not help their cause in any way.

Eric Seidman is an author of Baseball Prospectus. 
Click here to see Eric's other articles. You can contact Eric by clicking here
Matt Swartz is an author of Baseball Prospectus. 
Click here to see Matt's other articles. You can contact Matt by clicking here

Related Content:  Edwin Jackson,  Era

25 comments have been left for this article. (Click to hide comments)

BP Comment Quick Links

Richard Bergstrom

Um... could the high ERAs just be due to the horrible Diamondbacks bullpen allowing more inherited runners to score than usual? Or, perhaps, Hinch tends to keep his starters in longer than other managers because of that bad bullpen, causing the starters to be less effective in later innings (hence the HR/FB spike) and/or pulling his starters with runners on base more frequently than other managers?

I guess what I'm saying is maybe this article could at least discuss how bullpens can affect a starter's ERA.

Jun 02, 2010 03:35 AM
rating: 6
 
BP staff member Matt Swartz
BP staff

So that's a really small effect, and also somewhat in the wrong direction. Dan Haren has surrendered 5 of his 16 HR in the 1st inning, and 8 in the first three innings. Edwin Jackson has surrednered 5 of his 10 HR in the first three innings, while Kennedy and Rodrigo Lopez each have 5 of 11 in the first three innings.

We have a stat called Fair Run Average (FRA) which takes into account what RA should be adjusting for inherited runners. Haren is at 5.16 RA with 5.22 FRA, Jackson has a 6.03 RA with a 5.96 FRA, Kennedy has a 3.50 RA with a 3.35 FRA, and Lopez has a 5.20 RA with a 5.07 FRA. These type of things are small effects compared to the larger issues cited above relating the BABIP, HR/FB, and difference in performance with men on.

Jun 02, 2010 12:58 PM
 
Richard Bergstrom

Thanks for the clarification. Though it might've been in the wrong direction or a small effect, it was a big question in my mind (and by the +'s, in others minds) that wasn't previously addressed in the article.

Jun 02, 2010 16:03 PM
rating: 0
 
John Carter

Wow, that is a very significant point. Is that generally true among all pitchers - that variances in bullpen support have a very small effect on ERA? Or, just in these particular cases? Or, is there a large variance from pitcher to pitcher, but small from a team of pitchers to a team a pitchers?

Jun 02, 2010 17:30 PM
rating: -1
 
ScottyB

Unexplained variance (error variance) does NOT mean luck.

Jun 02, 2010 08:34 AM
rating: 3
 
PWHjort

Of course, it means something we're not measuring is influencing the data. Though if you buy the notion that we can measure every pitcher skill, most of the variance is just luck/park/defense.

Jun 02, 2010 10:00 AM
rating: 0
 
BP staff member Matt Swartz
BP staff

This is true, but I think most of what the article is doing is actually explaining this fact, rather than calling everything luck. That said, there IS such a thing as luck, and you CAN actually pinpoint it. There is a counterrevolution in sabermetrics claiming that too much is being called "luck", which is a good thing to highlight, but it's being taken too far. We KNOW that a binomial variable like BABIP should have a standard deviation of .019 for a whole season among pitchers who throw 150 IP. That means that when we observe an actual standard deviation, net of team, of .021, we CAN see that MOST of BABIP fluctuation is bad luck. Of course it's important to explain all the variance you can, but dismissive statements saying that you can't call anything luck are as dogmatic as saying that everything you can't explain is luck.

Jun 02, 2010 13:01 PM
 
shanecris

i lost where i can check the siera for all players.. where is that report again?

Jun 02, 2010 10:21 AM
rating: 0
 
bflaff1

@shanecris: You can find SIERA in the statistics section. Just customize something like the 'Pitcher Season - Standard' report. Or follow this link, assuming it works:

http://www.baseballprospectus.com/statistics/sortable/index.php?cid=353618

Jun 02, 2010 11:11 AM
rating: 0
 
Seth Cohen

As someone who lives in Phoenix and goes to a handful of games every year, I have another theory which has to do with the ballpark. I've noticed the roof open, and more importantly, outfield shutters open, more often than I seem to remember. I recall Schilling used to demand both be closed when he pitched here. I think when those shutters are open it turns into a wind tunnel in the hitters favor. Perhaps home-road splits in HR/FB could tell us. Just a thought.

Jun 02, 2010 12:08 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

Certainly interesting, but not really enough to explain things. We do see a spike in HR/FB for the D'backs offensively from 10.4 to 11.9%. However, the spike defensively is 13.0 to 18.7%, which is just way larger. If it were only year-to-year changes in park effects, that jump could not be as large. Good point, though.

Jun 02, 2010 13:03 PM
 
Richard Bergstrom

Wind can affect line drives and pop-ups in play too which make it harder for defenders to get to... and maybe since there are more line drives and pop-ups in play than home runs, the effect is more noticeable.. I'm not saying it describes the entire, or even the majority of the difference in defensive performance, but it might be a factor. We're basically talking the same cast of characters on defense as last year and I know defense can fluctuate on a yearly basis, but a team-wide drop?

Jun 02, 2010 16:02 PM
rating: 0
 
evo34

Are full SIERA numbers available on the web site somewhere?

Jun 02, 2010 13:52 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

They are in the four reports-- the Pitcher Standard Report, the Pitcher Rates Report, Pitcher Team-Standard Report, and the Pitcher Team-Rates Report. You can also get SIERAs if you do a customized stats report. I have a couple links in my favorites because I like certain things all together in one report that I can make a .csv out of.

Jun 02, 2010 13:56 PM
 
evo34

Thanks. Just to refresh, SIERA does not try to neutralize team defense, correct? So SIERA attempts to predict *actual* ERA going fwd. (vs. trying to assess team-neutral pitcher quality like FIP does)?

Jun 02, 2010 14:23 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

Well, it tries to predict ERA going forward as much as FIP does, only it incorporates a few other factors to make it less biased against certain pitchers. It tells you what ERA typically is for pitchers with certain GB, BB, and K rates, taking into account the interactions, etc. It also incorporates BABIP skill that pitchers do have. So FIP would assume that pitchers have the same distribution of BIP skill, while SIERA assumes that pitchers with high K% and low GB% will have lower BABIPs.

Jun 02, 2010 14:42 PM
 
evo34

Bottom line: does SIERA project what a pitcher's ERA is likely to be with his *current* team, or what his ERA would be if he played on a team with league-average defense?

Jun 02, 2010 15:46 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

Ah-- yes, league average defense.

Jun 02, 2010 15:49 PM
 
evo34

Thanks for the responsiveness. Any advice for a simple way (for the user) to adjust SIERA to make it a pure predictor of actual future ERA? Obviously, I am coming at this from a fantasy perspective, where you seek the best possible intra-season indicator of actual future ERA on the same team -- not what the ERA would look like if all the pitcher's teammates were replaced by average defenders. Fielding-independent metrics are certainly useful for trying to isolate pitcher skill between seasons, but for pure numerical projections of same-season ERA, a fielding-dependent (but otherwise luck-adjusted) metric is needed. Is there any simple way for someone to tweak SIERA to reflect the pitcher's team defense impact? Obviously, I would prefer to see this offered on the web site, but if not, would be open to suggestions as to how I could do the calculation myself.

Jun 02, 2010 16:37 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

I would look at SIERA over the past three years, obviously weighting right now a lot more to get a defense-neutral version of ERA and then just adjust by time. Look at how the teams FIP compares to its ERA, and figure that the difference between SIERA and ERA should be similar to the difference of the team FIP and the team ERA (though maybe shrink this difference a little bit). The fewer innings you are looking at, the more you want to regress the whole ERA back to the mean, though. For fantasy baseball, I would imagine it is probably more important to rank pitchers ordinally rather than giving a cardinal measure of ERA though? In that case, SIERAs with a FIP/ERA adjustment might do the trick pretty well. Comparing SIERA and PECOTA can probably do a good job of giving you extra information too.

Jun 02, 2010 17:27 PM
 
evo34

A few requests: can you make the BP stat pages (e.g., http://www.baseballprospectus.com/statistics/sortable/index.php?cid=244079) sortable by player *last* name, and able to show all players on a single HTML page, if desired?

More importantly, can you make available team SIERAs (preferably with the option to see team starter SIERA and team reliever SIERA)?

Thanks.

Jun 02, 2010 14:34 PM
rating: 0
 
surfdent48

Nice article. Can you give a link that lists the following for pitchers: (1) how many quality starts a starter has and how many turn into non deserving losses---almost always due to poor run support (Oswalt, Grienke, Cain, Volstad), (2) how many games a starter is saved from a loss when he leaves a game behind but the offence scores to take him off the hook for the loss, (3) how many non quality starts result in nondeserving wins (4) How many inherited runners are left and how many of these score vs. dont score. Thanks

Jun 02, 2010 14:50 PM
rating: 0
 
DavidJ

All of this information is available on the player's baseball-reference.com page. Oswalt, for example:

http://www.baseball-reference.com/players/o/oswalro01-pitch.shtml#pitching_starter

Jun 02, 2010 18:53 PM
rating: 0
 
Simon C.

What about an underlying factor that wasn't mentioned - the downgrade at 2B from Orlando Hudson to Kelly Johnson? That would make for an interesting study...

Jun 02, 2010 21:43 PM
rating: 0
 
BP staff member Eric Seidman
BP staff

Simon... it wasn't mentioned primarily because Hudson wasn't there LAST year either. Ryan Roberts was their everyday 2B, and there is no way the difference between Roberts and Johnson is causing anyone's ERA to fluctuate wildly.

Jun 03, 2010 08:17 AM
 
You must be a Premium subscriber to post a comment.
Not a subscriber? Sign up today!
<< Previous Article
Premium Article Future Shock: With Bor... (06/02)
<< Previous Column
Premium Article Checking the Numbers: ... (05/26)
Next Column >>
Premium Article Checking the Numbers: ... (06/04)
Next Article >>
Prospectus Q&A: Chris ... (06/03)

RECENTLY AT BASEBALL PROSPECTUS
Fantasy Article Player Profile: David Robertson
Fantasy Article The -Only League Landscape: National League ...
Fantasy Article Tale of the Tape, Dynasty Edition: Nick Burd...
Fantasy Article Dynasty League Positional Rankings: Top 50 R...
Premium Article Rumor Roundup: Huston Street Might Have Reli...
Pitching Backward: In Search of the Winninge...
Rubbing Mud: A Solution Does Not Exist

MORE FROM JUNE 2, 2010
Premium Article Future Shock: With Boras, Good Often Outweig...
Premium Article Transaction Action: Double Dealing
Premium Article Prospectus Hit and Run: Seeing Red
On the Beat: Wednesday Update
Premium Article Prospectus Q&A: Billy Butler
Premium Article Campus Notes: NCAA Regionals Preview, Part 2

MORE BY ERIC SEIDMAN
2010-06-17 - Premium Article Ahead in the Count: Is Jimenez In Over His H...
2010-06-09 - Premium Article Checking the Numbers: The Baby-faced Assassi...
2010-06-04 - Premium Article Checking the Numbers: A Developmental Dilemm...
2010-06-02 - Premium Article Checking the Numbers: Sneaky SIERA
2010-05-26 - Premium Article Checking the Numbers: Royal Pains
2010-05-24 - Premium Article Checking the Numbers: The Trade-Me Clause
2010-05-19 - Premium Article Checking the Numbers: Bad Ollie or Worse Oll...
More...

MORE CHECKING THE NUMBERS
2010-06-16 - Checking the Numbers: Where in the World is ...
2010-06-09 - Premium Article Checking the Numbers: The Baby-faced Assassi...
2010-06-04 - Premium Article Checking the Numbers: A Developmental Dilemm...
2010-06-02 - Premium Article Checking the Numbers: Sneaky SIERA
2010-05-26 - Premium Article Checking the Numbers: Royal Pains
2010-05-24 - Premium Article Checking the Numbers: The Trade-Me Clause
2010-05-19 - Premium Article Checking the Numbers: Bad Ollie or Worse Oll...
More...

INCOMING ARTICLE LINKS
2010-09-10 - Premium Article Ahead in the Count: The Biggest ERA-SIERA Di...
2010-07-30 - Premium Article Ahead in the Count: The Poor Return on Dan H...
2010-06-04 - Fantasy Article Fantasy Beat: Is Dan Haren A Cause For Conce...