CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

<< Previous Article
Premium Article Divide and Conquer, AL... (06/03)
<< Previous Column
Baseball ProGUESTus: C... (05/27)
Next Column >>
Baseball ProGUESTus: I... (06/10)
Next Article >>
Premium Article Collateral Damage: Sci... (06/03)

June 3, 2011

Baseball ProGUESTus

Can Baseball Expertise Be a Bad Thing?

by Sam Miller

Believe it or not, most of our writers didn't enter the world sporting an @baseballprospectus.com address; with a few exceptions, they started out somewhere else. In an effort to up your reading pleasure while tipping our caps to some of the most illuminating work being done elsewhere on the internet, we'll be yielding the stage once a week to the best and brightest baseball writers, researchers and thinkers from outside of the BP umbrella. If you'd like to nominate a guest contributor (including yourself), please drop us a line.

Sam Miller writes about baseball for the Orange County Register. He covered local government, education and autism before moving to the sports section in 2009. He lives in Long Beach with his wife and newborn daughter, and you can follow him on Twitter at @sammillerocr

1. Before the 1993 season began, Mike Scioscia was talking to a Los Angeles Times reporter, reminiscing about his 11 seasons with the Dodgers. Suddenly, he stopped.

"I hope you are not writing this article like I'm dying," he said. "My career isn't over.”

The Dodgers disagreed. They had just let Scioscia leave as a 33-year-old free agent. He had already lost playing time the previous season to rookie Mike Piazza, who was called up in September and started 16 of the team’s final 31 games. Scioscia’s final at-bat as a Dodger came as a pinch-hitter with his team trailing 6-1.

"Other guys who, in the near past, have had bad years or been injured have always been given the opportunity by the Dodgers to come back and have a good year," Scioscia said. "I feel like last year is the first bad year I've had in a long time.”

By one measure—not a measure anybody cared about at the time, maybe a spurious measure, but a measure—it wasn’t even a bad year. The Dodgers’ pitchers had a 3.16 ERA when he caught, and a 3.78 ERA when Piazza or Carlos Hernandez caught. That was consistent with what the Dodgers got almost every year. From 1984 through 1992, Scioscia’s final nine years, the Dodgers allowed 3.71 runs per nine innings when he was catching, and 4.00 runs per nine innings when somebody else was catching. (On a pitcher-by-pitcher basis, he was generally better, too.)

"I don't think it was a coincidence that the staff was number one or number two in earned-run-average when Mike Scioscia was here,” Orel Hershiser said at the time. “He's the one behind the plate calling the pitch selection. You can't have 10 pitchers and say they are all smart. Human nature says that all 10 aren't smart."

Imagine a world where Catcher ERA caught on. In that world, Mike Scioscia could have claimed credit for somewhere between 200 and 300 runs saved in his career. In that world, Scioscia would have some people debating his Hall of Fame case.

In that world, the Dodgers might not have replaced him with a slugging catcher who couldn’t catch, and in that world Jeff Mathis would make sense.

2. Shortly after I began writing about the Angels for the Orange County Register—and before I’d ever met Scioscia—I wrote a post about the reported “competition” in Spring Training for the starting catching job. This was in 2009, the year after Napoli’s .960 OPS.

“A bit of springtime competition could be good for both of them,” I wrote, “but there’s no way Mike Napoli and Jeff Mathis are really fighting over that catcher’s job. This is just for show, folks.”

I was dead wrong. Mathis started Opening Day, and the next day, and 76 other times that season. In the four years they shared the catching job, Napoli started 282 times, Mathis 231. If Mathis were to double in each of his next 100 at bats, his career slash line—.269/.314/.455—would still come up well short of Napoli’s .251/.346/.485 line as an Angel.

Mike Scioscia isn’t a stathead kind of guy. He’s not as anti-stat as you might imagine—he’s extremely wary of sample-size issues, he doesn’t believe in distinguishing between earned and unearned runs, and he thinks pitcher wins are a bad way of measuring starters—but he is extremely skeptical of advanced defensive stats. He says they don’t account for the role of advanced scouting, positioning and, yes, even the catcher’s role in calling pitches that reflect the scouting and positioning. There’s only one defensive statistic that he thinks can accurately reflect the player’s role: Catcher Runs Allowed. “An absolute tool as to how a catcher relates to a pitcher’s performance,” he called it.

“Let me put it to you this way,” he once said. “If you string out 162 games and you have one catcher who is giving up one run a game less when he catches, on the net runs end of it, he’s 162 runs ahead, right? So the other catcher has to produce 162 runs more than the other guy just to break even. I think a catcher is going to influence a game and a season behind the plate more than he is with his four at-bats a night.”

Perfect: the one advanced metric he approves of is one that had essentially been considered disproven for a decade. And the one advanced metric he approves of is the one that conveniently makes him look like he was a superstar.

3. All of this goes to the very nature of expertise. Mike Scioscia is an expert at catching. He knows more than I know about catching, and more about his own catchers, and more about his own decision-making process. Do you know how absurd it feels when I’m standing in his office arguing with him about Jeff Mathis’ catching skills?

But at the core of “expertise” is a dangerous belief in oneself. The more of an expert you are, experts (ironically!) tell us, the less you consider the horrifying fallibility of your own brain. Michael Lewis noted this in Moneyball: “There was the tendency of everyone who played the game to overgeneralize wildly from his own experience.” The failure of expertise is a theme of nearly every pop-science best-seller from the past decade—Malcolm Gladwell and The Wisdom of Crowds and Jonah Lehrer. (Wine experts can’t even tell the difference between red wine and white, when food coloring is added!)

“And the more knowledge that expert has, the worse, interestingly, it becomes, because the expert is using the knowledge very selectively to justify increasingly extreme predictions,” said Philip Tetlock, the author of Expert Political Judgement.

Would Jeff Mathis still be in the majors if Mike Scioscia had been a third baseman? Would the Angels be better off?

4. Each year has made it harder to say he’s definitely wrong. Maybe he’s been right all along.

The first time I looked at the two catchers’ “performance,” I found that Mathis outperformed Napoli in 2008 by runs allowed, but that the difference was almost entirely related to batting average allowed on balls in play. Try to come up with a flukier-sounding stat than Catcher BABIP. I dismissed it.

Then in 2009, five Angels starters pitched at least 100 innings. Mathis’ CERA was at least a half-run better for each of the five, his K:BB rates were better with four of the pitchers (and essentially tied with the fifth), and his BABIPs were again significantly better for four of the pitchers (and essentially tied with the fifth).

In 2010, five Angels starters pitched at least 100 innings. Mathis’ CERA was significantly better for four of them, and he again trounced Napoli in K:BB and BABIP.

There were differences in the way they called the game: Mathis appears more likely than Napoli to call off-speed pitches, especially in traditional fastball counts. And Mathis “earns” an extra strike call every two games or so. If switching a ball to a strike is worth .161 runs, as Dan Turkenkopf found, Mathis has an edge of about 10 runs over a full season, by pitch-framing alone.

Meanwhile, the CERA numbers Scioscia was citing kept piling up. And meanwhile, I kept mocking his decision to play Mathis. So now which of us was using knowledge selectively to justify increasingly extreme predictions?

5. There’s so much noise, though.

I wondered whether there were other variables that correlated as strongly with the Angels’ run prevention during the Mathis/Napoli years. I tried one variable and found nothing. I tried a different variable, and it hit:

2006:

With X: Angels allow 4.06 runs per games
Without X: Angels allow 4.64 runs per game

2007:

With X: Angels allow 3.76 runs per game
Without X: Angels allow 4.65 runs per game

2008:

With X: Angels allow 3.25 runs per game
Without X: Angels allow 4.60 runs per game

2009:

With X: Angels allow 3.82 runs per game
Without X: Angels allow 4.97 runs per game

2010:

With X: Angels allow 4.35 runs per game
Without X: Angels allow 4.32 runs per game

As clear as can be: X correlates to improved performance by Angels pitchers and defenders, even more (by a little) than Mathis starts vs. Napoli starts. Variable X, incidentally, is “Phillies Scored Two Or Fewer Runs The Same Day.” I’m pretty sure there’s no causal link.

6. A Brief History Of How The Prevailing Literature Rejects or Supports Mike Scioscia’s Perceived Genius/Idiocy:

1989: Craig Wright, Inner Circle Sabermetric Hall of Famer, writes in The Diamond Appraised: “I fell in love with (CERA) the minute I read it.” He uses it to evaluate catchers.
Scioscia: Smart, like Craig Wright.

1999: Keith Woolner writes for BP, “Catchers do not have significant differences among their game-calling abilities.” Wonders whether this means Frank Thomas should catch.
Scioscia: Very not smart, unlike Keith Woolner.

2002: Bill James writes that there may be significant run-suppressing ability for a catcher, but that it’s likely undetectable.
Scioscia: Probably not smart, unlike Bill James, but in a post-modern, undetectable-truth kind of way.

Later 2002: Woolner says any run-suppressing ability would be undetectable if it existed, but might not exist anyway.
Scioscia: Very likely not smart, unlike Keith Woolner.

2003: Chris Dial takes a DIPS approach to Woolner’s 1998 research and agrees with the conclusion.
Scioscia: Not smart, unlike Chris Dial.

2008: PITCHf/x comes along. Dan Turkenkopf suggests that framing ability alone might be worth 25 wins per year, concedes this sounds crazy, but “maybe we’ve been wrong the whole time.”
Scioscia: Super smart, like Dan Turkenkopf but without self-doubt.

2010: Bill Letson suggests a 13-win range between the best and worst catchers based on pitch-framing.
Scioscia: Smart, like Bill Letson.

Later in 2010: Sean Smith writes that “catchers have a significant impact on the performance of the pitchers they catch.” Not significant enough to justify Mathis starting over Napoli.
Scioscia: A little bit smart, like Sean Smith. Also a little bit unsmart, unlike Sean Smith.

2011: Mike Fast puts the range at around +/- 20 runs per year based on pitch-framing alone.
Scioscia: Somewhat smart, somewhat like Mike Fast.

Because this is Baseball Prospectus, here is a gratuitous chart:

Average smartness: 5.5.

7. Scioscia doesn’t actually use Catcher Runs Allowed the way you probably think he does. He’s not interested in each catcher’s overall numbers, for instance, but specifically how well each does with each pitcher. He says he doesn’t use even those numbers to make his decisions about playing time, but as a starting point to look for what is causing the differences. He won’t talk about what those causes are, and some of them are immeasurable. For instance, the Angels worked with Napoli on setting a target. That seems like a small detail, but the Angels don’t feel that way—they think it has a real effect on pitch execution.

Scioscia originally learned about Catcher Runs Allowed when he was a coach in the Dodgers’ minor-league system, where it was used not just to assess but to motivate. The Dodgers believed that if young catchers knew they would be evaluated based on this statistic, they would be more focused on figuring out how to stop runs from scoring by any means possible.

“Everyone has a philosophy they view the game through. Our philosophy is that the catcher-pitcher relationship is in play for every pitch of the game,” he says, and none of that sounds unreasonable.

I tell him that I actually trust his expertise, and if he told me that he was starting Mathis based on his eyes and experience I might even believe it could be the right call. It’s his use of the particular stat that bothers me.

“When you see it with your eyes and go back and check it,” he says, “I don’t know if there’s one case where it’s been wrong.”

8. We don’t live in that imaginary world. Catcher ERA didn’t catch on, not really, and nobody talks about Mike Scioscia’s Hall of Fame case. But if he wins another World Series, maybe two, then we will someday debate his case as a manager. This one decision—Mathis vs. Napoli—might be the defining detail people remember about him.

I don’t know where we’ll be when that happens. Right now, the literature suggests he is more likely wrong than right, but also more likely a supergenius than a total idiot. There will likely be some day in the future when we all think Mike Scioscia was right. Then there will be another day in the future when we all think Mike Scioscia was wrong, and so on.

Sam Miller is an author of Baseball Prospectus. 
Click here to see Sam's other articles. You can contact Sam by clicking here

24 comments have been left for this article. (Click to hide comments)

BP Comment Quick Links

Agent007

Is catcher evaluation statistically impossible to achieve? How do you combine all the things that catchers do -- framing the ball, calling pitches, blocking balls in the dirt, throwing out players attempting to steal, intimidating runners into not even trying to steal, blocking the plate, calming and refocusing pitchers, etc. -- into a defensive metric? Just because a catcher calls a pitch does not guarantee the pitcher will throw it well or to where he places the glove. And the pitcher's ability to hold a runner affects a catcher's caught stealing stats. He's the most important defensive player (pitchers change while the catcher remains) yet may be impossible to measure. Oh, and that's before calculating offense...

Jun 03, 2011 06:31 AM
rating: 3
 
dcarroll

I enjoyed this article--well written and amusing, particularly items 5 and 6.

Jun 03, 2011 08:02 AM
rating: 6
 
ScottyB

1. Great article! (me= smart like Sam Miller)

2. Something can be true but so hard to measure because of noise, but even a VERY small effect in this case would be meaningful

3. Scoscia was probably an outlier in his playing days- he was one of the rare catchers that made a difference. Napoli is likely an outlier the other way. Maybe there are no differences in the general population, but there is at the extremes.

Jun 03, 2011 08:15 AM
rating: 3
 
Michael
(736)

Thanks for the fun article.

There also has been some interesting research on catcher defense using a "With or Without You" approach. Check out the Hardball Times Annuals over the last 3-4 years and the Fielding Bible II.

Jun 03, 2011 08:36 AM
rating: 0
 
Ian Miller

Great work, Sam. One of the best things I've read on BP in a while, and that's really saying something!

Jun 03, 2011 08:37 AM
rating: 2
 
devine

"Because this is Baseball Prospectus, here is a gratuitous chart" -- Thank you for the best laugh I've had in a week.

Jun 03, 2011 08:39 AM
rating: 10
 
scareduck

Re section 5: what is the Pearson's r for "Phillies score 2 runs or less that day" vs. Mathis CERA?

Jun 03, 2011 08:40 AM
rating: 1
 
Richie

Bill James called Napoli an absolutely horrible catcher, though without offering what he based that on. So maybe ScottyB: smart, like Bill James.

Jun 03, 2011 09:35 AM
rating: 1
 
ScottyB

I would love to be smart like Bill James!!!!!!

Jun 03, 2011 18:47 PM
rating: 1
 
dianagram

"Mike Scioscia's Tragic Smartness" is so full of win!

Jun 03, 2011 11:33 AM
rating: 1
 
JasonC23
(97)

Very much enjoyed this article. Thanks, Sam!

Jun 03, 2011 11:51 AM
rating: 0
 
eighteen

Managers love players who remind them of themselves. It's self-validating. Scioscia can't choose an offense-first catcher without admitting his own skills as a player weren't as valuable as Napoli's.

It's the same reason Ron Gardenhire loved Nick Punto so much.

Jun 03, 2011 12:31 PM
rating: 7
 
Ian Miller

I've long been a subscriber to this theory. It seems especially prevalent in former-catcher managers evaluating their own catchers. And Scioscia is repeating all the mistakes of Mathis/Napoli with Mathis/Conger.

Jun 03, 2011 15:58 PM
rating: 1
 
Scott Gilroy

Really enjoyed the article and how you approached it. Still have a ways to go before we can assimilate all that a good defensive catcher brings, and what level of offense is acceptable from a lesser Offensive C to offset (bring in close to a wash)
Interesting food for thought

Jun 04, 2011 11:41 AM
rating: 0
 
Matt

This is awesome.

Jun 03, 2011 12:54 PM
rating: 0
 
Matt Kory

You know you've done a good job when the low numbers come out on your side. Great work, Sam. Very much enjoyed reading that.

Jun 03, 2011 14:46 PM
rating: 0
 
fgreenagel2

Great work. Thanks.

Jun 03, 2011 15:08 PM
rating: 0
 
WaldoInSC

That an unrelated variable correlates with lower pitcher ERA doesn't invalidate the correlation between Mathis and lower ERAs.

Southpaws have shorter life expectancies than righties. That doesn't invalidate the the causal relationship between tobacco use and life expectancy.

I think the causal relationship between Mathis and consistently lower ERAs would have to be classified as unproven but intriguing and possibly significant.

Jun 03, 2011 17:39 PM
rating: 4
 
DAra

Great article. I laughed out loud after reading #5.

Jun 03, 2011 18:36 PM
rating: 1
 
ncassino

One of the best articles on this site in a while (taking nothing away from the other writers).

Jun 03, 2011 21:00 PM
rating: 1
 
anderson721

Just wondering what the Angels' winning pct. was for games started by Napoli/Mathis during those 4 years. 7 whether any difference is significant.

Jun 04, 2011 03:55 AM
rating: 0
 
Jack Sprat

With PitchFX data being available, isn't catcher framing actually quantifiable?

Jun 04, 2011 14:40 PM
rating: 1
 
BP staff member Dan Turkenkopf
BP staff

Yes, as Sam mentions in the article, but figuring out the right adjustments has been difficult.

I first looked at this in 2008 and found a range of 13 wins per season (thanks Charlie for pointing out a mistake in my math).

Bill Letson followed up a few years later with a much more rigorous methodology, and pretty much the same results.

Mike Fast had a major breakthrough this year when he adjusted for a pitcher's tendency to throw near the edges of the zone and brought the range down to about 2 wins.

Jun 06, 2011 06:11 AM
 
nonspin

This was great.

Jun 05, 2011 13:27 PM
rating: 0
 
You must be a Premium subscriber to post a comment.
Not a subscriber? Sign up today!
<< Previous Article
Premium Article Divide and Conquer, AL... (06/03)
<< Previous Column
Baseball ProGUESTus: C... (05/27)
Next Column >>
Baseball ProGUESTus: I... (06/10)
Next Article >>
Premium Article Collateral Damage: Sci... (06/03)

RECENTLY AT BASEBALL PROSPECTUS
Premium Article Painting the Black: Pedro Alvarez's Trip Fro...
Premium Article Minor League Update: Games of Thursday, Apri...
Premium Article What You Need to Know: April 17, 2015
Going Yard: How Bryant Can Crush
The Call-Up: Kris Bryant
Premium Article Rubbing Mud: So You Want to Trade Your Draft...
BP Boston

MORE FROM JUNE 3, 2011
Prospectus Perspective: Sabean Takes on the ...
Premium Article Collateral Damage: Sciatica State
Premium Article Divide and Conquer, AL West: The Year of the...
Premium Article On the Beat: Back on the Attack
Fantasy Article Fantasy Beat: Weekly Planner #10
Fantasy Article Fantasy Beat: Value Picks in the Rotation
Fantasy Article Fantasy Beat: The Holy Trinity

MORE BY SAM MILLER
2011-10-03 - Premium Article Pebble Hunting: Three Major-League Teams Int...
2011-08-02 - Pebble Hunting: The Error Face
2011-07-19 - Pebble Hunting: Footloose and Fastball-Free
2011-06-03 - Baseball ProGUESTus: Can Baseball Expertise ...
More...

MORE BASEBALL PROGUESTUS
2011-06-24 - Baseball ProGUESTus: Anyway, Anyhow, Anywher...
2011-06-17 - Baseball ProGUESTus: The Man Who Knew Too Mu...
2011-06-10 - Baseball ProGUESTus: Interviews with an Inde...
2011-06-03 - Baseball ProGUESTus: Can Baseball Expertise ...
2011-05-27 - Baseball ProGUESTus: Curing the Frank McCour...
2011-05-26 - Baseball ProGUESTus: Answers from a Sabermet...
2011-05-24 - Baseball ProGUESTus: Answers from a Sabermet...
More...

INCOMING ARTICLE LINKS
2012-08-15 - Transaction Analysis: Mathis for Smart Peopl...
2012-08-15 - BP Daily Podcast: Effectively Wild Episode 2...
2012-01-24 - Overthinking It: The Player Popularity Test
2011-12-16 - Baseball ProGUESTus: The Men Behind the Men ...