Former Astros third baseman Morgan Ensberg wrote his take on baseball stats recently. It’s actually a pretty interesting read. I don’t want to dwell on it, but there is one little comment in particular I’d like to talk about:
"The term 'Garbage in, Garbage out' is the most accurate description I can give. If the sample used is garbage, then the answers won’t be accurate. Sabermetrics requires accurate information or organizations may misinterpret the data."
This is, of course, true as far as it goes. But how far does it go?
So let’s talk about baseball stats. I don’t mean metrics, like True Average or OPS or Runs Created. I mean the raw totals that are used to figure those metrics, the stuff that shows up in the box score. What do—and don’t—those numbers tell us about the game of baseball?
A matter of fact
Before we go any further, let’s go ahead and clarify what is meant when something is called a fact: Too often, the word "fact" is used to denote the truthfulness of something. But the word really means that something can be objectively proven to be true, not that it has been. To illustrate, consider these three statements:
- Ridley Scott was born November 30, 1937.
- He directed the film Aliens.
- His best film was Blade Runner.
The first is a statement of fact that happens to be true. The second is a statement of fact that happens to be false; he directed Alien, while James Cameron directed Aliens. The third is an opinion (mine, if you’re curious about attribution).
So not all of our facts have to be true—they just have to be provable given enough information. And given that constraint, we can measure how truthful a particular set of facts is.
What You Think You Know
So, given that, what facts do we know about a particular hitter, pitcher, fielder, game or team? Let’s restrict ourselves to considering those things under the purview of the official scorer.
Nobody would, I think, argue with the factualness of the "three true outcomes"—the walk, the strikeout, and the home run. With the benefit of instant replay and ball-0tracking technologies like QuesTec and PITCHf/x, we’re becoming very ready (and somewhat able) to second-guess the judgment of the umpire on those calls, but there is an objective standard to refer to in those sorts of disputes.
Now let me propose a standard for "baseball facts." A baseball fact is, simply put, something where the decision has a direct outcome on the game. Changing a strikeout into a walk has a very large effect, for instance—it provides both a baserunner for the offense and prolongs the inning.
Now let’s consider what happens when the ball is put in play. Is a hit a fact?
The answer is "sort of."
When a ball is put into play, there is, of course, a baseball fact that we can easily observe—whether or not the batter is put out or reaches base safely. (We can introduce a third distinction that is still a baseball fact—when a batter reaches safely but another runner is put out on base.) And typically the recording of a hit observes this distinction—but not always.
The guilty party here (and in so much else) is, of course, the error. Let’s consider the guidance provided to the official rule for determining an error:
It is not necessary that the fielder touch the ball to be charged with an error. If a ground ball goes through a fielder’s legs or a fly ball falls untouched and, in the scorer’s judgment, the fielder could have handled the ball with ordinary effort, the official scorer shall charge such fielder with an error. … The official scorer shall charge an outfielder with an error if such outfielder allows a fly ball to drop to the ground if, in the official scorer’s judgment, an outfielder at that position making ordinary effort would have caught such fly ball.
I want to draw special attention to the phrase "with ordinary effort." When the official scorer records a play as an error, he is really making two observations at once:
- What actually occurred on the play, and
- What the official scorer believes should have happened on the play
The first is clearly a determination of baseball fact; the second is clearly not. The official scorer’s judgement of what should have happened on the play has no bearing on the outcome of the game itself.
(It’s the same with the distinction from a wild pitch to a passed ball—you combine a fact—that a pitch got past the catcher—with an opinion over whether or not the pitch should or would have gotten past the average catcher.)
And, of course, the distinction between a hit and an error is pervasive throughout many of the stats recorded—it colors what is and isn’t an earned run, a run batted in, a putout or assist, and so on. So much of what we think we know about baseball is reliant upon the determination of an error—which isn’t a fact at all. Admittedly, it’s a subtle effect most of the time—if we treat a hit as a fact instead of as a judgment, we end up OK more often than not.
The trouble comes at the edges—with a hitter like Derek Jeter, for instance, who is better than average at reaching on an error. Jeter gets no credit for this—even though it helps him reach base more often and provide more runs, and therefore wins, to his team—because of the determination that what occurred was simply the fault of the fielder not making an ordinary effort.
Credit and Blame
And that’s what it comes down to—the decision to use the boxscore to record who is at fault, rather than objectively record what actually occurred. A pitcher is considered not at "fault" for runs that score after an error with at least two outs, even though he continues to pitch after that error and has an influence on the number of runs that score after that point.
(And pitchers certainly vary in their ability to prevent "unearned" runs. Fly ball pitchers typically allow fewer unearned runs, because they allow fewer ground balls in the first place. And good pitchers typically allow fewer runs of all sorts than bad pitchers, because they are better at stranding all baserunners, including the ones who reach on an error.)
It’s important at this point for us to recognize that the most influential man in baseball statistics isn’t Bill James–it’s Henry Chadwick, the inventor of the modern box score and the writer of the first rule book for the game. His contributions have deservedly gotten him enshrined in the Hall of Fame.
But Chadwick’s personal opinions continue to shape how we see baseball, over 150 years after he wrote that rulebook. Consider this entry from Chadwick’s book The Game of Baseball:
The only correct estimate of a batsman's skill is that made from the record of the number of times bases are made by "clean" hits, that is, by the ball being hit out of the reach of fielders, and not by hits which only yield bases through errors of fielding. There is quite a difference between an estimate of batting made on the number of bases made on hits, and that made on the number of times bases are so made. Suppose the first striker hits a long ball and makes a clean home run, he is credited with having made four bases on hits and with having made his base on his hit once. The next striker makes his first base by his clean hit, and the next three strikers do so. Likewise, and if every man following was to do the same, nine runs would be made without the loss of a hand out, unless from careless base running, for any clean hit on which the first base is secured prevents any player running the bases from being put out through having to vacate the base he occupies. Now, although the first striker made four bases on his hit, he only secured one run, whereas the players who made but one base on their hits necessarily each secured a run.
Chadwick spends a lot of effort in his writings expounding upon the belief that one must distinguish between a "clean hit" as a result of batter’s skill, as opposed to a reach on error, which he attributes solely to fielder skill.
(This mindset, along with a strong background in cricket, led him to consider a walk not as a credit to the batter but simply a "battery error" for the pitcher and catcher. He was, in spite of his great influence as the editor of the Spalding Guides, unable to convince others to observe this convention, and the walk has survived—although it took the later work of people like James to encourage a full appreciation of a walk as an attribute of batter skill.)
It’s actually incredible how much Chadwick was able to fit into that paragraph that contradicts the "sabermetric" school of thought in how to evaluate players. The walk is conspicuous in its absence. He explicitly claims that the single is as valuable as a home run as an indicator of batter skill—in spite of the fact that a home run has greater value, requires greater skill and is a more consistent indicator of skill.
The Moralist View
This point of view—call it the moralist view of baseball—turns on the distinction between a "clean hit" and an error—a distinction that simply does not exist in fact. It requires us to treat the subjective judgment of the official scorer—and really it’s many official scorers, across ballparks and across history—as fact.
Unlike the judgment of umpires, these judgments of official scorers are relatively unassailable—it’s not so much that they’re always right as that there’s no way to say conclusively that any of them are wrong, even when they come to different judgements on what would substantially be the same play.
Let’s consider, for a moment, Pete Rose’s quest for the all-time hits record. Rose, of course, has more career hits than anyone. He also rarely reached on error during his career.
There are two interesting components here. One is looking at Rose at home and on the road—at home infielders committed an error when fielding a ball put in play by Rose 2.5 percent of the time. On the road, meanwhile, the rate was 3.5 percent. (The difference, if you’re wondering, is statistically significant.)
And Rose’s reach on error rate also took a substantial dive late in his career—before 1975 he was slightly above average, while after that he was substantially below average.
Can we say that Rose was being favored by the official scorer? It seems likely, doesn’t it? Of course we don’t know for sure. And of course we can’t say for sure—because, again, the error isn’t a fact. There is absolutely no effect on the game in giving Rose a hit instead of an error, so we can’t look to the record and see how this judgment may have impacted what occurred on the field.
And the historical record is littered with these sorts of little quirks—some of which we know about, many of which we probably don’t. It’s the ones we don’t know about that worry me. Because every time we treat those subjective judgments as facts—without question or reflection on what those mean, what different scorers did with the same set of facts—we end up with a flawed understanding of what a player really did on the field.