CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

<< Previous Article
Premium Article The BP Broadside: The ... (05/05)
<< Previous Column
Baseball ProGUESTus: F... (04/29)
Next Column >>
Baseball ProGUESTus: M... (05/13)
Next Article >>
Premium Article Overthinking It: Year ... (05/05)

May 5, 2011

Baseball ProGUESTus

A Statistician Rereads Bill James

by Andrew Gelman

Believe it or not, most of our writers didn't enter the world sporting an @baseballprospectus.com address; with a few exceptions, they started out somewhere else. In an effort to up your reading pleasure while tipping our caps to some of the most illuminating work being done elsewhere on the internet, we'll be yielding the stage once a week to the best and brightest baseball writers, researchers and thinkers from outside of the BP umbrella. If you'd like to nominate a guest contributor (including yourself), please drop us a line.

Andrew Gelman is a professor of statistics and political science at Columbia University. He occasionally blogs on baseball, including here, here, here, and here.

I read my first Bill James book in 1984, took my first statistics class in 1985, and began graduate study in statistics the next year. Besides giving me the opportunity to study with the best applied statistician of the late 20th century (Don Rubin) and the best theoretical statistician of the early 21st (Xiao-Li Meng), going to graduate school at Harvard in 1986 gave me the opportunity to sit in a basement room one evening that October with about 20 other students, screaming at the TV, "Put Stapleton in!" Unfortunately, John McNamara didn't hear us, and the rest was history.

I'm much less of a sports fan than I used to be, but the lessons I've learned from reading the Baseball Abstracts have done much to form me as a statistician. James doesn't write much about statistical methods in any general sense—he comes up with what he needs to solve any particular problem—but from his practice one can extract some general principles:

  • Methodological pluralism: Rather than try to come up with a single number or a single approach to summarizing player abilities, team strategies, or any other topic, he tried out a bunch of different ideas. In statistics, I like to say that each substantive hypothesis deserves its own analysis: it's generally hopeless to expect that you can run a single regression and pull off the answers to each of your research questions, one coefficient at a time.
  • Controlled comparisons: Instead of comparing simple aggregates, be more careful and make comparisons on pairs or groups of similar players or teams. As economists Rajeev Dehejia and Sadek Wahba demonstrated in a pair of influential articles (they have been cited over 2400 times since their publication a decade ago), these comparisons work only when you are controlling for appropriate characteristics. In the case of Bill James's analysis, player age is typically a key comparison variable. From the standpoint of applied statistics, controlled comparisons combine the averaging that you get from having a moderate or large sample size with the insight that comes from understanding individual cases.
  • Conceptual models used as guides to comparisons: James has written many times that he does not study statistical questions, he studies baseball questions. Each analysis is grounded in some goal. A conceptual model such as the defensive spectrum, or the narrowing of abilities, or the contribution of speed to both offense and defense, drives the direction of the study and motivates many of the details of the analysis. I have tried to follow these principles in my own work.

One central method of statistics that Bill James does not draw upon very often (if at all) is fitting parametric models. For example, James found that the power two in the Pythagorean prediction for wins worked pretty well. He didn't try to estimate the power from data, nor did he, for example, try to come up with a conclusion such as, "each additional run is worth 0.093 wins." On the rare occasions that he did estimate a parameter (for example, the relative values of stolen bases and times caught stealing), he buried his methodology and had no interest in making a big deal about the estimation.

Fitting models is something that statisticians are trained to do and in fact do all the time. Why didn't Bill James follow the example of Pete Palmer and others and try to estimate the relative values of walks, singles, doubles, and other outcomes? I can't really say, but perhaps he felt that the formulas he used, such as runs created, which generally relied on few (if any) estimated parameters, worked well enough.

James's most famous number may be 27—his estimate of the age at which the typical player (including the typical superstar) reaches his peak. James has explained, illustrated, and justified this number in various places, but I've never seen him set it up as a statistical estimation problem: "find the value where the average curve hits its peak." He just doesn't seem to think that way. A statistician would naturally want to estimate the form of the curve (possibly using a nonparametric method such as a spline), estimate the peak, and then see how this peak varies over time, position on the field, player ability, and other measurable factors.

There is a mathematical reason, perhaps, for a Jamesian reticence about estimating parameters. It goes like this. Consider some curve (for example, the rising and then falling curve of ability for a single or average player, plotted vs. age). It will have some peak. At the peak, the curve will be flat (mathematically, it has zero derivative) and, as a result, the precise location of the peak in time will be difficult to specify. If a player is expected to have maximum ability around age 27, his actual best season might occur at 25 or 28, or even 35, perhaps. Even with averages it can be difficult to spot the exact peak. So perhaps it is better to come up with a reasonable number such as 27, check that it works with the data, and then use it as a baseline to think about the occasional shooting stars who peak early and the drug-assisted sluggers who have their statistically best years in their late thirties.

Another thing that I do all the time, but that James almost never seems to do, is make graphs. He loves looking at numbers but seems to avoid any and all chances to make scatter plots, line plots, and the rest. This may be simply a matter of taste. Two exercises in which I often use graphs are (1) checking and cleaning data, and (2) exploratory analysis—finding patterns in data beyond what is explained by my existing models. It's possible that James is so in touch with his data that he can do all the checking and cleaning just by looking at the numbers—he thinks of each data point as its own unique person or event rather than merely as one point in a distribution. If so, it may be that the Bill Jameses of the world can do their exploratory data analysis by looking at numbers, but the rest of us may benefit from graphical displays.

My two favorite Bill James lines:

  • When someone wrote asking him to look into some idea or another, James replied, "I'm not a public utility. If you care so much about this, do the analysis yourself."
  • Responding to a comment by some humanist type who was yammering on about how there are all sorts of truths that aren't in the numbers, James pointed out that the alternative to good statistics is not "no statistics," it's bad statistics. People who argue against statistical reasoning often end up backing up their arguments with whatever numbers they have at their command, over- or under-adjusting in their eagerness to avoid anything systematic.

I also love how he sprinkles his writing with commonsensical but non-obvious points. For example, when talking about a player being replacement level, he points out that this is not an insult—if you're "replacement-level," you're good enough to play for one of the best baseball teams in the world. Finally, I appreciate James's focus on defining players based on what they can do rather than what they can't. These are insights that don't sound like much in isolation but pack a punch when coming at the end of a statistical analysis.

Let me conclude this appreciation by listing a few things that Bill James has written that baffle me. One of the lessons of statistics, as with science in general, is that we can learn from anomalies. What are some of James's anomalies—those items he has written (or not written) that surprise me?

Quantitative analysis of baseball can take many directions. James has always focused on the decisions of a team's management: which players to hire or let go, what positions to play them at, when to platoon a hitter or rest a pitcher, when to yank a starter and put in a reliever, whether to save your best reliever for "save" situations. Related are other recurrent themes such as rating players or teams, adjusting for park effects, and estimating the offensive value of stolen bases.

But there are other quantitative aspects to the game. I think it's just as well that James has not tried to estimate what factors predict player compensation—I couldn't care less about this one, and I get bored when I open the newspaper and find that the entertainment page or the sports page has become the financial page—but it's notable that he hasn't written much about the topic, especially given his extensive experience in arbitration meetings.

Another much-studied topic in baseball is game strategy. James has occasionally written about when it's advisable to bunt and when a team should use a pinch-hitter, but I haven't seen him spend much time on calculations such as, "If you have a man on second with one out, you can expect to get 1.2 runs," those Markov chain analyses that are a natural part of the sabermetrician's trade. I wonder why James has not written more about these analyses—is it just because others have done it well, so he feels no need to duplicate the effort?

Similarly, I've never seen James write much about strategies within a plate appearance. If a pitcher has a few different pitches, should he just throw them at random? Or does it makes sense to be more likely to throw a fastball (say) on the first pitch? Which sorts of pitches are more likely to be fouled off, and by how much? I realize that I'm demonstrating my ignorance by even asking these questions in this way; my point here is that I'm surely not the only person whose knowledge of sabermetrics is bounded by the Baseball Abstracts at one end and Moneyball at the other, and I'm surprised that James never seemed interested in tackling these questions systematically. I'm not demanding or even asking that he do so (see the "public utility" quote above), just curious that he hasn't done so already.

A similar line of study concerns a batter's choices. In one of his books, James remarked that if you swing at more pitches, you're likely to end up with fewer walks but a higher batting average. This makes sense, but I'd be interested in seeing a more systematic analysis, along with related issues such as when it makes sense to let a first pitch go by, and how effective is the strategy of having a batter who can exhaust the pitcher by fouling off pitch after pitch after pitch. (That last strategy has always seemed a bit unsportsmanlike to me, but that's another story.) Again, I'm not saying that James should do this or that analysis, just wondering about his choices of what to focus on. He seems more comfortable reimagining the decisions of a team's general manager than thinking about the microdecisions of individual players.

Bill James is now one of the biggest names in baseball, but he used to be an outsider. The very first article in his 1984 Baseball Abstract is called "Inside-Out Perspective," and it expresses his opinion that when studying baseball it is better not to be too close to the individual players and outcomes: "There will be in this book no new tales about the things that happen on a team flight, no sudden revelations about the way that drugs and sex and money can ruin a championship team. I can't tell you what a locker room smells like, praise the Lord. But perspective can be gained only when details are lost..."

Things have changed, though. By the time the updated edition of his Historical Abstract came out, in 2001, James was writing, "Are athletes special people? In general, no, but occasionally, yes. Johnny Pesky at 75 was trim, youthful, optimistic, and practically exploding with energy. You rarely meet anybody like that who isn't an ex-athlete—and that makes athletes seem special." I've met 75-year-olds like that, and none of them was an ex-athlete. That's probably because I don't know a lot of ex-athletes. But Bill James...he knows a lot of athletes. He went to the bathroom with Tim Raines once! The most I can say is that I saw Rickey Henderson steal a couple bases in a game against against the Orioles.

Cognitive psychologists talk about the base-rate fallacy, which is the mistake of estimating probabilities without accounting for underlying frequencies. Bill James knows a lot of ex-athletes, so it's no surprise that the youthful, optimistic, 75-year-olds he meets are likely to be ex-athletes. The rest of us don't know many ex-athletes, so it's no surprise that most of the youthful, optimistic, 75-year-olds we meet are not ex-athletes. The mistake James made in the above quote was to write "You" when he really meant "I." I'm not disputing his claim that athletes are disproportionately likely to become lively 75-year-olds; what I'm disagreeing with is his statement that almost all such people are ex-athletes. Yeah, I know, I'm being picky. But the point is important, I think, because of the window it offers into the larger issue of people being trapped in their own environments (the "availability heuristic," in the jargon of cognitive psychology). Athletes loom large in Bill James's world—I wouldn't want it any other way—and sometimes he forgets that the rest of us live in a different world.

Just last month, James concluded an article in Slate on racism and society by writing, "this situation is not a failing of the sporting world. Rather, it is that the rest of society has been too proud to follow our lead." The ultimate outsider is now in the clubhouse.

I noted above that I like BIll James's methodological pluralism, his willingness to try out lots of ideas and get different insights using different methods. Sometimes, though, the results confuse me. For example, he's argued for decades that on-base percentage and slugging average are more informative than batting average and RBI—but then he provides the following four statistics for every player in his historical abstract: games played, home runs, RBI, and batting average. At the very least, why not give on-base percentage and runs scored? Similarly, James was really into the concept of "secondary average" for a few years before it seemed to disappear. I can't tell whether he decided it was a bad idea or simply became interested in other things.

My biggest Bill James puzzle involves batting order. Over and over he talks about bad leadoff men and great leadoff men and criticizes managers who lead off with a speedy "contact hitter" with a .280 OBP. Where to start? The 1985 Abstract features a long discussion of the San Diego Padres' lead-off problem and then continues a few pages later with a lengthy explication of James's frustration with managers who don't know how to set up a lineup.

But then, in his 1997 book on baseball managers, James looked at the subject one more time: "There is probably no subject within the province of managing which draws more comment than batting order...Let's start with the broadest question: How much difference would it make?" He ran a simulation (on the 1930 Cubs) and reported his results: "How much difference was there between the 'correct' batting order, and the same players in an obviously irrational order? Surprisingly enough, very little...50 runs per season [i.e., about 5 games, using the standard 10:1 conversion factor]...if the difference between a reasonable batting order and an unreasonable batting order is only 5%, what do you suppose would be the difference between two reasonable batting orders? That's right: it's nothing." James concludes his discussion in his usual pugnacious style: "Our model is far from perfect...But for now, this discussion has two groups. On the one hand, you have the barroom experts, the traditional sportswriters, the couch potatoes, and the call-show regulars, all of whom believe that batting orders are important. And then, on the other hand, you have a few of us who have actually studied the issue, and who have been forced to draw the conclusion that it doesn't make much difference what order you put the hitters in, they're going to score just as many runs one way as another. You can believe whoever you want to; it's up to you."

My question is, where does the Bill James of the Baseball Abstracts fit in to this scheme? It's perfectly fine—admirable, even—for him to change his mind on the importance of batting order, but it's odd that he doesn't acknowledge the shift. Was it actually okay all those years for those managers to be leading off with .250 hitters who never drew walks?

I don't want to conclude on a down note, though. It is only because Bill James's ideas, methods, and principles have influenced me so much and have burned themselves into my brain that I am aware of the places where he's changed. In statistics we like to say that God is in every leaf of every tree: whenever we work on any serious problem in a serious way, we find ourselves quickly thrust to the boundaries of what existing statistical methods can achieve.

21 comments have been left for this article. (Click to hide comments)

BP Comment Quick Links

Tarakas

Excellent analysis.

To me, the most important aspect of Bill James was his ability as a popularizer of the statistical analysis of baseball. He made the benefits of such an approach clear, accessible, and entertaining enough to draw in and convince people not normally inclined to read such a work or think in such a way.

May 05, 2011 05:52 AM
rating: 6
 
Richie

What Tarakas says. Both paragraphs.

May 05, 2011 08:32 AM
rating: 0
 
jamin67038

James said the difference between two 'reasonable' batting orders is nothing, but he never says that it's ok to put a guy with a .300 OBP in the leadoff spot- that would fall under 'unreasonable' batting orders.

May 05, 2011 08:47 AM
rating: 2
 
poedbz

Agreed, that would be my interpretation as well.

May 05, 2011 20:28 PM
rating: 0
 
RaysProf

The comments and analysis in this article is supportive of Michael Lewis's claim that what James really wanted to do when he published the Baseball Abstracts is to write. And since his writing was easily accessible to a population that lacked an understanding of the fundamental theorem of calculus, and even though others (Cook, Palmer, etc.) had produced models that were more statistically sound, it was James that people read.

I agree with Tarakas. We should give James credit for popularizing statistical analysis. But like the efforts of Mendel, the German-Czech monk cross pollinating plants to make headway in the field in genetics, this area of research has moved on.

May 05, 2011 08:50 AM
rating: 2
 
RaysProf

Looking at my comments I find typos, lack of agreement in tense and number - just poor grammar. I would mark myself down if I could.

Since it appears that my wish of an edit option on this webpage is unlikely to be granted, I need to remember that one can't simply bang out a response in the morning on a half-cup of coffee.

May 05, 2011 12:14 PM
rating: 2
 
djardine

Agreed. The world of sabermatricians may have moved on, but thanks to articles like this, Bill James will not soon be forgotten.

May 05, 2011 15:18 PM
rating: 0
 
GarryPowell

A friend of mine recently discussed books that have changed our view of the world. Some on mine were Catch 22, Guns,Germs and Steel and many books on Darwin and evolution. As a lifetime baseball fan, Bill James and his Baseball Abstracts fall into that category

May 05, 2011 10:05 AM
rating: 0
 
HonusCobb

This is how it worked for me. When I was in high school I picked up Moneyball (I'm now nearly 25). Moneyball led me to the works of Bill James. Those books taught me to question basically everything and drastically changed my view of the world. The credit belongs to SABRmetrics!

May 05, 2011 15:42 PM
rating: 0
 
RaysProf

Just curious, did you attend college? In my mind the number one goal of college should be to "train" students to be rationally skeptical. Sadly society possesses too many people who lack sufficient skepticism and thus subscribe to the view that their health can be improved by alternative medicines which have not passed controlled clinical tests, and that there exists vast government conspiracies to hide extra-terrestrial beings, to blow up buildings and to allow an individual, who doesn't quite look like all his predecessors, to hold of the office of the POTUS. But these same individuals show an irrational skepticism of the quality of vaccinations after decades of success at eliminating a host of horrible diseases because a former playboy bunny claims they should.

May 05, 2011 22:20 PM
rating: 1
 
HonusCobb

I did attend college. And college is when my view of the world took a 180 degree turn. But the questioning of conventional wisdom first really began when I started reading Sabermetric based books.

And I know exactly what you mean. But at the same time, those people who "lack sufficient skepticism" develop too much skepticism about newer developed ideas because those ideas contradict what they have always been taught. So it's not that they're not skeptical about things, they're just not skeptical in the form of the scientific method.

So I may have said that SABRmetrics deserved all the credit for changing my world view but it really just gave me a head start before college. What really changed my world view were philosophy courses, geography courses, history courses, and my gen ed science courses.

Just Curious RaysProf, are you a professor?

May 06, 2011 13:50 PM
rating: 0
 
HonusCobb

and perhaps also the author of this article?

May 06, 2011 13:53 PM
rating: 0
 
RaysProf

Yes, I am a professor at a small liberal arts college but of physics not statistics (though I use them often). My apologies, I misread your post and incorrectly assumed your interest in sabermetrics occurred post college. Ironically we both experienced a common set of revelations in a similar sequence. In my mind, baseball is a microcosm of humanity with sabermetrics playing the role of skepticism of the human observation. I have suggested teaching a course on the enlightenment through the readings of baseball.

May 08, 2011 12:00 PM
rating: 0
 
HonusCobb

Cool, what would be on your reading list for that course?

May 08, 2011 16:18 PM
rating: 0
 
BP staff member Ben Lindbergh
BP staff

By the way, folks, Bill James will make an appearance on Colbert tonight, which sounds like it would be worth a watch.

May 05, 2011 10:22 AM
 
dstamand

Excellent article. A rare interesting read from start to finish.

I was curious about the comments relating to, "Similarly, I've never seen James write much about ...." As I understand it, Bill James is also a consultant to baseball team(s?). If he wrote about all of his analytical insights, what would he provide on a paid consultant level?

Thanks.

May 05, 2011 10:40 AM
rating: 0
 
ofMontreal

This.

James' analysis has disappeared accept for the branded books that come out annually that are fun to read but don't give a lot. The Red Sox hiring of James was a coup on many levels.

May 05, 2011 13:15 PM
rating: 0
 
HonusCobb

One of my favorite Bill James lines was in "The New Bill James Historical Baseball Abstract." Cecil Fielder was in his top 100 first basement and his bio was only a sentence. Most of his short bios were at the very least a couple paragraphs. This may not be word for word as I am too lazy to go get my book out but it went something like this:

"Cecil Fielder - A big fat guy that hit home runs for a few years."

May 05, 2011 15:40 PM
rating: 0
 
Dave Holgado

I think his Don Mattingly comment was similarly terse:

"100% ballplayer, 0% bullsh*t."

May 06, 2011 09:13 AM
rating: 0
 
Isaac Lin

From what I understand, Bill James didn't take any advanced statistics classes in university and so he generally hasn't employed the types of techniques you cite (parameter estimation, Markov chains), and he generally hasn't written about the research of others.

Regarding secondary average, James knew it didn't express anything that isn't already covered by on-base percentage and slugging average; he was just looking for a short-hand way to explain, to the non-SABR crowd, a player's value beyond batting average. Now that the virtues of OBP and SLG are much more widely known, secondary average is no longer needed as a proxy.

May 06, 2011 19:02 PM
rating: 0
 
BarryR

I first read Bill James' work in 1976, the second edition of the Baseball Abstract. I had never seen home/road or lefty/righty splits before. The defensive spectrum was an amazing concept - not just what it is often reduced to, which is a list of positions in reverse order of offensive productivity. Because you can move players from the left end of the spectrum (SS, 2B, CF) to the right end (RF, LF, 1B) but can rarely move them in the other direction. Good organizations tend to pile up players at the left end, bad ones at the right end. I talked with him a number of times about the various Abstracts, which ultimately led to my working with STATS, INC. for a couple of years.
To those of you who have followed in the sabermetric revolution, the dark ages which he ended are inconceivable. No one (except maybe Branch Rickey) thought of OBP as the most important offensive stat until he showed it to us, nobody even knew what isolated power was.
One major way in which James differs from Gelman and his colleagues is the question of precision. In rigorous academic statistical analysis, a lack of precision is sloppy. But when dealing with a sport where bloopers become hits and line drives outs, where the difference between an error and a hit is who the official scorer is, precision is not as important. What is important is significance - the difference between 91.8 runs created and 92 is meaningless, between 82 and 92 matters. When it comes to fielding statistics - what matters and what doesn't is nearly impossible to determine, and, with more and more shifts taking place in the infield, becoming harder.
The difference in opinion on the lineup question is the realization that the difference between a .360 OBP and a .340 is about two times on base a month. Obviously a hitter with a .290 OBP shouldn't be batting first - or anywhere, really - but that isn't the question.
Bill is now in the belly of the beast and is limited in what he can say at times, but the principles he laid out still work. And the change he made in how people look at the game and those who have and do play it has been invaluable.

May 08, 2011 20:48 PM
rating: 1
 
You must be a Premium subscriber to post a comment.
Not a subscriber? Sign up today!
<< Previous Article
Premium Article The BP Broadside: The ... (05/05)
<< Previous Column
Baseball ProGUESTus: F... (04/29)
Next Column >>
Baseball ProGUESTus: M... (05/13)
Next Article >>
Premium Article Overthinking It: Year ... (05/05)

RECENTLY AT BASEBALL PROSPECTUS
Fantasy Article Expert League Auction Recap: CBS AL-Only
Fantasy Article State of the Position: Relief Pitcher
Fantasy Article The Quinton: Market Corrections and Underval...
The Week in Quotes: February 23-March 1, 201...
Premium Article Rumor Roundup: You Can't Predict Padres
Premium Article Transaction Analysis: It's the Latest Johan ...
Fantasy Article Fantasy Players to Target: Relief Pitchers

MORE FROM MAY 5, 2011
Premium Article Overthinking It: Year of the Stolen Base?
Premium Article The BP Broadside: The Premature Burial, by E...
Premium Article Divide and Conquer, AL West: The Disappointm...
The BP Broadside: Eight Things I Hope Are Tr...
Fantasy Article Fantasy Beat: Value Picks in the Bullpen

MORE BY ANDREW GELMAN
2011-05-05 - Baseball ProGUESTus: A Statistician Rereads ...
More...

MORE BASEBALL PROGUESTUS
2011-05-24 - Baseball ProGUESTus: Answers from a Sabermet...
2011-05-19 - Baseball ProGUESTus: Ask a Sabermetrician
2011-05-13 - Baseball ProGUESTus: Maris and More
2011-05-05 - Baseball ProGUESTus: A Statistician Rereads ...
2011-04-29 - Baseball ProGUESTus: Fantasy Baseball's Foun...
2011-04-22 - Baseball ProGUESTus: Baseball, Boyhood, and ...
2011-04-15 - Baseball ProGUESTus: The Scott Boras Factor:...
More...