Premium and Super Premium Subscribers Get a 20% Discount at MLB.tv!
May 6, 2014
Is Oakland's Run Differential for Real?
As of the moment that I write this, the best run differential in baseball is owned by…the Oakland A’s. Raise your hand if you saw that coming. Also, please raise your hand if—since I mentioned the A’s, you can work the word “moneyball” into this paragraph. I’ve run out of ways to do it.
Sure, like everything else around this time of year, we can yell “small sample size!” and use that as a convenient excuse to ignore the patently obvious. The A’s have scored a few big wins so far in 2014, including an 11-3 triumph over the Astros, a 10-1 win over the Astros, and a 12-5 win over the Astros. (In fairness to the Astros, the A’s also beat the Rangers 12-1 one night. Maybe they just like playing against teams from Texas.) That’s fueled a lovely run differential, and also the most wins in the American League. But is that run differential a reality or a small sample size mirage?
BP alumnus Jonah Keri, who edited the book that turned me into a sabermetrician, tapped me on the shoulder and provided the appropriate rainbow sprinkles to cajole me into running a few analyses on the subject. Additionally, since I’m of vaguely French-Canadian ancestry (although about all that’s left is that extra “e” in my last name from when my ancestors changed the family name to honor this guy), there’s an honor code to uphold.
How long does it take until a team’s run differential isn’t just random noise anymore?
Warning! Gory Mathematical Details Ahead!
In general, I like to look for the point where the correlation reaches .70. For the uninitiated, 0 means that there’s no relationship between the two numbers, and 1 means there’s a perfect relationship between them. It’s a matter of how close you want that number to be to 1 before you feel comfortable, and you can see a chart below of how the correlation approaches 1.0 as the season goes on. The correlation hits .70 after 39 games. So, around the 40-game mark—mid-May—run differential starts to be a good predictor of what things will look like at the end of the year.
But those who read that closely are probably thinking, yeah, the reason that the correlation is so good is that those 40 games are baked into the end-of-the-season numbers. In some sense, we’re comparing something to itself, which usually produces a pretty strong correlation. What if we wanted to know how long it took until we could be independently sure that the team’s run differential is real? For that, we need a more complicated method called Cronbach’s alpha.
The idea behind Cronbach’s alpha is similar to work that I’ve done on when stats for individual players stabilize. For individual players, I’ve taken groups of, say 100 PA per player, and split them into equal 50-PA bins. If a stat is stable—say strikeout rate—then we’ll see a good correlation between the two groups of 50 PA each. If it’s a big enough correlation, we’ll say it’s stable at 50 PA. Cronbach works on the same basic principle, except that mathematically it slices and dices those 100 PA into every possible way to split into groups of 50 and averages out the correlation between them.
Again, I took all team-seasons from 1962 onward. For each game, in sequence, I figured out what contribution to the team’s overall run differential it made. That’s a nice way of saying I figured out how many runs the team won or lost by. I entered these numbers into the Cronbach’s alpha formula and looked to see how deep into the season I had to go before Cronbach’s alpha hit that magical .70 number. It turns out that after 140 games, you get there. Now, that means that you have two samples of 70 games per team-season being compared to each other. So, we can say that after 70 games, a team has a sample that’s independently big enough to make statements about its true talent level. If the season started over for some reason, we could expect that those 70 games would give us a good idea of what to expect over the new 162-game season.
Of course, the season doesn’t reset like that. The 70-game threshold is for when you want to know some deeper truth about the team in question, rather than just to peek ahead to what the final standings will look like. Still, it suggests that we can make some reasonable conclusions about a team’s true talent level by the All-Star break.
Run Differential: The Key to Understanding Life, The Universe, and Everything
And as for those A’s, check back in a week and a half to see whether they still have the best run differential in the league. At that point, we’ll know whether we can expect it to last through the rest of the summer.