keyboard_arrow_uptop

Last week, I wrote an article about the fact that no Baseball Prospectus staffer picked the Rangers to be serious contenders this year. (Neither did anyone else I could find.) Because the Rangers lead the race for the second Wild Card spot, the question was whether we, as a sabermetric community, missed something that ought to have allowed us to forecast Texas’ success, or whether they were a true fluke. I concluded, in the end, that we underestimated the value of the team’s organizational depth, suffered from recency bias that unfairly diminished our expectations of once-great players, and undersold the degree of variance possible in a league with many teams on the edges of contention.

The Rangers were a good first case study in this area, but the best and most important team on which to challenge our views is the Royals.

As with the Rangers, no BP writer picked the Royals to finish first in their division this season. That’s an even more glaring error than it was in Texas’ case, because the Royals are coasting toward the easiest division championship of any team in baseball this year. No one pegged the Royals for second in the AL Central, either. Eleven of us pegged them to finish third, a rather rude way to treat a defending pennant winner. Then again, with a sub-.500 third-order winning percentage in 2014 and a disappointing winter under their belts, we weren’t really thinking about the Royals as a pennant winner. Rather, we considered them a team of the times, a pop-up powerhouse without true excellence or staying power.

Were we wrong? I’ll restate the caveats I gave in the Texas article: Not all doomed predictions are failures of the science of prediction. Some are simply done in by the violent vicissitudes of chance and variance, which aren’t the same thing but work in devilishly close harmony. Others are made with incomplete information; we can’t call for a mulligan every time new data emerges.

Did we fail by not predicting that the Royals would be really, really good? Or did they simply outrun any reasonable prediction?

Possibility #1: We missed on the Royals, and there’s no excuse.
Last time, I began by laying out the case that the Rangers were a fluke. I feel confident that I drew my own conclusions without being influenced by the order in which I presented the scenarios (not least because I came to my conclusion before I did the writing, as I have this time, as well), but just to head off as many mental blocks as possible, we’ll start this time with the premise that we ought to have seen Kansas City coming.

Supporting Evidence: In essence, the support for this side of the argument boils down to the fact that the Royals were the hottest team in baseball in the second half last season, carrying into the playoffs. They were not just hot, but hot in a culminating sort of way, finally meeting the lofty expectations pinned on them a few years earlier, when the organization was hailed as having perhaps the best farm system ever assembled.

The Royals’ best starting pitcher for much of last season was Danny Duffy, and Yordano Ventura finished sixth in a strong Rookie of the Year race. (He should have finished higher.) From the All-Star break on, Eric Hosmer had a 124 OPS+. Mike Moustakas continued to battle slumps and struggle hitting against the shift, but got really hot in October. While James Shields and Norichika Aoki were important cogs in the machine, the majority of the Royals’ 2014 success was the result of their prospect power finally coming to bear. Heck, even Shields’ contributions, and those of Wade Davis, can be traced back to the team’s impressive accumulation of high-upside minor-league talent during Dayton Moore’s reign as GM; they were acquired for Wil Myers, our no. 7 overall prospect the year of the trade.

If we’d bought into the Royals, we would have been buying the idea that while development isn’t linear, guys don’t often un-break out. We’d have been investing in the futures of Lorenzo Cain, Hosmer, and Moustakas, all of whom have been excellent this season. (We’d also have been investing in the futures of Duffy and Ventura, who have added nothing this season.) Given what the analytical community most prizes these days (young, cost-controlled players with growth potential), picking against the Royals represented not only questionable judgment, but uncharacteristic cynicism about a team built in what we typically consider the right way.

There’s also an argument that saber sorts have been too slow (much too slow; embarrassingly slow) to come around to the idea that some bullpen performances are sustainable. The Royals thrived on their bullpen last summer and fall, as has been so well documented, but instead of admiring the strength and depth of that unit, the majority of us kept our expectations quite tempered. Maybe we’re not giving this set of exceptional relievers (the Royals lead MLB in relief DRA, at 3.43) enough credit.

Possibility #2: The Royals came from nowhere, and to nowhere shall they return.
Supporting Evidence:
Kansas City has a 102 cFIP, meaning that their pitchers’ true talent level is below average. They’re 10th in park-adjusted Defensive Efficiency, seventh in baserunning runs, seventh in True Average. All four National League powers have higher third-order winning percentages than the Royals, as do the Blue Jays, Astros, Indians (!), and Yankees. These are just stats, but they move one slowly toward the conclusion that the Royals aren’t having quite as strong a season as they appear to be having.

The Royals have given Paulo Orlando and Jarrod Dyson over 400 plate appearances, and gotten roughly a .700 OPS out of them. That’s utterly unexpected and wildly valuable, given the defense those guys play, and that helped them weather the prolonged absence of Alex Gordon. So, too, did the addition of Ben Zobrist. Zobrist and Johnny Cueto are Royals because of their organizational depth and their ability to trade from it without denting the big-league roster; that is to their credit. It would have been very hard, though, to somehow foresee the team trading for a player at the deadline and having him hit .336/.421/.520 over his next 145 plate appearances.

We didn’t anticipate Moustakas hitting .281/.347/.446, and I’m not sure we could have been expected to do so. Moustakas had batted .236/.290/.379 in nearly 2,000 big-league plate appearances prior to this year, and despite showing some serious pop last October, he only had a .259 OBP in the postseason. Most of us didn’t assign any projected value to Edinson Volquez or Chris Young, which was probably myopic, but their uneven track records would have made it folly to place much faith in them.

Verdict: I’m unable find fault with our collective inability to see the Royals coming. To me, this is a distinctly different case than that of the Rangers. In their case, we allowed a mixture of injuries and down seasons from proven players to blind us to their strengths. In the Royals’ case, on the other hand, we erred by underestimating the sustainability of their bullpen dominance, and we failed to foresee the remarkably strong performances they would get from some very unlikely places.

Going forward, those are things we should remember, and build into future predictions. I wrote a piece in May about the strength of the Kansas City positional roster, but I was wary about what would happen if and when some core position player got hurt. In retrospect, I overstated the risk there. As Dyson and Orlando have shown (and as Kelby Tomlinson has shown in San Francisco, and as Ryan Goins and Chris Colabello have shown in Toronto, and are you starting to see a pattern?), even guys with very little established offensive ability can hit pretty well over what feel like long stretches. One of the fundamental truths of baseball, and one we too often forget, is that no single player can win a team a title, so the absence of one can’t take a team out of the running, either. Variance can allow even bad players to look good for two or three months at a time.

More importantly, bullpens aren’t what they used to be, and we shouldn’t project them the way we used to project them. There’s still a problem of establishing a sample size we can consider reliable, but relievers aren’t wild cards, and the regression factors we use to account for the variance we expect over those small samples need to be modified in light of the way pitchers are now trained and deployed.

We couldn’t have fairly assumed the Royals were going to carry their dominant late-season relief efforts from 2014 not only into 2015, but all the way through it. We had to see this sort of thing happen a few times before it could possibly be rational to expect it, even in an extreme case. From now on, we might assign more weight to bullpen quality when projecting a team’s season, but back in April, we were working with the best information we had.

You need to be logged in to comment. Login or Subscribe
rudyrosen
9/08
And BP continues to miss two huge factors that contribute to the Royals success, defense and contact hitting. This team has the best record in the AL since opening day 2013...keep telling yourself you're unlucky and you will keep making bad predictions on this team. You didn't miss on Volquez and Young because they are having lucky seasons. You missed on them because you didn't factor in that they were pitching in front of this defense. Just like you have missed on other underwhelming Royals arms the last few years that manage an ERA 0.50 to 1.00 below your projections. Keep thinking you got unlucky and keep blowing it...it's much more convenient than looking in the mirror and acknowledging that this team has exposed a flaw in your model.
Ogremace
9/09
Perhaps the defense is benefiting Volquez, but Young has always survived on aberrantly low BABIPs, pitching in front of various defenses. Dave Cameron wrote about this earlier this year. As for the projections, being wrong about certain pitchers a couple times doesn't prove a bias, unless you have a concrete reason to explain those errors. Which is basically what Matt addressed in the article regarding relievers. Maybe the projections are bad for this team and will continue to be. But tue Royals aren't the first team to outplay projections two years in a row.
timber
9/09
This is actually the third year in a row.
bleaklewis
9/08
How will bullpen projections change? DO you assign more weight to more recent seasons or is there a bigger change that needs to be made?
sbnirish77
9/08
"I'm unable find fault with our collective inability to see the Royals coming." The Royals 'coming' was last year. They were already here for this year. Maybe BP missed this year because they never adequately explained last year. From this years Prospectus ... "then for whatever reason they started to win". "In the Royals' case, on the other hand, we erred by underestimating the sustainability of their bullpen dominance, and we failed to foresee the remarkably strong performances they would get from some very unlikely places." The bullpen alone can't come close to closing the gap between BP projection and reality. As far as unexpected performance. I'll give you Moustakas but the loss of Gordon more than evened that ledger. PECOTA has always downgraded an aging player so when any older player performs well (Volquez, Young, leading RBI (gasp) guy Kendry Morales) its a convenient excuse. There's a ton of those players for every team. The most obvious reason for the Royals success - they were in a division in which three others teams (Tigers, Indians, White Sox) so grossly under-performed the TWINS are going to finish 2nd. This division, that was supposed to be the most difficult to win, proved to be a patsy.
oldbopper
9/09
The answer seems to be pretty clear that the failure to put adequate emphasis on the sustained quality of the Royals bullpen led to this embarrassing mistake. With starters being asked to do less and less the importance of the bullpen is quickly becoming the number one factor in season long excellence. Bullpens are no longer a bunch of stiffs (unless you are the Red Sox) leading up to, hopefully, a shut down closer. The Yankees recognized this and have put together a strong group with two closer quality finishers and it seems completely reasonable to expect Betances and Miller to continue their performances for next year and beyond. To not realize that Herrera, Davis and Holland were going to repeat their 2014 success seems surprising for knowledgeable baseball men.
Dodger300
9/09
"We couldn't have fairly assumed the Royals were going to carry their dominant late-season relief efforts from 2014 not only into 2015, but all the way through it." Late-season? That sounds like an admission that you weren't paying much attention last year. Here are the ERA's of the Royals big three relievers from March, 2014 through July, 2014: Wade Davis, 0.97 Kelvin Herrera, 1.76 Greg Holland, 1.77 It sure sounds like a facts are twisted to create a false narrative. As for explaining why no one BP even picked the reigning American League to finish as high as second in their division this year, I think it is best to rely on Occam's Razor: "When you have two competing theories that make exactly the same predictions, the simpler one is the better." The simplest and best explanation is "We missed on the Royals, and there's no excuse."
bishopscreed
9/09
I think its interesting to say nobody could have seen Moustakas coming. From a perspective of analyzing past performance I think that's right. However, from a scouting perspective, it was pretty easy to see that he was completely refashioning his swing to defeat the shift. The Royals batted him second in their opening day lineup this year because they already knew he was a different hitter. It would have been impossible for PECOTA to see that coming; would it have been impossible for BP? As for the rest of the team, PECOTA had some glaring misses that are more impactful than understandably missing on the bullpen, and it seems like analysts took PECOTA at face value. Forget about Dyson and Orlando posting .700 OPSs; even though Dyson settled in as a 1-2 win supersub about 4 years ago. What matters more is that PECOTA pegged Lorenzo Cain for a .703 OPS, Alcides Escobar for 0.7 WARP, and Eric Hosmer for a .279 BA. Mathematically, that all makes a kind of sense, but human analysts need more than a mathematical model to make predictions. If you had asked Kevin Goldstein six years ago whether Hosmer would be more than a .280 MLB hitter, what would he have said? The current Royals team started to come together in 2013, when they won 86 games after starting 22-30. They've been winning long enough that a fluke is quite unlikely. It's more likely that they've had some specific coaching successes (Moustakas especially) and crafted an approach that suits their somewhat atypical home park. Now, to assume they would win in 2015 just because they were good the previous two years would be folly, but it also seems possible that expecting regression can become an excuse to accept model predictions uncritically, without really thinking through the case for each player. At this point, PECOTA has missed badly on the Royals for three consecutive years. After all that, I have a couple of suggestions for future changes to PECOTA: First, find a way to incorporate batted ball data so that the algorithm knows whether a player has been victimized by defensive shifts, and maybe increase the comparability of other players who have been similarly victimized. Next year, when Adam Laroche or someone suddenly bounces back, his profile could at least have a hint that he might have pulled a Moustakasesque retooling. Second, think about expanding the percentage ranges for players with odd career trajectories, like Moustakas, Hosmer and Alex Gordon before him. To the human eye, it's clear those guys have made adjustments over a course of years to reach the potential that was once expected of them as prospects. The computers don't seem to know this. As a Royals fan I pay more attention to their players than any others, but I'm sure there are lots of similar cases out there, where Pecota could try to find comparable players who aren't just statistically comparable, but also ontologically comparable in various ways. It would be interested to see if such an addition would result in a system that verified worse or better than the current one.
gpurcell
9/09
When you dig into the projection systems (not just PECOTA but ZIPs and other ones) what you find is that they do not explain much variance--R-squared values in the .33 range. Further, you'll find that they don't improve much on projections based simply on regressions to the league average. This indicates either that A) the models are substantially underspecified, B) the modeling approach based around explicitly incorporating regression to the mean is flawed, or C) there is fundamental variation in performance that represents chance and luck that prevents accurate predictions. If I had to bet money the failure of projection systems is mainly a combination of factors A and B.
gpurcell
9/09
I'd argue the problem is the basic WAR/WARP approach itself and the role the Royals bullpen plays in their success. Using team WAR/WARP to project a season employs the statistic as an aggregate measure across all contests. But this very assumption is incorrect because a season is not a summation of total performances--it is a set of individual contests. A historically awesome bullpen allows what appears to be a fairly insignificant contribution to aggregate WAR/WARP--relief pitching performance--to be leveraged only in those cases where that excellence will impact the marginal outcome of contests. There may be issues within the individual projections not just for PECOTA but for all the projection systems--I suspect there is fundamental problem with the approach of using a sledgehammer to regress individual performance to the league average--but what the Royals mainly show is that season results must be modeled as a series of discrete contests.
kcboomer
9/09
It is clear that when it comes to the Royals saber analysis has failed. The sum of their parts seems to add up to more than the whole. They have had a lot of things go wrong this year (Alex Rios is awful, Infante is worse, Gordon missed two months, Cueto has been a bust, Perez continues to decline at the plate, etc.) and have prevailed. The bullpen thing should not have been missed. Madsen, Hochevar, and Young clearly added significant depth to the pen and added it right where the Royals needed it in the middle innings. I think this is the case of BP relying on a platitude ("Bullpen regress") rather than analysis. Volquez may be nothing more than good scouting. Maybe a scout saw something that could be fixed. Who knows but we should just accept that humans are hard to predict. Maybe the much vilified Ned Yost figured something out. If you notice that in most losses he tends to use his weaker bullpen pieces saving his studs for games they can win. There was one guy who did predict this: Rick Sutcliffe. From Day One he predicted the Royals would win the AL Central.
lmarighi
9/09
Predicting that Mike Moustakas would hit so well this year would have involved some serious drug-induced hallucinations. With all due respect to the previous poster who said he could see it in Moose's swing, there has been an awful lot of amateur (and professional, probably) scouting that "found" swing changes (or mechanical changes for pitchers) that predicted greatness, only to have them flop. So if PECOTA was to incorporate all the adjustments scouts see, it would be at least as inaccurate as it currently is, or maybe even more. Also, claiming that Ryan Madson was obviously a stellar addition this year? Nonsense. He hadn't pitched in the big leagues since 2011. How the hell can anyone predict with accuracy how well he will do after almost 4 years of rehab and setbacks? Predicting that they would likely have a good bullpen seems like a more reasonable place to start, although I don't think anyone predicted it to be terrible. I thought Matthew did a pretty good job of going through the thought experiment, and while I definitely think there are things that can be learned from this, I don't think it's an obvious miss.
bishopscreed
9/12
To be clear, I didn't mean that I am a scout; I meant that the Royals broadcasters and fans in general were aware of Moose's changes as early as March. That may happen rarely, but it can happen. Rather than seeing incorporation of scouting information as impossible, I choose to see it as challenging and cutting-edge. :)