February 8, 2011
PECOTA Through the Looking-Glass
As backronyms go, “Player Empirical Comparison and Optimization Test Algorithm” is squarely on the intimidating side. That’s why it’s somewhat comforting that our BP-branded projection system also has a human face, even if that face couldn’t muster a very convincing mustache. For those who’ve joined us late, PECOTA’s name is jack-of-all-forecasts Nate Silver’s nod to Bill Pecota, a light-hitting utility infielder of the 1980s and ’90s whose scrappy play earned him entrance to even the most hardened of baseball analysts’ hearts, despite questionable artistic taste and the kind of stat lines that normally invite ridicule from the sabermetric set (especially when they’re associated with someone wearing a Kansas City uniform).
Odes to Pecota the player have already been written, so I’ll refrain from presenting a complete history, but we all know the type. Pecota played as many positions as he did seasons, putting Willie Bloomquist to shame in terms of positional flexibility (if not value) by spending time everywhere on the diamond that it’s possible to appear, with the possible exception—at least in some seasons—of the basepaths. Although he retired a year before the offseason that produced our first annual, robbing us of the ability to put something snarky about him on paper, we likely would have pointed to his utter lack of patience and power as compelling reasons not to employ him. Still, with 21st-century hindsight, we can say with some confidence that Pecota’s glove may have made him worth playing at times, since our new-and-improved implementation of FRAA gives him credit for 36 runs in the field, albeit with a margin of error of 23.
That said, it’s hard to deny that this action shot perfectly captures Pecota in his offensive element; all too often, his swings came up empty. PECOTA’s Wikipedia page tells us, “The acronym was actually based on the name of journeyman major league player Bill Pecota, who with a lifetime batting average of .249 is perhaps representative of the typical PECOTA entry.” That first clause is undeniably true; the second seems like supposition, especially with no citation to support it. Still, there’s something to that notion: While PECOTA wouldn’t have lasted as long as it has if it spit out Bill Pecota–like batting lines for Albert Pujols, on occasion its output comes closer to that lowly baseline than our instinctive (and often overly optimistic) internal algorithms might lead us to expect.
PECOTA’s pesky insistence on regressing to the mean would have made light of even its namesake’s occasional achievements had the system been around during his playing days. In Colin Wyers’ introductory article yesterday, he mentioned a set of simplified PECOTAs that had been used to assess the system’s historical performance. I asked him to pluck Bill Pecota’s retrojections out of that pile for me, in hopes of making a point. Take a look at a five-year snapshot of Pecota’s career, with 1992 retrojection included:
Like Michael Dukakis, Pecota tanked in 1988; the following season, he showed a bit of pop for the first and only time (in an extremely limited sample), and in 1990, he managed a league-average performance. In 1991, Pecota got off to a solid start and was unexpectedly installed as the Royals’ starting third baseman (over Kevin Seitzer, a far superior player) by manager Hal McRae, K.C.’s third skipper that season. He responded by posting career highs in every offensive category. Following that performance, it might have been tempting to forecast more of the same from the erstwhile utility man: Not only was he riding a three-year trend of improvement, but he’d undergone a change in role that could conceivably have boosted his confidence and better suited his game.
In retrospect, PECOTA wasn’t buying it, quite reasonably expressing some pessimism about the likelihood of a marginally talented player on the wrong side of 30 having established a new performance plateau. As it turns out, Pecota outstripped the projected decline, sinking all the way back to near-1988 levels of futility. As Pecota’s Icarus-like 1991 season suggests, it’s easy to convince ourselves that we know more about a player than a projection system without a mind of its own, especially when we have some personal investment in his success, and (as is often the case) subjective factors can be enlisted to bolster our argument.
Sure, maybe his manager’s vote of confidence or a more prominent role had spurred Pecota to new on-field heights; maybe his exodus from the only organization he’d ever known after the ’91 season contributed to his decline. Such things are impossible to rule out, but equally impossible to prove: Even if we came to some empirical conclusion about the typical response to a change in scenery or role, we couldn’t be sure that it would apply to any particular player. Given the difficulties involved, it may seem like the only responsible thing to do is to make like PECOTA and ignore the human element, but it’s hard (and perhaps ill-advised) to disregard unquantifiable factors entirely.
Last Friday, Tommy Bennett penned a paean to the period of anticipation before a PECOTA release, when anything still seems possible and no cold, hard numbers have yet come forward to deflate our fondest hopes for certain players. The truth is that sometimes, much as we may value its point of view, PECOTA can be a bit of a buzzkill, the equivalent of the guy at the party who brings the proceedings to a screeching halt by asking the host to turn down the music, if not the neighbor who calls the cops. Regardless of the percentage of satisfied customers, each release is accompanied by at least a few assertions that we’ve underestimated certain players or shortchanged the efforts of the league as a whole, and this year has been no exception. In some past cases, we’ve found that we had overlooked something and that there were indeed tweaks to be made, which is why we welcome your feedback—it’s much like the internal testing we conduct prior to publication, but on a much larger scale. In others, though, PECOTA is simply doing what PECOTA does, with little regard for our feelings.
If you scan the PECOTA leaderboards of years past, you’ll notice that regardless of category, PECOTA injects statistical lithium into the coming year’s totals. For instance, here’s what PECOTA had to predict about the most prolific home-run hitters of the past three years:
Even though PECOTA underwent tune-ups under the hood in each of these years (if not quite as drastic as this offseason’s engine overhaul), the system stayed consistent in calling for 13 players per season to reach the 30-HR plateau. The ranks of the actual 30-HR clubs were considerably more swollen, although an offensive drought last season made things fairly close. We can see the same effect at work when looking at the best projected and actual individual home-run performances:
As Tommy pointed out, PECOTA’s conservative figures don't mean that outliers won’t appear; it simply means that PECOTA can’t see them coming. (In some senses, that’s a blessing, since it makes the season more interesting and saves us the trouble of freshening up Colin’s nutrient bath.) The numbers you see in the spreadsheets are weighted-mean projections, which PECOTA considers the most likely outcomes. We’ll soon be presenting percentile forecasts that offer some idea of a player’s chances of exceeding (or failing to meet) his baseline projection, but even then, we won’t be able to tell you with certainty which players will defy PECOTA’s expectations by laying waste to the league.
In some cases, players get lucky. In others, they simply cease to be unlucky, and in still others, their true talent level takes an unanticipated step forward. Once those seemingly anomalous seasons take place, PECOTA incorporates them into its projections for the following year and revises its estimates upward, but rarely anticipates a repeat performance, barring a favorable spot on the aging curve.
That doesn’t stop us from identifying players whom PECOTA might like more than the prevailing opinion, but where does it leave us with a few of this year’s trickiest test cases? Take Javier Vazquez (please). As someone whose ERA has routinely failed to match his peripherals (or more accurately, the peripherals we generally expect to predict ERA), Vazquez has come with plenty of baggage even at the best of times. Nonetheless, after Vazquez bought another ticket out of New York with an abysmal performance last season, PECOTA foresees a rebound to a sub-4.00 ERA and a healthy strikeout rate in Florida. Meanwhile, 2010 super-slugger Jose Bautista is projected to shed nearly half of his homers (which would still leave him with his second-best season to date).
In the case of each player, we can do more than simply throw up our hands and attribute last year's surprising performance to divine dice rolls. Vazquez experienced a sizeable velocity drop (whose effects can be quantified); Bautista made well-publicized changes to his stance and swing. PECOTA doesn’t know those things, but you and I do, even though we might not know their precise significance. Given the increasing granularity of baseball data capture, perhaps the passage of time and future additions to PECOTA’s code will make it possible to adjust the forecasts not only according to what numbers were produced, but to a greater degree, how they were produced. For now, feel free to indulge your inner PECOTAs, but remember to forecast responsibly.
Thanks to Colin Wyers for research assistance.