CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

Futures Guide 2014 is Now Available in Paperback and Three E-book Formats.

Premium and Super Premium Subscribers Get a 20% Discount at MLB.tv!

<< Previous Article
Future Shock: Cincinna... (02/07)
<< Previous Column
Reintroducing PECOTA: ... (10/01)
Next Column >>
Reintroducing PECOTA: ... (02/08)
Next Article >>
Premium Article Prospectus Hit and Run... (02/08)

February 7, 2011

Reintroducing PECOTA

They're Here!

by Colin Wyers

And here we are–the release of the 2011 PECOTAs.

While I have your attention, I’d like to say a few words about the production of the PECOTAs this year. I guess I don’t have to tell you that I’m filling some pretty big shoes here–Nate Silver is probably the most famous sabermetrician not named Bill James, and PECOTA is where Nate made his biggest mark in our community. And so I’m building on his work–and work by people like Clay Davenport and Gary Huckabay, too. I am, as they say, standing on the shoulders of giants. (In big shoes, apparently–this is what I get for mixing my metaphors.) So I owe them, and others I’ve probably neglected to mention, my deepest thanks.

But even with all that, I wouldn’t have gotten this far without a lot of help. I couldn’t have accomplished what I’ve done without the help of everyone here at Baseball Prospectus, who have really given me all the support I could ask for. But I want to thank a few people especially–it’s a fine line to walk, as if I list too few I risk upsetting someone unfairly left off, and if I list too many, you won’t finish reading the list. So with that in mind–very special thanks to Rob McQuown, Mike Fast, Ben Lindbergh, and Steven Goldman. Gentlemen, take a bow, and everyone, please, give them a round of applause.

Now, then, on to business.

This is the first release of PECOTA, and as such will continue to undergo revisions through the remainder of the offseason. The program we use to generate the PECOTAs is continually evolving, and when we discover new ways to improve the forecasts, we'll make those changes and pass the updated forecasts on to you. We’ll also be updating periodically to keep up with players who switch teams.

In addition, we have several PECOTA features yet to roll out out–first will be the Depth Charts, which combine the PECOTA forecasts with estimates of a player’s role and playing time. Those should be ready a week from now, and will be available in two forms: the team Depth Chart pages, and the Player Forecast Manager (which is receiving some upgrades as well).

After that, we’ll be publishing the PECOTA cards, featuring perks like the percentiles and the ten-year forecasts. We’ll update you more as we get closer to that point.

We’re also revamping the cards to use the new Wins Above Replacement Player model we’ve developed. PECOTA has already been adapted to use the new WARP, so WARP baselines have shifted a bit from what you’re used to seeing. The biggest change is among relief pitchers, who take a major hit. Please keep this in mind as you review these forecasts. We know that many of you are relying on these forecasts for your fantasy teams, and we thought that it was better to get the forecasts out now rather than wait for when the entire site was ready to transition to new WARP.

Now, some of you may be asking, “How good are the PECOTAs this year?” Of course, we won’t know the answer for another eight months or so. But we can come up with an educated guess, if we make the assumption that there’s nothing special about predicting the 2011 season, and that a system that works over previous seasons will work in succeeding seasons.

In the course of producing the PECOTAs, we generate forecasts for every player who played from 1950 through 2010. These aren’t quite the same as the full PECOTAs–they are park-neutral, rather than being adjusted for the home park a player plays in. They are not age-adjusted. And for the most part, they do not reflect minor-league performance–we have major-league data for all of MLB history, but very little minor-league data. Still, they do represent a substantial portion of the PECOTA process.

It is time-prohibitive for us to generate full age curves for all of these historic forecasts, but we did adapt a simplified set of age adjustments for past purposes. These simplified PECOTA forecasts aren’t as accurate as the full PECOTAs, but they give us a chance to view how well PECOTA fares over a large swath of history.

There’s only one other projection system available and therefore able to be pitted against PECOTA for such a large part of baseball history: the Marcels, originally developed by Tom Tango. (The version we’re using in these tests was published by Jeff Sackmann.)

I “re-baselined” each forecast for each season in the test by subtracting the average forecast and adding in the average performance of the players forecasted (weighted by playing time, in both instances). Then I took a look at two tests–one is root mean square error, which tells us that 68% of forecasts were within that margin of error. The other is simply counting which forecast was closer to a player’s actual performance. Looking at offensive stats first:

P_OBP_RMSE

P_SLG_RMSE

M_OBP_RMSE

M_SLG_RMSE

P_OBP_SR

P_SLG_SR

0.036

0.066

0.036

0.066

56%

54%

In terms of RMSE, our simplified PECOTA is in a dead heat with the Marcels. (And again, PECOTA is giving Marcels an edge, as it is forecasting everyone for a neutral park; the Marcels make no park adjustments, but most players do not switch teams or parks between seasons.) In terms of “success rate” (in other words, the percentage of head-to-head projection matchups "won"), PECOTA has a slight edge.

Now, for pitching:

P_ERA_RMSE

M_ERA_RMSE

P_ERA_SR

1.22

1.22

48%

Again, RMSE shows a dead heat. In terms of success rate, the Marcels have a slight edge over the simplified PECOTAs.

Like I said, this sort of testing emphasizes breadth rather than depth–PECOTA has foregone several of its advantages, like park adjustments and minor-league data. And yet it’s still producing accurate forecasts.

Now, just as a reminder–PECOTAs are available only to our subscribers (Premium and Fantasy) who are signed up for a whole year. If you haven’t already subscribed, you can do so here. Doing so doesn’t just get you access to PECOTA, but also to our fantasy tools like the Player Forecast Manager and Team Tracker, as well as exclusive access to some of the best baseball writing out there. If you’re already a subscriber, thank you for your support, and I truly hope that you all enjoy what we do as much as we enjoy doing it. I am continually amazed at how intelligent and knowledgeable our readers are, and I really do think that the people who read BP are among the best baseball fans I’ve ever known.

That’s all I've got–have fun, folks.

Colin Wyers is an author of Baseball Prospectus. 
Click here to see Colin's other articles. You can contact Colin by clicking here

256 comments have been left for this article.

<< Previous Article
Future Shock: Cincinna... (02/07)
<< Previous Column
Reintroducing PECOTA: ... (10/01)
Next Column >>
Reintroducing PECOTA: ... (02/08)
Next Article >>
Premium Article Prospectus Hit and Run... (02/08)

RECENTLY AT BASEBALL PROSPECTUS
Premium Article The View from the Loge Level: Ode to Joe
Premium Article What You Need to Know: July 25, 2014
Premium Article Transaction Analysis: Kendrys Redux
Premium Article Skewed Left: The Mismatched Incentives of Dr...
Premium Article Notes from the Field: What's Wrong With Appe...
Premium Article The Call-Up: Chris Taylor
Premium Article Raising Aces: Shelby Miller's Fall From Grac...

MORE FROM FEBRUARY 7, 2011
Future Shock: Cincinnati Reds Top 11 Prospec...
The Payoff Pitch: Whose Money Is It, Anyway?...
Fantasy Beat: BP Scoresheet Early Draft Prep
The Week in Quotes: January 31-February 6

MORE BY COLIN WYERS
2011-02-21 - BP Unfiltered: Depth Charts and PFM Open
2011-02-18 - Premium Article Manufactured Runs: Projecting Pujols
2011-02-16 - Premium Article Prospectus Preview: AL West 2011 Preseason P...
2011-02-07 - Reintroducing PECOTA: They're Here!
2011-02-04 - BP Unfiltered: A few other quick words on PE...
2011-02-04 - BP Unfiltered: PECOTA looks at Jimmy Ballgam...
2011-02-02 - BP Unfiltered: A few quick words about PECOT...
More...

MORE REINTRODUCING PECOTA
2012-03-08 - Reintroducing PECOTA: House of Cards
2012-02-08 - Reintroducing PECOTA: The Weighting is the H...
2011-02-07 - Reintroducing PECOTA: They're Here!
2010-10-01 - Reintroducing PECOTA: The Seven Percent Solu...
2010-09-30 - Reintroducing PECOTA: Aches and Pains
2010-09-29 - Reintroducing PECOTA: The Hits Just Keep On ...
More...

INCOMING ARTICLE LINKS
2011-02-10 - Premium Article Prospectus Hit and Run: The Five Disappointm...
2011-02-08 - Transaction of the Day: Vladimir Guerrero
2011-02-08 - Fantasy Article Fantasy Beat: PECOTA Projected Bargains