CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

<< Previous Article
Premium Article Kiss'Em Goodbye: Los A... (09/28)
<< Previous Column
Reintroducing PECOTA: ... (09/27)
Next Column >>
Reintroducing PECOTA: ... (09/29)
Next Article >>
Premium Article On the Beat: Eyeing Th... (09/29)

September 28, 2010

Reintroducing PECOTA

Whatever Happened to the Man of Tomorrow?

by Colin Wyers

Yesterday, Dave Pease went ahead and talked about the process of generating the PECOTA forecasts in the past. Now I’m here to talk about how PECOTA has fared, and where PECOTA is headed.

First, let’s look at the hitter forecasts for this season. We have four test candidates:

There are a lot of other forecasting systems in the wild; I chose to look at Marcel and CHONE in comparison because they’ve fared well historically and they have sound underpinnings.

Let’s look at how well each system forecasted the overall offensive levels of the league as a whole. We’ll use OPS, since it’s a “good enough” offensive estimate for the sort of study we’re doing, and it’s typically calculated by every forecaster, so it’s a very transparent way to compare systems. Looking only at players in common between the four projection sets, the weighted average of OBP, SLG, and OPS for each system:

 

OBP

SLG

OPS

Obs.

0.332

0.414

0.746

PECOTA

0.342

0.427

0.769

PECOTA (book)

0.344

0.430

0.774

Marcels

0.340

0.437

0.777

CHONE

0.339

0.430

0.769

This was a down year for offense on the whole; the most recent PECOTAs and CHONE forecasts were a shade closer on projecting the league offensive environment, but even then there were .023 points of OPS between them and the observed results.

So I adjusted each set of forecasts to line up with the lower offensive environment, and looked at the root mean square error of each forecast from the observed result, weighted by the number of plate appearances that player had. (Root mean square error represents the standard error—in other words, 68 percent of the time, you should expect outcomes to occur within that distance from the forecast.) I now present the most boring chart I have ever had the honor of showing you:

 

OBP_RMSE

SLG_RMSE

OPS_RMSE

PECOTA

0.032

0.061

0.069

PECOTA (book)

0.032

0.062

0.069

Marcels

0.032

0.062

0.070

CHONE

0.032

0.061

0.069

That’s, uh, not a lot of difference. Strictly speaking, it’s no difference at all. Now let’s take a look at your standard rotisserie categories in fantasy baseball:

 

AVG_RMSE

R_RMSE

RBI_RMSE

HR_RMSE

SB_RMSE

PECOTA

0.030

22.5

22.3

6.9

5.9

PECOTA (book)

0.031

27.3

27.8

7.8

6.3

Marcels

0.032

23.7

24.2

6.8

6.1

CHONE

0.030

27.7

27.3

7.5

6.7

We see a bit more separation here, but not much. PECOTA and CHONE were tops at predicting batting average, the Marcels were best at predicting home runs, and PECOTA was tops at predicting runs scored, RBI, and stolen bases. (PECOTA looks better at these projections because we use our Depth Charts to model a player’s specific role and playing time—that has little to no impact on his OPS, but has a big effect on his counting stats and thus his fantasy value.)

Now, let’s look at pitchers, using ERA as our measurement. First, the predicted versus observed ERAs, as a group:

 

ERA

Obs.

4.06

PECOTA

4.12

PECOTA (book)

4.50

Marcels

4.18

CHONE

4.24

Again, the run environment this year was lower than any of these forecasts expected it to be. After adjusting the forecasts to account for the difference between expected and actual run environment, a look at how each forecasting system did:


ERA_RMSE

PECOTA

1.17

PECOTA (book)

1.21

Marcels

1.21

CHONE

1.18

We see a little more separation in the pitcher projections that we did in the hitter projections, but not a lot. Looking at the other roto categories for pitchers (CHONE doesn’t project saves, so it received no score in that category):

 

WHIP_RMSE

SV_RMSE

W_RMSE

SO_RMSE

PECOTA

0.20

4.7

3.2

33.6

PECOTA (book)

0.22

5.4

3.6

38.1

Marcels

0.36

5.1

3.6

36.5

CHONE

0.20

---

3.7

40.2

PECOTA runs the table here, tying CHONE in WHIP and leading in every other category. Again, PECOTA gets ahead here largely through the playing time forecasts driven by the Depth Charts.

These are not especially surprising results. This has been the state of forecasting for a while now—you have a pretty tight bunching of the advanced forecasting systems. (That’s what Nate Silver found back in ’07, for instance; even in ’04, hitting forecasts were pretty tightly bunched.) It’s been a while since PECOTA’s main competitors were a handful of fantasy touts and the Favorite Toy, and these results reflect that.

Now, of course, PECOTA has always done more than simply project a player’s basic stat line—we have a lot of other things going on, like the 10-year forecasts, the percentiles and the upside/downside ratings. It’s one of PECOTA’s main attractions, but it shouldn’t be its major downside as well.

One of the drawbacks of PECOTA’s additional complexity is simply how long it takes to produce forecasts. But that’s also a consequence of using Excel to generate them. We’ve cut that dependency a while ago, and are continuing to work to integrate PECOTA more with our other statistical offerings. That’s important to you because it means you get your forecasts sooner—because the word “fore” is of course a major component in forecasting.

But it’s also important for the accuracy of any individual forecast. I can take one hitter’s forecast and substitute any number of outlandish findings for him, and that on its own won’t move the needle on those RMSE figures I showed you—it takes a systemic problem affecting a lot of forecasts to show up in that sort of test.

And PECOTA is a computer program—essentially, a list of instructions. It will follow those instructions unerringly, regardless of whether those instructions are correct. It takes a human to write instructions for the computer to follow, and as we all know, humans make mistakes now and then.

Some of you may remember the PECOTA forecast for one Matt Wieters’ debut season. It struck a lot of people as being outlandish—I was certainly one of them. In this case, PECOTA was a victim of its own complexity—by taking so long to produce forecasts, there wasn’t enough time to properly proof the PECOTAs.

Minor league players are heavily dependent on methods of putting their stats in terms of expected major league performance; in this case, it was the Davenport Translations. The foundation of this process is a set of league difficulty ratings that establish how a league compares to the majors.

What seems to have happened is that, when spitting out translations for the two leagues that Wieters played in (and only those two leagues, mind you), those league difficulty factors were significantly inflated from what they should have been—the Eastern League was not only rated higher than the other two Double-A leagues, but above both Triple-A leagues as well. And the High-A Carolina League placed above both of the other Double-A leagues as well.

For most players, that wasn’t going to make a noticeable impact—very few players who are expected to be anywhere close to the majors have only one year of stats split between the Eastern and Carolina Leagues. But one is enough to produce the Wieters forecast.

For last year’s book, we had a fairly involved proofing process of the PECOTAs. (Notably we neglected to do that for the first run of the Depth Charts. There’s a lesson to be learned there, and we’ve learned it—proof everything before publishing.) That’s good, but we want to do better than that. So in addition to having humans proof the PECOTAs, we’re building a set of unit tests to run alongside the PECOTAs, testing each element to make sure it’s functioning properly. What this means is that the output of the PECOTAs are going to be tested at several steps along the process, to ensure that everything is functioning correctly.

We’re also utilizing these tests to make sure that when changes are made to the PECOTAs, that they’re actually improving the underlying accuracy of the product. And we will be notifying subscribers when changes are made to the methods between PECOTA updates.

Of course, PECOTA has had some infamously mistaken forecasts that wouldn’t have been caught regardless of the amount of proofing. Most of them have been of Ichiro Suzuki. Tomorrow, we’ll go ahead and address how PECOTA missed the boat on Ichiro, and what we’ve learned from those mistakes.

Colin Wyers is an author of Baseball Prospectus. 
Click here to see Colin's other articles. You can contact Colin by clicking here

Related Content:  PECOTA,  Forecasting,  The Process

44 comments have been left for this article.

<< Previous Article
Premium Article Kiss'Em Goodbye: Los A... (09/28)
<< Previous Column
Reintroducing PECOTA: ... (09/27)
Next Column >>
Reintroducing PECOTA: ... (09/29)
Next Article >>
Premium Article On the Beat: Eyeing Th... (09/29)

RECENTLY AT BASEBALL PROSPECTUS
Premium Article Monday Morning Ten Pack: September 15, 2014
Fantasy Article Deep Impact: Week 25
Daily League Strategy: Easy Peavy
Premium Article Transaction Analysis: Rays to the Top
Premium Article What You Need to Know: September 16, 2014
Prospectus Review: 'That Foul Tip Had Better...
Premium Article Baseball Therapy: Starving Young Royals, Bat...

MORE FROM SEPTEMBER 28, 2010
Premium Article Kiss'Em Goodbye: Los Angeles Angels
Premium Article Prospectus Perspective: Racing for the Cy
Premium Article Overthinking It: Closing in on the Bases
Premium Article Another Look: Baseball Digest

MORE BY COLIN WYERS
2010-10-01 - Reintroducing PECOTA: The Seven Percent Solu...
2010-09-30 - Reintroducing PECOTA: Aches and Pains
2010-09-29 - Reintroducing PECOTA: The Hits Just Keep On ...
2010-09-28 - Reintroducing PECOTA: Whatever Happened to t...
2010-09-26 - BP Unfiltered: Retiring the Sauce
2010-09-22 - Manufactured Runs: A Walk in the Park
2010-09-14 - Premium Article Prospectus Perspective: Missing the WAR
More...

MORE REINTRODUCING PECOTA
2010-10-01 - Reintroducing PECOTA: The Seven Percent Solu...
2010-09-30 - Reintroducing PECOTA: Aches and Pains
2010-09-29 - Reintroducing PECOTA: The Hits Just Keep On ...
2010-09-28 - Reintroducing PECOTA: Whatever Happened to t...
2010-09-27 - Reintroducing PECOTA: What A Long, Strange T...
More...