CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

World Series time! Enjoy Premium-level access to most features through the end of the Series!

<< Previous Article
Premium Article Prospectus Q&A: Trever... (03/12)
<< Previous Column
BP Unfiltered: BP Book... (03/12)
Next Column >>
BP Unfiltered: On Hall... (03/15)
Next Article >>
MLB 10 (03/12)

March 12, 2010

BP Unfiltered

PECOTA Update

by Dave Pease

2010 PECOTA Projection Analysis

As promised, here are the results of our preliminary analysis of the 2010 PECOTA projections.  We took the methodology for multiple builds of the projections (which will be easier to identify when we start using versioning, which we will do when we've got everything else cleaned up) and ran all the inputs for the 2009 version of PECOTA through it.  Then we compared those projections to the actual results of the 2009 season.

In all cases, lower numbers are better.

RMS Error results

System
Release Date
R
H
2B
3B
HR
RBI
BB
K
SB
CS
TOTAL
2009 PECOTA 3-28-2009 10.30 12.91 4.86 1.46 4.16 10.12 8.78 12.25 3.87 1.47 70.18
BP2010 PECOTA 1-5-2010 8.92 12.21 4.28 1.50 4.22 10.35 9.81 14.52 3.68 1.77 71.26
Current PECOTA 3-12-2010 9.12 14.16 4.30 1.50 4.43 9.98 9.09 10.42 3.68 1.50 68.18

Bias-adjusted RMS Error results

System
Release Date
R
H
2B
3B
HR
RBI
BB
K
SB
CS
TOTAL
2009 PECOTA 3-28-2009 8.10 9.50 4.16 1.46 4.06 8.94 7.73 9.73 3.72 1.43 58.53
BP2010 PECOTA 1-5-2010 7.11 8.68 4.10 1.46 3.93 8.46 7.83 10.07 3.67 1.50 56.81
Current PECOTA 3-12-2010 7.16 8.90 4.12 1.47 3.93 8.15 7.88 9.91 3.56 1.45 56.53

BP used the 2010 PECOTA projections as the basis of our LABR draft strategy this weekend.

PECOTA Ten-Year Forecasts and Hitter Cards

We have diagnosed the main problem with our ten-year forecasts as reported on the PECOTA beta hitter cardsNate Silver generated one set of comps in the original PECOTA process and used those comps to generate the long-term projections.  We were trying to re-generate new comps in year n+1 based on the player's career thus far with his projections for year n included, and repeating.  In addition to introducing considerable extra complication, this process generally created much less favorable long-term projections, as many readers noted.

We've adjusted the long-term projection process to work the way Nate originally designed it.  Once we've got everything stabilized and released, we'll be revisiting this topic.

Another problem we had was that a player projected to be out of baseball entirely had a value of zero, while if he was good enough to be projected to get some playing time but bad enough to be below average, he'd have a negative value.  For example, in a player's tenth percentile, he might be out of baseball entirely, returning 0 WARP, but in his fiftieth he performed well enough to stay in baseball, and rated -1.0 WARP in the playing time he was projected to get.  Because we didn't distinguish between a 0 WARP in baseball and a 0 WARP being out of baseball, this enabled some very weird results for players with this condition.  We've changed the process to differentiate between the two, and the values should now be much more reasonable.

The K bar in the player profile graph has been reversed, and now works as it did previously.

We still have a problem with some players having their higher percentile projections zeroed out, while the lower are filled--we've identified this internally as "the Koyie Hill problem".  This is a similar issue to the reversed projections problem above, and it'll be fixed this weekend.

PFM Settings Update

We've made some small changes to the position adjustment setting of the PFM:

"Level 0" is now "OFF"
"Level 1" is now "ON".

Levels 2 and 3 are disabled. We are looking into bringing those levels back into play--lets us know how much you used and miss them.

The default is now "ON", not OFF.

Depth Charts, Weighted Means Spreadsheet, and PFM

We've pushed two updates to the Depth Charts, Weighted Means Spreadsheet, and PFM this week.

The first one was sent out late March 9.  With this update, we attempted to address some of the issues people had noted with Depth Chart team statistics that Clay mentions in this post.  While this might have made the team projections more satisfactory, it quickly became apparent that it did so at the expense of the individual player projections, which are what the vast majority of subscribers are using PECOTA for this time of year.  If you downloaded a spreadsheet or ran PFM between the evening of March 9 and now, please do it again and use the individual player stats you get from the current data.

As an aside, the modifications that were made to the March 9 data were more-or-less applied on a league-wide basis, so draft order and dollar value from the PFM probably won't change very much.  The raw statistics that were predicted will be fairly different, though.

This morning, we pushed out another update which puts things largely back to their previous state.  The RMSE analysis above is run on this version of PECOTA.

Depth charts will be updated at least every other weekday through the start of the season.

Weighted Means versus Fiftieth Percentile

Traditionally, PECOTA has used weighted means projections for it's default projections--the Depth Charts, PFM, Weighted Means Spreadsheet (obviously), and player cards have all used or highlighted the weighted means stat line.

This year, we've been using the fiftieth percentile for these applications instead.  Until recently, we haven't had the weighted means at all for PECOTA in 2010.  We now have the weighted means in the cards--see the bottom row of the 2010 projections table--and the weighted means are being used for the ten-year projections in the cards.  Everywhere else, though, we're sticking with the fiftieth percentile projections for now, so you'll see the projections in the fiftieth percentile line of the cards match the Depth Charts, PFM, and Weighted Means Spreadsheet (which I realize means the spreadsheet is now misnamed).

PECOTA Pitcher Cards

We're working on the pitcher cards now.  They'll be available next week.

Dave Pease is an author of Baseball Prospectus. 
Click here to see Dave's other articles. You can contact Dave by clicking here

130 comments have been left for this article.

<< Previous Article
Premium Article Prospectus Q&A: Trever... (03/12)
<< Previous Column
BP Unfiltered: BP Book... (03/12)
Next Column >>
BP Unfiltered: On Hall... (03/15)
Next Article >>
MLB 10 (03/12)

RECENTLY AT BASEBALL PROSPECTUS
Minor League Update: Games of Thursday, Octo...
Pebble Hunting: An Illustrated Guide to the ...
Raising Aces: Ghosts of World Series Past
Playoff Prospectus: PECOTA Odds and Game 3 P...
Playoff Prospectus: A Decade of Planning an ...
Playoff Prospectus: Never-Wrong Ned?
Playoff Prospectus: PECOTA Odds and Game Fou...

MORE FROM MARCH 12, 2010
Premium Article Prospectus Hit and Run: Nomar and the Trinit...
Premium Article Transaction Analysis: Import/Export
MLB 10
MLB 10
Premium Article Prospectus Q&A: Trever Miller

MORE BY DAVE PEASE
2010-03-19 - Fantasy Beat: Weekly PECOTA Update
2010-03-12 - BP Unfiltered: PECOTA Update
2010-02-09 - Premium Article BP Unfiltered: The Untitled Joe Sheehan Book
2009-07-13 - Prospectus Idol: Round Eight - Finals Week
2009-07-05 - Prospectus Idol: Round Seven - Media Week
More...

MORE BP UNFILTERED
2010-03-20 - BP Unfiltered: 3/19 Pecota Update: Weighted ...
2010-03-18 - BP Unfiltered: Chicago DePaul Booksigning
2010-03-15 - BP Unfiltered: On Halladay
2010-03-12 - BP Unfiltered: PECOTA Update
2010-03-12 - BP Unfiltered: BP Book Tour: Where Were You ...
2010-03-10 - BP Unfiltered: SABR Offers Discount to BP Re...
2010-03-10 - BP Unfiltered: Yankee Stadium Coming Down
More...