Sorry it took so long, but this turned out to be a major update in a lot of ways - particularly on the pitchers.
If there was one type of comment that repeated frequently in the threads, it was that the pitchers looked way too pessimistic in their projections - far more so than the hitters. My first instinct was to simply accept it as PECOTA being PECOTA - the normal tendency of the group is down. There are a lot more different ways, and far less effort involved, in making a major league player worse than there are to make him better.
On further study, it may still be that, and it is possible that I’m chasing an idea the wrong way, but it is the best I’ve come up with for now. There is really only one significant change that went into this projection set from the last one - the pitchers are shown based on their 75% score, not their 50% score. I’m still trying to figure out the exact whys, but when I reran the 2008 and 2009 projections using the new PECOTA program, the best results for major league pitchers came from using a projection around the 75th percentile; using the 50th percentile resulted in large biases towards worse performance, exactly as noted in the player comments for the previous run. This effect was not found in the hitters; the best results for them did come from using the 50th percentile.
I believe that this is the result of a selection effect - that pitchers who do better than their true expectation are the ones who actually pitch in the majors; any slippage quickly results in reduced playing time or demotion to the minors (except for teams with no reserves, like last year’s Brewer rotation). While I would expect that to be true for hitters as well, I think the combination of lower margins (pitchers don’t separate from each other as widely as hitters do) and higher injury rates makes them more susceptible to the bias, and that’s what PECOTA is picking up. My worry is that while running the program in this configuration will make for better results in the major league forecasts, it will create undue optimism for a lot of borderline and minor league players - that by focusing on accuracy for a select subset (major league pitchers), I’ll degrade the accuracy for the group as a whole (all Organized Baseball pitchers). The problem does seem to have appeared with the 2009 PECOTAs; the main change between 2008 and 2009 was the inclusion of a lot more minor league players in the database, about 15 years worth.
But it does make a big difference in the performance of the model for major leaguers. Here are some results from my error testing. In each case, I have pro-rated the PECOTA projection to the pitcher’s actual innings pitched, and showing the root-mean-square error of the given component. So if I projected a 4.50 ERA, and the pitcher really had a 4.00 ERA in 180 innings, my error is 10 runs. The selection group was about 300 pitchers, chosen because they had forecasts from PECOTA and a number of other projection systems (like ZIPS and CHONE). The “new pecota” was run with all 2009 data removed from the system - it used no information that wasn’t available in February 2009.
(from 2/14/09 weighted means sheet)
2009 new PECOTA
Running at 75% produces a dramatic improvement in hits and earned runs, a moderate improvement in walks, negligible improvement in HR, and a negligible worsening in strikeouts - and very narrowly missing a clean sweep of all categories.
Running just against the February 2008 version of the PECOTAs for the 2008 season, we get
This was based on 600 pitchers, who averaged a lower inning total, which is why the overall scale of the numbers comes down, but we can see that the “new PECOTA”, running at 75%, follows the same pattern as in 2009 - big improvements in H and ER, moderate improvement in BB, negligible differences in SO and HR.
It is natural to ask, then, how the hitters compare to the old PECOTA. For 2009, about 400 players, and scaling to plate appearances, the errors by each category are
With the hitters, we have an unfortunate step back in BB and SO, but improvements ranging from small to large in every other category. I’m still looking to see if I can improve those”non-event” scores, but I’ll happily take the gains.
Now, as to the depth charts themselves, I think I’ve gotten all of the free agents purged from them. A lot of the odd minor leaguers showing up on the depth charts are also gone - most of those were listed because they were the next best option the teams appeared to have in their own system, at the time I did the depth chart; as teams have signed more players, the need to reach so far down in the minors has diminished, but I’m still concerned about a team like the Twins, with no one behind Span that I’d trust in CF for more than a couple of games. Please do keep in mind that these charts are trying to look at the entire season, not just the Opening Day roster, and they are working on the assumption that injuries will happen. When they do, someone will get called up - someone who might have no shot at the major leagues when everyone is healthy. Some of the playing time projections that seem low are because PECOTA expects a player to suck, and I have a hard time believing a team will stick with him all season, regardless of how they declared him a starter early on. That will become even more true as spring training gets underway, and someone “wins” a job on the basis of a .387 spring over someone with a longer, better history of performance.
Lineup order is generally not that big a deal for me - they are notoriously changeable, and it is primarily to enhance or decrease the PA of players expected to be largely high or low in the order. When I do see a manager make specific statements about how he wants, I will work them in.
Unfortunately, the changes that came in to PECOTA here has set back the production of the individual player cards. I do think the hitter card is down to one bug that I can fix today, and that the pitcher card format issues can be done by Monday if I don’t have to shovel any more snow (I live just south of the snowiest major city in the USso far this year. Those are entirely on me; I keep changing things to try and get them right.