Although we’d like to confirm these results independently, Chone Smith, a poster at Baseball Think Factory, has run a comparison of projection systems in 2006 and PECOTA has come out on top, with a huge lead in position player projections and a very narrow second place in pitcher projections. You can find more details here.
Make no mistake: the other projection systems are improving. Dan Szymbroski’s ZIPS projections have come an awful long way. I like the work that the BIS guys are doing. Top Tippett’s projections for Diamond Mind, not referenced here, have always been very good.
So we need to make PECOTA a moving target, and it is improving too. The league difficulty adjustments that we’ve built in this year correct a longstanding pet peeve of mine. We’ve revised the formula for the starter/relief pitcher adjustment, which was producing a few funny results for middle relievers last year. And we’re always catching little things in the extensive beta testing that we do each year, things that can help you to gain an extra point in your roto league here and there. There’s really no ninja magic that give the PECOTAs a leg up on their competition — it’s thousands of man hours of hard work.