Reading through the comments of yesterday’s announcement that the PECOTA projections have been released, it is evident that there is a lot of concern over several aspects of the data, ranging from the projected standings to individual quirks. We understand and appreciate that this reflects a lot of passion for what we do here at Baseball Prospectus. To be blunt: we messed up, and are working to fix the issues.
One issue involves the run environment: individual player projections do not match up with the run totals on the projected standings.
Another problem revolves around BABIP, as defense was being double-counted (double-counted).
These and other issues are being worked on and we hope to have an update provided by the end of the day, to unmess up. Please stay tuned for further updates.
Thanks, Eric. I know it's tough to balance the desire to be perfect with the desire to get things out to people in a timely fashion. I appreciate the communication.
BTW, I was also noticing how the NL seem to have way more top-level offensive performers (much higher VORPs at the top end) than the AL. Is that a PECOTA hiccup, or is that just the way the cookie crumbles this year?
It's a little bit ridiculous that we've become so accustomed to free content online that we assume any paid content must be perfect. BP users totally get what they pay for -- a totally unique resource within the baseball world. We pay for staff members like Eric who feel awful and embarrassed when mistakes are made. He'd feel the same way regardless of whether or not anyone said anything.
It's certainly your right as a paying customer to complain, and my right, subsequently, to criticize those complaints. Just my two cents.
Your response doesn't resonate with me. I would understand it better if I were complaining about free content. This is no different than going to the supermarket, buying a gallon of milk, and finding the expiration date was that same day.
Sure, you can go back, complain, and get another gallon and basically be right where you need to be - but it doesn't make it right that they left bad milk on the shelf.
I also disagree about BP's uniqueness - there are plenty of other internet outlets to get similar info. BP remains among the best of course!
I completely agree with sweeney. I feel as though I get way more than I pay for here, and that stuff like PECOTA is something that we as users should be critiquing, and BP should be working on, which is exactly what happens.
I'd rather have a site put something out, even if it isn't perfect, as long as they are willing to accept criticism. If they are fearful of response so much that it limits creativity, then we all lose.
Thanks again for all you do BP, it is well worth my forty some odd dollars.
You must not buy PC games. They constantly are released with bugs, sometimes to the point where the program is unplayable. I agree that it does leave a bad taste in your mouth, but what is even worse is when the company does not fix it. At least the fine folks here at BP are working immediately to rectify these problems...THAT is what you pay for.
If you were the one doing the work, you would probably appreciate a little understanding. Had PECOTA been only a few weeks later, there would have been a riot on these same message boards, even if there were less errors. They're balancing a fine line between pissing off impatient people and pissing off people that just live to take others down a few pegs. Rough job. I'm jealous though.
As one who typically begs that the sheets be put up in late January and is typically rewarded in a very short time frame, I can't complain about the issues. Glad to see they are being fixed.
One request: while it is informative to see the playing time adjusted version, the raw version seems more valuable (at least to me). I can mentally adjust for anticipated playing time but knowing that a guy has a good chance to breakout if he gets a chance - I can't see that very clearly.
At 7:45 Central time, I notice a 1-29 update to the PFM and Depth Charts. I, for one, would like to say that I think BP deserves a pat on the back for
a) Admitting frankly and quickly that they screwed up without mincing words.
b) Making it right with due speed.
Ya know, there are other projection systems out there that have a good deal of merit behind them (ZIPS and CHONE to name two), but I think when you have historically been an industry leader you tend to take far too many 'shoot from the hip' pop-shots when you screw up. I think BP is undeserving of such criticism at times, and for what's it's worth I wholeheartedly agree with fmcsweeney's post above.
The point is: If you screw up.. Admit it...Fix it... Move on. It appears BP has done that in a timely fashion.
My last post on the topic - you can have last word after this and I wont' say a thing :)
BP is great. I have been reading them for years, before they went pay, and was among the first to pay the subscriber fee and haven't regretted a thing since.
The idea we can't complain about something is fairly ridiculous, it was even a mild complaint mixed with praise. And mistakes happen all the time to all of us, and it is not a big deal. The idea we shouldn't talk about leaves a bad taste in my mouth.
My only issue is that this was easily preventable with 10 minutes of scrutiny of the data before going live with it. It's just carelessness; and that happens to be something I personally have a low tolerance for.
You may be missing something important. This group of users/readers is so intensely interested in the projections, and so knowledgeable about what looks right or wrong about just about every player, that BP's posting of what's in essence a beta version is just fine.
We all give the data a scan that no single creator of the data can do.
This is a productive process, actually. Post the beta, get user feedback, correct the data, and move on. What was the delay? 1 day? Not bad at all.
I have to agree. I love BP, and can't wait EVERY DAY to see new articles. I value Pecota for fantasy reasons, but I don't see why issues can't be resolved before they are uploaded. I don't mind that there are issues, but at the same time, i wonder how there will be mid season updates when the first projections come out so screwy.
I just re-downloaded, and Ortiz is +12 at 1B, not 3B.
However, the +12 isn't realistic. I suspect this comes from a small sample size (as he mostly DHed) projected over 150 games. First I would Marcel his last three seasons, then do runs as a rate (runs per chance) and regress it to league average before extrapolating over a season's worth of chances.
Big YAY for a HOWEID for each player. Could you PLEASE PLEASE PLEASE include the same HOWEID on the PFM. That would make it much easier to match up data for fantasy worksheet purposes.
Big boo for HOWEID for comparables (and only one comp?).
Seems to be that if you set up the system using X number of RP's and X number of SP's (instead of just X number of P's), all the closers are all left off the list because they hold the Pos designation of CL.
You predict no NL team will win more than 86 games. You also predict no team will lose more than 94. Has either of these events ever happened before in a full season?
Your predictions suggest the closest, most competitive season I can remember with no "great" or "miserable" teams.
Typically to be a really good team, you need to have some very good players, and some really good fortune as well. You need to keep your regulars healthy and productive and on the field all at once. You need good seasons from your bench players and your bullpen guys as well.
Yes, there's a lot of that that's skill. But not all of that. And so what we do is project the skill, but not the "luck." The same on the bottom end - no reasonable analyst should or would have projected the Mets to lost that many games. Yes, the Mets were not a particularly good team on paper, but the incredible run of misfortune that befell them isn't exactly something anyone could see in advance.
So what you end up with is a median forecast - one if all the breaks don't go your way, but all the breaks don't go against you. And it's further comprised of the median estimates of each player on the team - one between the potential for a career year and a season-long slump.
So, yes, you're going to see a tighter spread of projected standings than you would observe in the real end-of-season standings. That's because there's somethings we simply can't predict. You still have to play the games - we're talking about probability, not destiny.
Some people are going to say that these sorts of projections are useless without perfect clairvoyance. Well, we don't have it and we don't pretend to. I leave those sorts of claims of perfection to the tarrot card readers and the mystics; you can check their track records and see what they're worth.
As for us - we can tell you that typically a good projection will typically come within nine wins of the team's final-season record, give or take. That's a lot of wins. That just means there's still baseball to be played. I'm always going to work on doing better projections. But I don't know if I'd want perfect projections even if I could have them. This way you have to actually play the games. And I don't know that I'd want it any other way.
Thanks for your explanation. I (partially) understand your methodology and margin for error. I have several concerns about your projections but everyone can have differences of opinions. In searching for an answer to my question about team wins and losses I found:
http://www.baseball-reference.com/leagues/NL/
I believe that in a full season of 162 games there has never been a year where there was a league leading win total of 86 or less and never a season where the worst team lost 94 or fewer.
I think "good" teams like Philadelphia are under projected and "poor" teams like Washington are over projected. Did each project member review each team or were there individuals researching the various clubs?
You are suggesting a historic, competitive season and I will try to be the first to congratulate you after 2010 season if you are correct on either count.
I'm not defending Baseball Prospectus' particular set of predictions.
However, it is certainly reasonable to predict that no specific named team will finish with > 86 wins or < 94 wins even though the chances of that actually happening for the league as a whole are quite slim. Even if the predictions accurately pegged each team's true talent level, we will expect each team's actual win totals to be dispersed around their true talent levels so that the actual outcomes will be more extreme than 100% accurate predictions of what their expected wins would be.
I hope I'm writing that in a manner that can be understood.
Thanks for the patience. Not being as sophisticated mathematically (or intellectually) as others I have to occasionally depend on non scientific smell tests. The original projections failed for me (and many other subscribers)in numerous cases. I guess I am more satisfied with the projections now that some of them have changed substantially, such as Nats from 82 to 76 wins.
No, BP is not projecting a historic, competitive season. BP's forecasts acknowledge the reality that the spread of expected performance will generally be less than the spread of actual performance.
There seems to be a recurring issue from prior years in that the league positional parameters are not reflected in the player report.
As an example; in a 7 team AL league with one catcher per team only 3 catchers show up on the report. Similar issues with other positions.
This quirk causes an incomplete player list and also affects the dollar allocations as some positions wind up with too much $$ allocated and some positions (catcher for example)get too little $$.
In the Positional Adjustment setting, make sure to select level 1 to ensure that all positions are represented like you want. In your example, there should be 7 catchers after changing this setting.
I understand that in terms of minimizing error (in the RMS sense), a healthy amount of regression to the mean needs to be applied. But to my way of thinking, there needs to be an additional constraint to projections. The leader-board of a projection has to resemble in structure the actual final leader board. So if in a given year we have 20-25 hitters with a VORP of greater than 50, the PECOTA leader-board needs to have 20-25 hitters with VORP of greater than 50. In this year’s board we have only 7. To use an analogy, this is like projecting at a team level a total number of wins and losses less than 162. Or like projecting an election result with less than 538 electoral votes doled out.
So I view the issue of projection as trying to fit the best result in a framework that already exists. In other words, to decide which players will perform at the various bins of performance.
That analogy isn't quite right. You're comparing projecting the spread of results to the total of results.
To see the difference, look at the electoral votes case. Pretend both candidates are equally liked in every single state. Every single state has a 50% chance of going for Candidate A or B. PECOTA is essentially trying to project talent level. It would say each candidate is projected to get 269 votes. The average spread is going to be much larger. I'm not exactly sure, but on average I'd assume one candidate would get at least 300, just by random distribution of states. Saying that the predicted electoral leaderboard has to resemble the real leaderboard isn't right. The true talent level of each candidate is 269.
This is also seen well with the win totals. What pecota is doing isn't like predicting fewer wins/losses than 2430, it's projecting less spread. Again, if every every team had around a 50% chance to win any individual game. There would be, on average, two teams with 90 or more wins and two teams with 70 or fewer wins. But you shouldn't want PECOTA to project based on the estimated spread. Every team has a 80-82 win talent level. Projecting by spread would be like saying that the 50.6% (82) talent team is going to win 93 and the 49.4 (80) talent team is going to win 67. Sure those two teams are the most likely to reach those totals, but it's only slightly less likely that another team does it.
This is the same thing as Players. Yes, there are going to be 20-25 hitters with 50 VORP, but you should want PECOTA to project talent level, not spread.
Agree with JesseB. I find the current weighted mean projections very helpful precisely because they are sobering - for instance, that young hotshots Justin Upton and Gordon Beckham may well regress a bit from their performances last year. For upside, I look at the 75th percentile projections (and so am eager to see those).
If I'd overstate my claim, what I'm trying to say is that PECOTA is providing the right answer to the wrong question. If the question is "What is our best estimate of player's A true talent level in 2010?" or "What projection would provide us with the least error?", then PECOTA and other such systems are providing the best answers. I believe that the more interesting question is indeed to project how the 2010 leader boards will end up looking. PECOTA is not providing us with this answer. To project the leader boards we need to assign predetermined values to different players.
I think that a projection whose structure (e.g standard deviation) is dramatically off from any plausible final result, is missing something vital.
When I'm referring to leaderboards, I'm not only referring to who finishes 1st or 3rd. I'm mainly concerned with the distribution and range of the results. In other words, I'm very troubled by the fact that the VORP/WARP/EQA of the top 50 players (regardless of their identity!!) in PECOTA and many other projection systems is very different than what actually occurs.
Well, clearly there's a fundamental disagreement of opinion here about the objectives of forecasting.
To do what you're suggesting, one would need to pick before the season which players will outperform their means and which will underperform their means. Any single deterministic pick is merely a guess. To derive any value from it you'd need to do it stochastically, and express the results in terms of probabilities; this may be difficult for many people to wrap their minds around.
From my perspective, an interesting supplement would be for BP to do something like this. Using PECOTA (but the entire spread of projections, not just the weighted means), publish probability estimates of how likely individual players are to win the HR title, BA title, etc.,; of how likely it is that somebody will hit 50 HR, hit .350, etc. Or, at the team level, how likely it is that some team will win 100 games, how likely it is that winning 95 games won't be enough to win the AL East, etc.
I like what you are suggesting. Although I mentioned NL team wins and losses in an earlier post (and I now notice the revised projections for several teams have changed significantly) I am much more interested in individual performances - especially as discussed above.
My interest is in the probability of a significant change or a bold projection. I appreciated the Weiters projection from last year even though it was not matched by his performance. A quick review of the player projections this season seems to have few "surprises" on the up side and most of the established players below prior levels of production. I understand these may be the most likely outcomes but if there are few surprises then the Pecota service losses some value for me. I don't mind a wrong projection at the end of the season but if there is no "wow" or startling projections to cause me to do more research then I feel I am wasting time and energy.
If possible I would like to have access to a spreadsheet that could be manipulated by the subscriber to show Percota projections with a spread of scenarios. I think that would be helpful to a lot of fans.
Also, I have been a subscriber for many years and enjoy the site content. Please consider these as suggestions and not personal criticisms.
I actually would think that it would be an advantage to PECOTA to do so. My reasoning is as follows:
Respected analysts such as Tom Tango have shown that in 2009, the sophisticated analysis tools of PECOTA did not provide more accurate results than a projection system that is based on a very simple weighting function (AKA Marcels). Marcels has a very strong element of regression to the mean. In my proposal there would be less regression, with the unfortunate consequence of less accuracy, but we would gain the benefit of a more realistic distribution of results. I would think that there is a good chance that in such a scheme PECOTA would do better than the simpler systems.
You admit that by increasing the spread of projections, you'd be moving players' projections away from their true talent level. So in what way would PECOTA do better in this alternate way of doing things? You agree that it would get worse at accuracy measurements (RMSE as you mentioned above) and with questions like "How many total HRs will these 10 players hit next season?"
Aside from the above, here's something else that PECOTA would get worse at: Predicting team results. Right now, it's okay at predicting the season based on summing the true talent level value of each player. If you altered the projections, they wouldn't be true talent level estimates anymore. They would be skewed. A team that had players whose value was changed by the spreading would see a change in their forecast relative to teams with less change. But if PECOTA's moving away from true talent, it would only be getting worse.
No doubt alskor, and here I thought that working to determine true talent levels was an exercise in eliminating the noise of inherently unpredictable factors manifest in playing the games. Otherwise called luck.
Does all this mean that Broxtons SVs will be adjusted up from his projected 16 or does BP believe that Sherrill will actually get half the Dodgers saves at 16 too?
Over the years I have had the exact thoughts as Yadelman, and as much has I like to think I know about stats and probabilities it is just always hard for me to accept the usual prediction of 35-36 HRs for Adam Dunn when he hit 40 5 years in a row. It always seems that projection systems take the last year(or 3 years) production, lop-off 10-20% and bingo! you will be right about .700(approx.)...right around where the industry leaders in projections are. It is easy to predict vanilla...but what we want you to predict is Rocky Road!
You need to predict Outliers here people!...oh wait, what is the definiton of Outlier again??
(Of course I realize that to get the best overall pool of data, PECOTA is clearly amazing, but you HAVE to add to it your own ideas if you ever want to win a Fantasy league...you have to pick SOMEbody! Thats kind of what the Breakout and Attrition scores are for, but again, same problem. I bet Justin Upton has Breakout protential and Randy Winn has high Attrition...and I didn't even peak!)
This is a reply to JesseB (30925). (For some reason, on certain networks, the "Post reply" to specific posts doesn't work for me).
Before we answer the question "Is a projection accurate or not?", we need to answer the question "Is the projection valid?". If a projecttion does not resemble in it's structure the real world, then it is not valid. When I'm refering to "structure", I'm referring to the distribution of the results. Let's for the moment forget about the actual identity of the top 50 performers. The actual numbers put up by the top 50 (ehoever they are), should resemble the projected top 50 for a projection to be valid. Only then can we proceed to the next stage of projecting the actual results for individual players.
I realize that different people would look at this in different ways. I'm coming from the standpoint of a fan and I've never got into fantasy baseball, so this could be a reason for divergence.
Finally, and not related to the above, I really would like to commend BP on the new spirit of energy that their recent additions bring to the table.
I suppose we just want different questions answered by the projections. I want an answer to the question, What is each specific player's over under? His true talent level. You are getting answers to the questions: (1) Without regard to particular players, what is the most likely leaderboard for next season and (2) which players are most to fill each slot (a player can only assume one slot)?
As a fan, I'm less interested in player leaderboards but would rather know what each player is likely to contribute to his team. By distributing the players around a leaderboard, you don't get an answer to that question. A player whose true talent is a 4 win level, is most likely to contribute 4 wins to my team, not 6 as it might seem if the projections were spread out.
As to your last note. I agree, I really do like the new talent that BP has added. It makes the place feel new again.
http://www.baseballnotebook.com/ Read some of the articles about forecasting from the archives section at the bottom, specifically "Projection sets that shouldn't add up" and "The quality of a forecast set." If I predict that Willy Taveras is going to hit 18 home runs next season in 300 at bats, my prediction will be an outlier. If that's what you want, have at it. I'd rather get a set with a lowest margin of error per projection.
They projected Matt Wieters to be a stud last year because that's what the numbers were telling them, not because they wanted to make an outlier projection.
If they don't feel that there are that many players going to take a step forward, I'd rather them not create some and make us figure out which ones they really believe in and which ones are bogus.
69 comments have been left for this post.
Thanks, Eric. How many of the messy things may already be baked into the book? Or are these mainly on things that are only in the on-line PECOTA?
Very little. Most of the problem was actually coming from the depth chart routine, which runs in between the PECOTA and the website.