Great article. One key point that I think needs to be looked at is that there is a large percentage of people that right out of the gate despise statistics because they dont understand or dont want to understand the meaning derived from the math. Scouts and coaches are given the benefit of the doubt out of the gate while stats continually need to be vetted. In the words of the mlb network resident genius Harold Reynolds "I just know that that aint right."

As a stats guy myself (in another field), I cringe when I see error variance attributed to "luck"- and this does happen a lot in sabermetric analysis (and most other walks of life, too).

Error variance is something we failed to predict, some of it is randomness or luck (chance would be a better word than luck, though), but it is mostly due to things we neglected to consider in our model.

I would also love to see more variance built into sabermetric numbers. For example, given that we do (pretty much) know how big a sample we need for certain stats to stabilize) we can build in confidence intervals or something around our stats.

Then we can have a debate on candidate A with 6.2(+-.5) WAR vs. candidate B with 5.9(.3), or with 1 year defense stats, something like 2.0(1.6). We kinda do this with PECTOTA's quartile projections, I'd love to see this with backwards looking stats, too. It would keep us griunded in the fact that stats often have a lot of error built in.

There's a great section in the Francona book that deals with this very issue from a manager's perspective:

"...the problem was that the number would change as we played. On Thursday Mike Lowell could be a good fit to play, but on Friday he could be a bad fit because of what happened to the numbers Thursday. It was a little too fluid for me."

At what point do you reach a level of statistical significance where that Thursday/Friday dynamic isn't the case?

Similarly, if you flip a coin and get heads seven times in a row, are the odds that you're going to get tails on the eighth flip 50/50? Perhaps, if you view the eighth flip as part of a series. But as an independent act, it's 50/50 just like the rest.

Good food for thought here...

One of the most interesting things I learned from following the Orioles last year was considering what (besides the knee jerk reaction of "luck")are the characteristics of a team that wins a disproportionate number of 1 run games.

It reminds me of the refinement of the Voros theory that pitchers don't influence the results of balls in play (but on further study some do, at the edges).

The conclusion I reached from following the Orioles was that there are good reasons to be skeptical of the ability of a team that wins a dispropotionate number of their 1 run games to be able to continue that ability, but the ones that have a really good bullpen are the ones that are most likely to be able to sustain success in 1 run games.

Nice article that brings up a salient point.

Why are r^2 so rarely reporte in sabermetric work - is it a sort of early convention that's persisted?

I think the biggest issue with Sabermetrics is confusing better with good. There are plenty of examples of Sabermetric models that have value and provide interesting levels of analysis. The biggest problem I see with Sabrematricians is their extrapolation of something demonstrably better into being considered right. Those are different things. We can say WAR has value and is informative; which isn't the same thing as saying WAR correctly values player's contributions.

"...we sneakily start with the assumption that our models are correct."

The curse of economics. Although the upside is that when sabermetric models are wrong, they aren't catastrophically wrong! As in the case of economics.

Great article. One of the big points that was made at the GM Panel at the SABR Analytics conference (either by Jed Hoyer or Rick Hahn, I forget which) is that if you give some information to a player and it doesn't work, he will never trust you again. It doesn't matter that you would be right 99.5% of the time. So, information presented to players needs to be presented correctly, even if it is "right".

http://sabr.org/latest/2013-sabr-analytics-general-managers-panel

Great article. I'm an accountant and part of my job is to prepare budgets. I love doing it because if I'm right I look good. If I am wrong, then I make a comment that I can't predict the future or I would have hit the lottery years ago. I never tell my boss that my model was incorrect.

Well done. I think one can never write about confirmation bias enough -- that belief is so ingrained in people. This article feels like a continuation of your essay at the back of the 2013 annual.

One question I haven't solved: We'd all agree that judging the process instead of the results is most important. And a good manager hires people "smarter" than him, including, but not limited to, a position like an analyst. But if that manager has to resist judging the results, and he's unable to fully grasp the methods being used, how is he to evaluate the researcher?

Of course, I'm really limiting things to the quality of research itself, not communication or interpersonal abilities.