keyboard_arrow_uptop

We recently went to great lengths to develop the new version of SS/SIM, the Scoresheet simulation value metric here at Baseball Prospectus. As fellow Scoresheet owners, we realize that optimally these valuations would have been complete several weeks ago. Unfortunately, the original formulation of SS/SIM was not available to us this year, and our replacement took more time to develop than we had anticipated. We set out to create an all-new run-based value metric for Scoresheet team owners to use in player valuation. This time consuming effort was spearheaded by BP staffers with Scoresheet experience, and included research into the Scoresheet engine, talking with Jeff and Dave Barton, and consulting with Scoresheet experts for feedback.

Development of SS/SIM for both hitters and pitchers starts with the Scoresheet definition of replacement level, namely the awful Triple-A players that are used when you run out of playing time in the Scoresheet simulation. From there we have captured the most important aspects of the Scoresheet simulation and built them into the SS/SIM methodology to produce a single metric for Scoresheet owners. These adjustments seek to raise the effective replacement level and account for the differences between MLB and Scoresheet. [edit: BJM 2011-03-28]

Hitters:

The offensive formulation builds a value estimate from the replacement level based on expected offensive output. The offensive valuation is built from a positionally adjusted perspective that also emphasizes overall offensive output, supplemented with a defensive adjustment that accounts for each player's Scoresheet defensive eligibility and range rating, as well as a penalty for expected errors. 

The results of this process are valuations that use a very low replacement level–note that Player(AAA) is remarkably low in value compared to the replacement level used for metrics designed for evaluating MLB players. SS/SIM as a hitter metric provides a complete depiction of a player's value in Scoresheet because it uses replacement value specific to Scoresheet and incorporates the latest research on Scoresheet defensive range value and appropriate valuation of errors to form a comprehensive view of the player.

PItchers:

The pitching formulation is a bit more complicated. Pitcher valuation is based around projected IP and ERA because the foundation for value in Scoresheet for pitchers is their ERA, and because pitchers in Scoresheet are basically limited to their real life playing time. Because we expect Scoresheet owners to alter pitcher usage based on ERA, each pitcher is projected to throw a certain percentage of his real life innings for the Scoresheet fantasy team. The projection for how many of his real life innings will be utilized is separated by starters and relievers based on the expectation that Scoresheet managers will maximize usage of their relief pitchers because they have lower ERAs, which is the converse of real life MLB usage. As a result of this usage pattern, the ratio of SP IP to RP IP is lower in Scoresheet than MLB, and our methodology reflects that difference as well. We also put a premium on pitchers whose ERA is better than league average to account for the importance of ERA and the multiplicative nature of actual run production in the simulation. This adjustment is proportional to the extent that a pitcher is better than replacement level, so it impacts premium starters the most, but it also increases the importance of very good relievers as well. Given the expectation for how many innings a pitcher would pitch for his Scoresheet team, we can derive his SS/SIM by comparing his forecast for runs allowed over that time to what the Pitcher(AAA) would allow.

The results of this process are valuations that use a low replacement level, and also incorporate several differences between Scoresheet and MLB. Primarily, the significance of ERA within the simulation is because of how expected runs are scored, as a product of expected runs allowed by pitchers and expected runs scored by hitters, more or less. This multiplicative effect is captured in SS/SIM through the premium placed on elite pitchers. Additionally, a pitcher's ERA is the primary driver in our adjusted usage patterns, which capture the difference in usage between SP and RP from Scoresheet and MLB. SS/SIM as a pitcher metric provides a complete depiction of a pitcher's value in Scoresheet because it incorporates all of these factors to form a comprehensive view of each pitcher.

Usage:

We expect that as a run based metric, SS/SIM will have an approximate equivalency of 10 runs to 1 win. In practice, we hope that SS/SIM helps as a way to measure players of different positions against each other, and on a large scale, to help identify what value to expect from each player. On a more granular basis, because these are projected values, small differences in SS/SIM are probably worth prioritizing below other considerations, like team needs and draft context. Comparisons between pairs of players, especially when looking at draft picks after the middle of the draft, should largely be based on team composition to that point–whether you need 175 IP at 4.55 ERA or 65 IP at 3.60 ERA largely depends on who else you have on your roster.

SS/SIM is available on the player cards and in the Player Forecast Manager.

You need to be logged in to comment. Login or Subscribe
uptick
3/26
Have you guys ever thought about doing something similar to this for strat-o-matic cards?
mcquown
3/28
The difference here is that for SOM cards, all the information for the 2010 season is codified into card format already, so it's a matter of adding up run values for the various card events and comparing to whatever "replacement" value you have in your league. Since Strat-O-Matic sells their "ratings disk", you can get much of the specific card information and compute such things using a spreadsheet and the defensive values based on the X-charts. [some details are missing from the ratings disk, such as SI/DO/TR breakdowns for hits, though you know TB, and SI*/SI**/SI(open) distinctions, and fbA-fbC distinctions, nothing which impacts run values a lot] With Scoresheet, the game is simmed based on current-year stats, and is at the whims of managerial decisions (such as the Napoli question in this thread). So PECOTA projections of batting stats and errors - as well as predicted playing time estimates - are part of the mix. What would be interesting would be a "preview" card rating for the following season's Strat-O-Matic set, but in many ways, that's just impossible, as the game company hasn't made decisions on the ratings for various players yet - most notably defensive ranges, which impact both position player values (directly) and pitcher values (indirectly).
boness
3/26
Any chance this could be added to the Pecota spreadsheet?
dantroy
3/26
The depth charts, too?
o2bnited
3/26
Brian you can get it in spreadsheet form from the PFM output: http://pfm.sportsprospectus.com/index.php?raw=csv
UtahColts
3/29
When you download the raw output, the SS/SIM for pitchers has all zeroes....?
o2bnited
3/29
Thanks, this has been resolved.
hotstatrat
3/26
How predictive are fielding percentages? Do you take a player's last three seasons, regress them to the mean, then tack on an aging factor as fielding percentages tend to improve as a player gets older? More or less fudging than that?
o2bnited
3/28
The predictive power of prior years' fielding percentages is not as high as we would like it to be. We are doing our best by combining some more advanced metrics, like FRAA, into how we calculate expected fielding percentage and expected chances for each player. Otherwise, your description is fairly accurate, yes.
jtwalsh
3/28
With all due respect, I disagree with your premise equating SS value of a pitcher to ERA. I have. Played SS for 10 years and have read what I could find on strategy. Hits and walks given up (whip for lack of a better term) is much more predictive of SS performance. Strike outs are nice because it takes your fielders out of the equation. I ran the PFM and overall I believe it is underestimating pitching, and not doing a great job identifying the most valuable pitchers in SS. Most SS leagues are continuing leagues, so factoring upside would also be a nice option for the pfm.
jrmayne
3/28
I believe you're significantly mistaken on this. ERA controls over WHIP, and hard.
dfloren1
3/10
If what you're saying is true (i.e., "ERA controls over WHIP, and hard"), then Scoresheet must have changed its basic algorithm, and hard, since Mr. Barton wrote the following: " This IS the way it works. A couple posters repeatedly make the claim that *only* ERA matters as to what will happen in Scoresheet. But that is simply not true, nor has any studying of Scoresheet stats (either by us extensively over the years, and also by folks on this list) ever shown that ERA determines what happens in Scoresheet games regardless of what the other team's hitters did in real life. As far as whether ERA is too important compared to a pitcher's hit and walks allowed per inning pitched (WHIP), we do factor *both* ERA and WHIP into the algorithm that determines the odds of each at-bat's outcome. Basically the better a pitcher's WHIP and/or ERA the less the chance of a plate appearance leading to a hit, and the worse those numbers the greater the chance, all also influenced by how the hitter did in real life, and the fielders. But the sim is rather complex - there is no simple number I/we can give for the relative weighting of those two factors." - see http://www.attheplate.com/notes/ss_pitch.htm Thus I never ignore projected and actual WHIPS when drafting or trading for pitchers. If they've since radically changed the algorithm to seriously under-emphasize WHIP, I need someone to tell me.
o2bnited
3/28
I am fairly certain that Scoresheet uses ERA as the primary driver of pitchers' expected runs allowed in the simulation, so I believe this premise is not mine, it is the nature of the simulation engine itself. For more information, you can check out the Scoresheet talk discussion group on Yahoo! where these topics have been covered at greater length and in greater detail with greater insight than I could ever offer here. http://sports.groups.yahoo.com/group/scoresheet-talk/ We will look into factoring UPSIDE into future iterations of SS/SIM. Thanks for your feedback.
fawcettb
3/28
I'm a 20 year scoresheeter, and I find this mostly incomprehensible, and when it is understandable, barely credible. Trying to reduce scoresheet value to a single number simply flies in the face of the SIM's complexity.
o2bnited
3/28
I agree that in general, it's impossible to reduce a player's value in a complicated environment like the Scoresheet simulation into a single number, and still capture all pertinent and relevant information. However, we think that having a single number that is as good as any single number could be is a useful tool for Scoresheet drafts. We did some analysis and this iteration of SS/SIM seems to hold up well for typical league formats like 24 team BL, 10 team AL, and 12 team NL. For a more shallow league, like a 20 team BL, or a 10 team NL, you may need to make some adjustments. We do always suggest that you run your league through PFM to account for the various crossovers or other oddities that may exist in your specific league.
missouri2725
2/18
How do you run a SS league through PFM?
jrmayne
3/28
Thanks for the work. I'm having a few issues. 1. Chris Snyder is ahead of Mike Napoli. 2. If I read this right, you're using player AAA as replacement level. Like the others, I view this as misguided. 3. Felix is expected to allow 66 total earned runs per the spreadsheet. Replacement AAA is 1.8 (league ERA) or roughly 7.30. 230 IP at 7.30 ERA is about 186 runs. That's a 120-run difference, but Felix is only listed at a value of 88. What have I calculated incorrectly? 4. Pujols is listed as 75 runs above Player AAA. That number is also quite a bit lower than I calculate. --JRM
o2bnited
3/28
In essence, we start with player AAA as the guideline, but we adjust upwards to account for the fact that very few teams will ever use much of player AAA and values that are based only off of player AAA will seem very inflated and impractical (just as you mention in 2). We found that experts feel like player AAA is too low of a replacement level to use for a metric to be effective. So while we started out with player AAA, we made an effort to adjust accordingly when appropriate so that both the process and the results seemed reasonable. I won't say that this process is perfect, and I'm open to suggestions to improve it. 1. This may be due to our projections having Napoli playing primarily away from catcher, even though he is catcher eligible in Scoresheet. I will investigate. 2. I understand the desire to have a higher replacement level (see above). The adjustments that we make for defense (compared to average not replacement level) and for overall offensive output (a pure runs above average view) have the effect of raising this replacement level for hitters. 3. The ERA that we're measuring Felix against is lower (better) than what you have there. Not because player AAA will perform better, but because in practice, the pitchers that you could find available will have ERAs a couple of runs lower than pitcher AAA would have. I think the effective replacement ERA for SP is around 5.50 or so. 4. I think this is resolved in the higher replacement level adjustments, but if not, I'd be interested in your methodology and how it compares to ours. Thanks for your continued feedback. :-)
o2bnited
3/28
The Snyder vs Napoli thing seems to be playing time related--Snyder is projected to have a good bit more PA than Napoli. Napoli's SS/SIM rate, or SS/SIM divided by PA is higher than Snyder's, though.
jrmayne
3/28
Sure! It's just that the article above appeared to me to read differently than this response. That's what confused me, because I had understood that replacement level would not be Player AAA. Glad to see it isn't.
o2bnited
3/28
Understood. I put in an edit above that clarifies that the adjustments we perform serve to raise the effective replacement level in addition to addressing the specific concerns around the Scoresheet simulation.
jtwalsh
3/28
From Scoresheet Web Site: http://www.scoresheet.com/BB_FullSim.php •Each pitcher's complete stats are used, including the number of strikeouts, walks, hits, earned runs and innings pitched he recorded that week in the majors - even wild pitches! •Pitchers "pitch carefully" when first base is open and a runner is in scoring position, giving up more walks but fewer big hits •Team-dependent stats such as wins and saves matter less than individual stats such as ERA •Your closer will only pitch in save situations
fieldofdreams
3/28
This is nice, would have been nicer to have BEFORE the draft this year. In the future could you: -list this in the PECOTA sheet along w/ Scoresheet ID? Merging on strings is NOT fun -Could you provide some explanation for how best to use PFM for a SS draft? Dollar values don't mean a lot in that kind of draft. Maybe make SS an option?
rawagman
3/28
I think Scoresheet should definitely be a third league type, along with roto and points.
o2bnited
3/28
I like this idea, but I'm not sure how we would implement it. Let me think it over some more and maybe we can implement something in the future iterations of PFM.
o2bnited
3/28
You can get SS/SIM in the PFM's downloadable data (see link in this comment thread), which includes typical ID systems. We are working to include Scoresheet ID's in the future. Unfortunately, the way that Scoresheet updates their player lists, this requires some manual effort and we are working to streamline that process. For now, the best way to use PFM is to put your players available and taken into the PFM and look at the relative value that the dollar values project. The minimum dollar value for the salary sets the PFM's replacement level for dollars, so every player projected to be worth that much or more should be drafted given the league and setup you put into PFM. The relative values in dollar values should be of some interest, but like you said, they don't correlate as well as the SS/SIM values should to overall value (they are also a second iteration of positional adjustment, which isn't really necessary).
deadringers
3/04
I'm not sure I understand how to use this too for Scoresheet SIM? There are only 2 options and no sim option.
markpadden
3/29
Did I miss an update on the 10-year projections? Maybe for the All-Star break? Seriously, I don't understand how BP management (assuming there is such a thing) can remain silent for a month and a half while a core product is MIA. I've never quite seen a company (fail to) work like this..
o2bnited
3/29
Please see Dave Pease's recent post for an update: http://www.baseballprospectus.com/article.php?articleid=13362
bballexec
4/01
One of the questions I have is how did you deal with OF defense. All OF are lumped together in Scoresheet not split between RF/LF/CF. Wouldn't a CF playing RF have a much higher value? Alternatively, wouldn't a RF playing CF, which can happen in Scoresheet have less value. Also, defense is built into every rating, but the defense should not be a factor for a DH. It seems to me defensive value needs to be separately to allow for positional flexibility. Although maybe a number without defense is more useful.
dfloren1
3/10
Bballexec: You are right that CF's have better defensive numbers than RF or LF. What Scoresheet basically requires you to do is to know the difference between a CF and a corner OF either by basically being acquainted with current MLB depth charts, or visiting the "player list" pages in SS which provide a specific defensive rating number for each player at each position(s) that person is qualifies by SS to play. Once you know their defensive numbers you can decide where to roster each OF (CF, RF, or LF) even without knowing whether, say, Adam Jones is a CF in real life. Just remember to roster your OF who has the highest defensive number at your CF roster position, and if you just so happen to draft 2 or more real-life CF for your SS team, you will have an advantage over your competitors by fielding a better defensive outfield. Ceteris paribus, in SS it's always better to roster a CF in a corner OF position than a non-CF because it helps your pitchers cut down on runs allowed over the course of a season. For example, SS gives Shane Victorino a 2.16 defensive rating. This means he is a CF because SS basically never gives RFs and LFs that high of a number. Now let's look at Michael Morse, who gets a 2.04 rating. That means he's a corner OF. As far as I can tell, whether you roster a guy who qualifies as an OF at LF or RF, it makes absolutely absolutely no difference in SS. Here's a helpful quotation from the SS website that might answer some of your questions ... "The range of the player in center field for you is about 1.4 times as important as either the left or right fielder when figuring your overall team range. This means you should have at least one high range outfielder to play center field for you. The 'average' CFer has a range of about 2.16, while the 'average' LFer and RFer have ranges of about 2.07. Since the range of your CFer matters 1.4 times as much as at other positions, it is better to have a 2.16 range player in center, along with two 2.07 range players in left and right, than it is to have three 2.10 players filling your three OF spots. (Most Scoresheet teams have a player of at least 2.11 range playing CF for them.)" - http://www.scoresheet.com/PDFfiles/AL_rules_pckt.pdf (page 5 has a whole lot more info on how defensive ratings affect results in SS (e.g., penalties for rostering a player at a position he's not qualified by SS to play.) You're right that SS lumps all the OF together in one list when you're setting up your draft lists, and when viewing your league roster page. I used to find this a bit inconvenient (i.e. to have to go visit another SS webpage just to review a player's defensive stats), but then I learned that you can get the information on the undrafted players page by clicking on the player's stats line (which is hyperlinked) and when you click on it you see more information about that player at the bottom of the screen including his specific defensive number(s) at whatever position(s) he plays. Better yet, when your scorecards get published, it lumps the defensive numbers of all your rostered players together to give you a single number (positive or negative) which is the multiplier the SS algorithm applies whenever a ball is in play during a game, which has some small effect on whether a hit, out, error, or a run occurs.
missouri2725
2/18
The list of available players cuts off the #1 hitter (Pujols) and #1 pitcher (Jimenez) for my AL league -- the lists start with Fielder and Soria instead. ???
ripperlv
2/09
Maybe I'm blind, but I can't find SS/SIM for Ubaldo Jimenez on his player card. Also I have him on my team tracker but when I input Forecast, he disappears from the team?