CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

<< Previous Article
Premium Article Future Shock: Prospect... (03/25)
<< Previous Column
Fantasy Article Fantasy Beat: Know You... (03/14)
Next Column >>
Fantasy Article Fantasy Beat: Walden I... (04/05)
Next Article >>
The Week in Quotes: 3/... (03/28)

March 26, 2011

Fantasy Beat

Improved SS/SIM for 2011

by Ben Murphy

We recently went to great lengths to develop the new version of SS/SIM, the Scoresheet simulation value metric here at Baseball Prospectus. As fellow Scoresheet owners, we realize that optimally these valuations would have been complete several weeks ago. Unfortunately, the original formulation of SS/SIM was not available to us this year, and our replacement took more time to develop than we had anticipated. We set out to create an all-new run-based value metric for Scoresheet team owners to use in player valuation. This time consuming effort was spearheaded by BP staffers with Scoresheet experience, and included research into the Scoresheet engine, talking with Jeff and Dave Barton, and consulting with Scoresheet experts for feedback.

Development of SS/SIM for both hitters and pitchers starts with the Scoresheet definition of replacement level, namely the awful Triple-A players that are used when you run out of playing time in the Scoresheet simulation. From there we have captured the most important aspects of the Scoresheet simulation and built them into the SS/SIM methodology to produce a single metric for Scoresheet owners. These adjustments seek to raise the effective replacement level and account for the differences between MLB and Scoresheet. [edit: BJM 2011-03-28]

Hitters:

The offensive formulation builds a value estimate from the replacement level based on expected offensive output. The offensive valuation is built from a positionally adjusted perspective that also emphasizes overall offensive output, supplemented with a defensive adjustment that accounts for each player's Scoresheet defensive eligibility and range rating, as well as a penalty for expected errors. 

The results of this process are valuations that use a very low replacement level--note that Player(AAA) is remarkably low in value compared to the replacement level used for metrics designed for evaluating MLB players. SS/SIM as a hitter metric provides a complete depiction of a player's value in Scoresheet because it uses replacement value specific to Scoresheet and incorporates the latest research on Scoresheet defensive range value and appropriate valuation of errors to form a comprehensive view of the player.

PItchers:

The pitching formulation is a bit more complicated. Pitcher valuation is based around projected IP and ERA because the foundation for value in Scoresheet for pitchers is their ERA, and because pitchers in Scoresheet are basically limited to their real life playing time. Because we expect Scoresheet owners to alter pitcher usage based on ERA, each pitcher is projected to throw a certain percentage of his real life innings for the Scoresheet fantasy team. The projection for how many of his real life innings will be utilized is separated by starters and relievers based on the expectation that Scoresheet managers will maximize usage of their relief pitchers because they have lower ERAs, which is the converse of real life MLB usage. As a result of this usage pattern, the ratio of SP IP to RP IP is lower in Scoresheet than MLB, and our methodology reflects that difference as well. We also put a premium on pitchers whose ERA is better than league average to account for the importance of ERA and the multiplicative nature of actual run production in the simulation. This adjustment is proportional to the extent that a pitcher is better than replacement level, so it impacts premium starters the most, but it also increases the importance of very good relievers as well. Given the expectation for how many innings a pitcher would pitch for his Scoresheet team, we can derive his SS/SIM by comparing his forecast for runs allowed over that time to what the Pitcher(AAA) would allow.

The results of this process are valuations that use a low replacement level, and also incorporate several differences between Scoresheet and MLB. Primarily, the significance of ERA within the simulation is because of how expected runs are scored, as a product of expected runs allowed by pitchers and expected runs scored by hitters, more or less. This multiplicative effect is captured in SS/SIM through the premium placed on elite pitchers. Additionally, a pitcher's ERA is the primary driver in our adjusted usage patterns, which capture the difference in usage between SP and RP from Scoresheet and MLB. SS/SIM as a pitcher metric provides a complete depiction of a pitcher's value in Scoresheet because it incorporates all of these factors to form a comprehensive view of each pitcher.

Usage:

We expect that as a run based metric, SS/SIM will have an approximate equivalency of 10 runs to 1 win. In practice, we hope that SS/SIM helps as a way to measure players of different positions against each other, and on a large scale, to help identify what value to expect from each player. On a more granular basis, because these are projected values, small differences in SS/SIM are probably worth prioritizing below other considerations, like team needs and draft context. Comparisons between pairs of players, especially when looking at draft picks after the middle of the draft, should largely be based on team composition to that point--whether you need 175 IP at 4.55 ERA or 65 IP at 3.60 ERA largely depends on who else you have on your roster.

SS/SIM is available on the player cards and in the Player Forecast Manager.

Related Content:  Scoresheet,  Pitcher Usage

33 comments have been left for this article. (Click to hide comments)

BP Comment Quick Links

Wade

Have you guys ever thought about doing something similar to this for strat-o-matic cards?

Mar 26, 2011 06:00 AM
rating: 1
 
BP staff member Rob McQuown
BP staff

The difference here is that for SOM cards, all the information for the 2010 season is codified into card format already, so it's a matter of adding up run values for the various card events and comparing to whatever "replacement" value you have in your league. Since Strat-O-Matic sells their "ratings disk", you can get much of the specific card information and compute such things using a spreadsheet and the defensive values based on the X-charts. [some details are missing from the ratings disk, such as SI/DO/TR breakdowns for hits, though you know TB, and SI*/SI**/SI(open) distinctions, and fbA-fbC distinctions, nothing which impacts run values a lot]

With Scoresheet, the game is simmed based on current-year stats, and is at the whims of managerial decisions (such as the Napoli question in this thread). So PECOTA projections of batting stats and errors - as well as predicted playing time estimates - are part of the mix. What would be interesting would be a "preview" card rating for the following season's Strat-O-Matic set, but in many ways, that's just impossible, as the game company hasn't made decisions on the ratings for various players yet - most notably defensive ranges, which impact both position player values (directly) and pitcher values (indirectly).

Mar 28, 2011 11:53 AM
 
Brian DewBerry-Jones
(244)

Any chance this could be added to the Pecota spreadsheet?

Mar 26, 2011 07:43 AM
rating: 0
 
dantroy

The depth charts, too?

Mar 26, 2011 07:53 AM
rating: -1
 
BP staff member Ben Murphy
BP staff

Brian you can get it in spreadsheet form from the PFM output:

http://pfm.sportsprospectus.com/index.php?raw=csv

Mar 26, 2011 09:50 AM
 
thebigfella.us

When you download the raw output, the SS/SIM for pitchers has all zeroes....?

Mar 28, 2011 19:53 PM
rating: 0
 
BP staff member Ben Murphy
BP staff

Thanks, this has been resolved.

Mar 29, 2011 07:48 AM
 
John Carter

How predictive are fielding percentages? Do you take a player's last three seasons, regress them to the mean, then tack on an aging factor as fielding percentages tend to improve as a player gets older? More or less fudging than that?

Mar 26, 2011 11:04 AM
rating: -1
 
BP staff member Ben Murphy
BP staff

The predictive power of prior years' fielding percentages is not as high as we would like it to be. We are doing our best by combining some more advanced metrics, like FRAA, into how we calculate expected fielding percentage and expected chances for each player. Otherwise, your description is fairly accurate, yes.

Mar 28, 2011 09:05 AM
 
jtwalsh

With all due respect, I disagree with your premise equating SS value of a pitcher to ERA. I have. Played SS for 10 years and have read what I could find on strategy. Hits and walks given up (whip for lack of a better term) is much more predictive of SS performance. Strike outs are nice because it takes your fielders out of the equation.

I ran the PFM and overall I believe it is underestimating pitching, and not doing a great job identifying the most valuable pitchers in SS.

Most SS leagues are continuing leagues, so factoring upside would also be a nice option for the pfm.

Mar 27, 2011 17:00 PM
rating: 1
 
jrmayne

I believe you're significantly mistaken on this. ERA controls over WHIP, and hard.

Mar 28, 2011 07:23 AM
rating: 0
 
doublesteel

If what you're saying is true (i.e., "ERA controls over WHIP, and hard"), then Scoresheet must have changed its basic algorithm, and hard, since Mr. Barton wrote the following:
" This IS the way it works. A couple posters repeatedly make the claim that *only* ERA matters as to what will happen in Scoresheet. But that is simply not true, nor has any studying of Scoresheet stats (either by us extensively over the years, and also by folks on this list) ever shown that ERA determines what happens in Scoresheet games regardless of what the other team's hitters did in real life. As far as whether ERA is too important compared to a pitcher's hit and walks allowed per inning pitched (WHIP), we do factor *both* ERA and WHIP into the algorithm that determines the odds of each at-bat's outcome. Basically the better a pitcher's WHIP and/or ERA the less the chance of a plate appearance leading to a hit, and the worse those numbers the greater the chance, all also influenced by how the hitter did in real life, and the fielders. But the sim is rather complex - there is no simple number I/we can give for the relative weighting of those two factors."
- see http://www.attheplate.com/notes/ss_pitch.htm

Thus I never ignore projected and actual WHIPS when drafting or trading for pitchers. If they've since radically changed the algorithm to seriously under-emphasize WHIP, I need someone to tell me.

Mar 10, 2013 00:13 AM
rating: 0
 
BP staff member Ben Murphy
BP staff

I am fairly certain that Scoresheet uses ERA as the primary driver of pitchers' expected runs allowed in the simulation, so I believe this premise is not mine, it is the nature of the simulation engine itself. For more information, you can check out the Scoresheet talk discussion group on Yahoo! where these topics have been covered at greater length and in greater detail with greater insight than I could ever offer here.

http://sports.groups.yahoo.com/group/scoresheet-talk/

We will look into factoring UPSIDE into future iterations of SS/SIM. Thanks for your feedback.

Mar 28, 2011 08:43 AM
 
fawcettb

I'm a 20 year scoresheeter, and I find this mostly incomprehensible, and when it is understandable, barely credible. Trying to reduce scoresheet value to a single number simply flies in the face of the SIM's complexity.

Mar 28, 2011 05:11 AM
rating: 2
 
BP staff member Ben Murphy
BP staff

I agree that in general, it's impossible to reduce a player's value in a complicated environment like the Scoresheet simulation into a single number, and still capture all pertinent and relevant information. However, we think that having a single number that is as good as any single number could be is a useful tool for Scoresheet drafts. We did some analysis and this iteration of SS/SIM seems to hold up well for typical league formats like 24 team BL, 10 team AL, and 12 team NL. For a more shallow league, like a 20 team BL, or a 10 team NL, you may need to make some adjustments.

We do always suggest that you run your league through PFM to account for the various crossovers or other oddities that may exist in your specific league.

Mar 28, 2011 09:10 AM
 
missouri2725

How do you run a SS league through PFM?

Feb 18, 2012 13:27 PM
rating: 0
 
jrmayne

Thanks for the work. I'm having a few issues.

1. Chris Snyder is ahead of Mike Napoli.

2. If I read this right, you're using player AAA as replacement level. Like the others, I view this as misguided.

3. Felix is expected to allow 66 total earned runs per the spreadsheet. Replacement AAA is 1.8 (league ERA) or roughly 7.30. 230 IP at 7.30 ERA is about 186 runs. That's a 120-run difference, but Felix is only listed at a value of 88. What have I calculated incorrectly?

4. Pujols is listed as 75 runs above Player AAA. That number is also quite a bit lower than I calculate.

--JRM

Mar 28, 2011 07:35 AM
rating: 0
 
BP staff member Ben Murphy
BP staff

In essence, we start with player AAA as the guideline, but we adjust upwards to account for the fact that very few teams will ever use much of player AAA and values that are based only off of player AAA will seem very inflated and impractical (just as you mention in 2).

We found that experts feel like player AAA is too low of a replacement level to use for a metric to be effective. So while we started out with player AAA, we made an effort to adjust accordingly when appropriate so that both the process and the results seemed reasonable. I won't say that this process is perfect, and I'm open to suggestions to improve it.

1. This may be due to our projections having Napoli playing primarily away from catcher, even though he is catcher eligible in Scoresheet. I will investigate.

2. I understand the desire to have a higher replacement level (see above). The adjustments that we make for defense (compared to average not replacement level) and for overall offensive output (a pure runs above average view) have the effect of raising this replacement level for hitters.

3. The ERA that we're measuring Felix against is lower (better) than what you have there. Not because player AAA will perform better, but because in practice, the pitchers that you could find available will have ERAs a couple of runs lower than pitcher AAA would have. I think the effective replacement ERA for SP is around 5.50 or so.

4. I think this is resolved in the higher replacement level adjustments, but if not, I'd be interested in your methodology and how it compares to ours.

Thanks for your continued feedback. :-)

Mar 28, 2011 09:01 AM
 
BP staff member Ben Murphy
BP staff

The Snyder vs Napoli thing seems to be playing time related--Snyder is projected to have a good bit more PA than Napoli. Napoli's SS/SIM rate, or SS/SIM divided by PA is higher than Snyder's, though.

Mar 28, 2011 09:52 AM
 
jrmayne

Sure! It's just that the article above appeared to me to read differently than this response. That's what confused me, because I had understood that replacement level would not be Player AAA. Glad to see it isn't.

Mar 28, 2011 10:14 AM
rating: -1
 
BP staff member Ben Murphy
BP staff

Understood. I put in an edit above that clarifies that the adjustments we perform serve to raise the effective replacement level in addition to addressing the specific concerns around the Scoresheet simulation.

Mar 28, 2011 10:51 AM
 
jtwalsh

From Scoresheet Web Site:


http://www.scoresheet.com/BB_FullSim.php

•Each pitcher's complete stats are used, including the number of strikeouts, walks, hits, earned runs and innings pitched he recorded that week in the majors - even wild pitches!
•Pitchers "pitch carefully" when first base is open and a runner is in scoring position, giving up more walks but fewer big hits
•Team-dependent stats such as wins and saves matter less than individual stats such as ERA
•Your closer will only pitch in save situations

Mar 28, 2011 07:47 AM
rating: -1
 
fieldofdreams

This is nice, would have been nicer to have BEFORE the draft this year. In the future could you:
-list this in the PECOTA sheet along w/ Scoresheet ID? Merging on strings is NOT fun
-Could you provide some explanation for how best to use PFM for a SS draft? Dollar values don't mean a lot in that kind of draft. Maybe make SS an option?

Mar 28, 2011 09:26 AM
rating: 0
 
R.A.Wagman

I think Scoresheet should definitely be a third league type, along with roto and points.

Mar 28, 2011 09:41 AM
rating: 0
 
BP staff member Ben Murphy
BP staff

I like this idea, but I'm not sure how we would implement it. Let me think it over some more and maybe we can implement something in the future iterations of PFM.

Mar 28, 2011 10:03 AM
 
BP staff member Ben Murphy
BP staff

You can get SS/SIM in the PFM's downloadable data (see link in this comment thread), which includes typical ID systems. We are working to include Scoresheet ID's in the future. Unfortunately, the way that Scoresheet updates their player lists, this requires some manual effort and we are working to streamline that process.

For now, the best way to use PFM is to put your players available and taken into the PFM and look at the relative value that the dollar values project. The minimum dollar value for the salary sets the PFM's replacement level for dollars, so every player projected to be worth that much or more should be drafted given the league and setup you put into PFM. The relative values in dollar values should be of some interest, but like you said, they don't correlate as well as the SS/SIM values should to overall value (they are also a second iteration of positional adjustment, which isn't really necessary).

Mar 28, 2011 10:01 AM
 
deadringers

I'm not sure I understand how to use this too for Scoresheet SIM? There are only 2 options and no sim option.

Mar 04, 2012 12:33 PM
rating: 0
 
evo34

Did I miss an update on the 10-year projections? Maybe for the All-Star break? Seriously, I don't understand how BP management (assuming there is such a thing) can remain silent for a month and a half while a core product is MIA. I've never quite seen a company (fail to) work like this..

Mar 29, 2011 00:47 AM
rating: 0
 
BP staff member Ben Murphy
BP staff

Please see Dave Pease's recent post for an update:

http://www.baseballprospectus.com/article.php?articleid=13362

Mar 29, 2011 07:32 AM
 
bballexec
(376)

One of the questions I have is how did you deal with OF defense. All OF are lumped together in Scoresheet not split between RF/LF/CF. Wouldn't a CF playing RF have a much higher value? Alternatively, wouldn't a RF playing CF, which can happen in Scoresheet have less value. Also, defense is built into every rating, but the defense should not be a factor for a DH. It seems to me defensive value needs to be separately to allow for positional flexibility. Although maybe a number without defense is more useful.

Mar 31, 2011 21:45 PM
rating: 0
 
doublesteel

Bballexec: You are right that CF's have better defensive numbers than RF or LF. What Scoresheet basically requires you to do is to know the difference between a CF and a corner OF either by basically being acquainted with current MLB depth charts, or visiting the "player list" pages in SS which provide a specific defensive rating number for each player at each position(s) that person is qualifies by SS to play. Once you know their defensive numbers you can decide where to roster each OF (CF, RF, or LF) even without knowing whether, say, Adam Jones is a CF in real life. Just remember to roster your OF who has the highest defensive number at your CF roster position, and if you just so happen to draft 2 or more real-life CF for your SS team, you will have an advantage over your competitors by fielding a better defensive outfield. Ceteris paribus, in SS it's always better to roster a CF in a corner OF position than a non-CF because it helps your pitchers cut down on runs allowed over the course of a season. For example, SS gives Shane Victorino a 2.16 defensive rating. This means he is a CF because SS basically never gives RFs and LFs that high of a number. Now let's look at Michael Morse, who gets a 2.04 rating. That means he's a corner OF. As far as I can tell, whether you roster a guy who qualifies as an OF at LF or RF, it makes absolutely absolutely no difference in SS. Here's a helpful quotation from the SS website that might answer some of your questions ...

"The range of the player in center field for you is about 1.4 times as important as either the left or right fielder when figuring your overall team range. This means you should have at least one high range outfielder to play center field for you. The 'average' CFer has a range of about 2.16, while
the 'average' LFer and RFer have ranges of about 2.07. Since the range of your CFer matters 1.4 times as much as at other positions, it is better to have a 2.16 range player in center, along with two 2.07 range players in left and right, than it is to have three 2.10 players filling your three OF spots. (Most Scoresheet teams have a player of at least 2.11 range playing CF for them.)"
- http://www.scoresheet.com/PDFfiles/AL_rules_pckt.pdf
(page 5 has a whole lot more info on how defensive ratings affect results in SS (e.g., penalties for rostering a player at a position he's not qualified by SS to play.)

You're right that SS lumps all the OF together in one list when you're setting up your draft lists, and when viewing your league roster page. I used to find this a bit inconvenient (i.e. to have to go visit another SS webpage just to review a player's defensive stats), but then I learned that you can get the information on the undrafted players page by clicking on the player's stats line (which is hyperlinked) and when you click on it you see more information about that player at the bottom of the screen including his specific defensive number(s) at whatever position(s) he plays. Better yet, when your scorecards get published, it lumps the defensive numbers of all your rostered players together to give you a single number (positive or negative) which is the multiplier the SS algorithm applies whenever a ball is in play during a game, which has some small effect on whether a hit, out, error, or a run occurs.

Mar 10, 2013 01:01 AM
rating: 0
 
missouri2725

The list of available players cuts off the #1 hitter (Pujols) and #1 pitcher (Jimenez) for my AL league -- the lists start with Fielder and Soria instead. ???

Feb 18, 2012 13:48 PM
rating: 0
 
ripperlv

Maybe I'm blind, but I can't find SS/SIM for Ubaldo Jimenez on his player card. Also I have him on my team tracker but when I input Forecast, he disappears from the team?

Feb 09, 2014 12:56 PM
rating: 0
 
You must be a Premium subscriber to post a comment.
Not a subscriber? Sign up today!
<< Previous Article
Premium Article Future Shock: Prospect... (03/25)
<< Previous Column
Fantasy Article Fantasy Beat: Know You... (03/14)
Next Column >>
Fantasy Article Fantasy Beat: Walden I... (04/05)
Next Article >>
The Week in Quotes: 3/... (03/28)

RECENTLY AT BASEBALL PROSPECTUS
Fantasy Rounders: Turn On the Green Light
Premium Article The Prospectus Hit List: June 29, 2015
Premium Article Raising Aces: 2015 Draft Audit, Part 3
Premium Article Prospect Profile: Yoan Moncada
Premium Article Transaction Analysis: The Sandberg Goeth
Premium Article BP Milwaukee
Premium Article Monday Morning Ten Pack: June 29, 2015


MORE BY BEN MURPHY
2014-01-31 - Fantasy Article TTO Scoresheet Podcast: Second Basemen
2014-01-24 - Fantasy Article TTO Scoresheet Podcast: First Basemen
2014-01-17 - TTO Scoresheet Podcast: Catchers
2011-03-26 - Fantasy Beat: Improved SS/SIM for 2011
2008-04-21 - Fantasy Article BP Kings Preview
2007-04-11 - Fantasy Article BP Kings Update
2006-11-09 - Predictatron Recap
More...

MORE FANTASY BEAT
2011-04-27 - Fantasy Article Fantasy Beat: Your Wolf Pack: Should It Grow...
2011-04-05 - Fantasy Beat: Donnybrook - Mike Stanton
2011-04-05 - Fantasy Article Fantasy Beat: Walden In, Rodney Out in Anahe...
2011-03-26 - Fantasy Beat: Improved SS/SIM for 2011
2011-03-14 - Fantasy Article Fantasy Beat: Know Your Enemy
2011-03-01 - Fantasy Beat: 2011 Projected Average Roto Va...
2011-02-25 - Fantasy Beat: Donnybrook - Carlos Gomez
More...

INCOMING ARTICLE LINKS
2014-03-14 - Fantasy Article TTO Scoresheet Podcast: Episode 9: SS/SIM
2011-03-31 - Fantasy Article Fantasy Beat: Scoresheet Lineup Tips