CSS Button No Image Css3Menu.com

Baseball Prospectus home
  
  
Click here to log in Click here for forgotten password Click here to subscribe

<< Previous Article
Premium Article Contractual Matters: R... (04/08)
<< Previous Column
Premium Article Ahead in the Count: Pr... (03/25)
Next Column >>
Ahead in the Count: La... (04/15)
Next Article >>
Premium Article Future Shock: The Top ... (04/08)

April 8, 2010

Ahead in the Count

Projecting Free Agent Performance

by Matt Swartz

the archives are now free.

All Baseball Prospectus Premium and Fantasy articles more than a year old are now free as a thank you to the entire Internet for making our work possible.

Not a subscriber? Get exclusive content like this delivered hot to your inbox every weekday. Click here for more information on Baseball Prospectus subscriptions or use the buttons to the right to subscribe and get instant access to the best baseball content on the web.

Subscribe for $4.95 per month
Recurring subscription - cancel anytime.


a 33% savings over the monthly price!

Purchase a $39.95 gift subscription
a 33% savings over the monthly price!

Already a subscriber? Click here and use the blue login bar to log in.

Before a team attempts to sign a free agent, they try to anticipate his value to them by approximating what they feel that player’s production will be over the coming years. Using that, they can then decide what to offer him. There are two prominent methods for evaluating free agents to come out of the sabermetric community in recent years, first here at Baseball Prospectus by Nate Silver in 2005 and later at FanGraphs in 2009. Both have made simple projections for the free-agent crop, and compared those to the salaries they received.

In moving into covering this topic as well, I have already noted that free agents who switch teams tend to decline more quickly than free agents who were not allowed to depart, which suggests that there is selection bias in who teams retain. Today, I'll look at all players who signed multi-year deals from 2007-09, and first demonstrate that these players generally fail to live up to their projections, and then delve into the fact that both re-signed players and those who sign with new teams underperform similarly in the first year of their contracts.

Players who re-sign for multiple years often agree to their deals outside of the offseason; indeed, some sign extensions pretty far in advance of a deal's period of service. For purposes of this piece, if a player signs a deal before he reaches six years of service time, the first year of this “deal” is the first year of the deal in which he began with six years' service time. So, for example, Joe Blanton and the Phillies avoided his last year of arbitration in 2010 by signing a three-year deal covering 2010-12, but the first year of his deal for purposes of this article is going to be 2011, the first year after he'd completed six seasons of service. To evaluate these deals, I used projected and actual VORP, since I wanted a counting statistic to allow for the fact that injured free agents turn out to obviously be bad investments. I specifically used VORP instead of WARP when projections were involved, since it was recalculated from a higher baseline beginning with the 2008 season, rendering 2007 and 2008 WARP projections too high and therefore working from a different scale. Although VORP does not evaluate defense or any balls-in-play performance metrics for pitchers, it will serve its purpose on the aggregate—the results are not ambiguous.

Of the 165 free agents who began multi-year deals in the last three seasons, just 72 managed to top their PECOTA projection in the first year of their contracts. The failure to get there was pretty extreme in some cases (I’m looking at you, Andruw Jones and Carlos Silva), leading the group to fall short of their projection by 14 percent; as a whole, they produced 2,483 VORP compared to 2,878.7 projected VORP by PECOTA. This was true for both re-signed players and players who signed with others teams, as re-signed players underperformed their VORP projection by 13 percent, compared to players who signed with other teams, who underperformed their VORP projection by 15 percent. This is consistent with Sean Smith’s finding that his CHONE projections were 16 percent too high for players on the free agent market.

 

Context PECOTA Actual Difference
Re-Signings 1808.8 1572.6 -13%
Signed w/New Teams 1069.9 910.4 -15%
Combined 2878.7 2483.0 -14%

 

Does this mean that PECOTA and other projection systems are universally optimistic? No. On the aggregate, they both predict about 81 wins per team each season, indicating that it is merely the sample of players who signed free-agent contracts that are the problem. This appears to be an issue with this particular sample of players who reach free agency. We'll get into why this is the case shortly.

It is important to couple this fact with my finding from February that players who sign with their old teams age far better than players who sign with new teams. I found players who signed contracts with their previous teams did not decline much if at all during the two-year and three-year contracts, while players who signed with new teams declined during similar deals; players who signed four-year contracts generally aged well. Here's the breakdown, and remember, breaking things down by age did not indicate any selection bias:

 

Two-Year Deals'WARP Year 1 Year 2
Re-Signings 39% 61%
Signed w/New Teams 63% 37%

 

Three-Year Deals' WARP Year 1 Year 2 Year 3
Re-Signings 36% 34% 30%
Signed w/New Teams 45% 39% 16%

 

Four-Year Deals' WARP Year 1 Year 2 Year 3 Year 4
Re-Signings 20% 33% 29% 17%
Signed w/New Teams 29% 25% 21% 24%

 

Pretty much all projections forecast decline for players old enough to reach free agency, so the implication of the information above is that teams who re-sign their players actually end up getting pretty good value, even if the first year of the deal does not look so hot, while players that sign with new teams tend to underperform their PECOTA projections and then decline further.

This is not meant to highlight a problem with projection systems in general. No projection system can take into account all of the qualitative information that goes into teams’ projections of performance or the actual performances; this is merely a biased sample. Just like looking at PECOTA projections of pitchers who had Tommy John Surgery the previous summer will yield a biased sample, and therefore produce an over-projection of innings if nothing else, looking only at players who signed with their previous teams and excluding the following year is a sample that is biased in a different way. Included in this sample are players whose previous teams chose not to retain their services—this group of players generally underperformed their PECOTA projection the first year of their contract, and then declined further as the contract went on. This sample also includes some players who did re-sign with their previous teams but whose teams initially did not extend them past that season. Of course, it also includes players who just became eligible for free agency for the first time.

It's important to remember that teams’ decisions against re-signing players are not always based on the ability to spend. Frequently they have inside information about the player that deters them from offering a contract. This includes information about the daily bumps and bruises and how well his body bounces back from them, how much medical treatment pitchers require between starts, and how diligent players are about their conditioning. Even players whose teams waited until the offseason to re-sign their players comprise a biased sample as well—these are players whose teams often chose to evaluate them further before tendering a contract. While sometimes this is because Scott Boras is just hell-bent on “testing the market,” frequently the teams simply had reservations and wanted to see how well those players held up.

The reason this matters so much is that those two samples of players are used to determine the old version of MORP and the FanGraphs’ “Dollars” estimate. Projections for these players are consistently too high, and yet they are being used to price the value of all players. Both of these methods will underestimate the actual amount of dollars paid per win, because they overestimate the number of wins these players will produce. This will be exacerbated when many of these players wind up declining beyond what these methods project.

As I move forward with MORP, I will take a different route to price talent. Knowing these biases, it is apparent that using all players with six years of service time will produce a better sample. This entire method will be revealed further in the coming weeks, but just like it was essential for me to rigorously test whether MORP should be linear in January and February, it was also essential to study the sample used. The results clearly show that using a projection system that is unaware of the context in which players were signed will not accurately price free agents.

Matt Swartz is an author of Baseball Prospectus. 
Click here to see Matt's other articles. You can contact Matt by clicking here

Related Content:  Free Agents,  Projections,  The Who,  Free

16 comments have been left for this article. (Click to hide comments)

BP Comment Quick Links

Brian24

Matt, I'm not clear on what the percentages in the last three tables are indicating. Each row seems to add to 100%; does this mean you're showing the percentage of the total WARP generated by these players in each year of the contract? And if so, how does this relate to the projections?

Apr 08, 2010 12:07 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

Correct. It is percent of total WARP for the whole deal in each year. It shows that players who re-signed maintained their value well but players signing with new teams did not.

Apr 08, 2010 17:56 PM
 
Richard Bergstrom

I'm a bit curious how statistically significant this is.

Bear with me a second as I try to make sure I'm following the math.

165 players in the sample minus 72 that topped their PECOTA projections = 93 players

2878.7 projected VORP minus 2483 actual VORP is a difference of 395.7 VORP.

395.7 VORP divided by 93 players is about 4.25 WARP per player.

From what I understand, a rule of thumb is that 10 VORP = 1 win. So basically, over a multiyear contract, you are saying that players underperform their contract by half a win. Not even necessarily that they underperform by half a win each year, but that they underperform by a little less than half a win over the entire contract.

Isn't that half a win underperformance over a 2-4 year contract pretty negligible? Is it just noise? If not, and if something like age is truly not a factor, then perhaps there is some other factor?

Also, how much of a difference between projection and actual VORP is considered underperformance? If a player underperformed by 0.1 VORP, were they counted among the 93 underperformers? In addition, 93/165=56%... so basically we're saying 56% of free agents underperform and 44 percent overperform... which again, with a sample size of 165, might mean the 44/56 split might just be an error margin/variance.

Apr 10, 2010 22:09 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

In a one-sided test, 56% under-performances is significant at the 94.8% level (t-stat=1.63, with t=(93/165-.50)/sqrt(.50*.50/165). The argument is backed up by theoretical evidence though, which is very important here. There is a reason why it SHOULD be a biased sample. The additional evidence of the difference between aging of re-signed and newly-signed players is also evidence of this kind of bias.

Also, this isn't 1/2 a win over the whole contract. It's just the first year. It gets worse than that for newly-signed players and better than that for re-signed players.

Apr 11, 2010 09:02 AM
 
Richard Bergstrom

Thanks for the clarification. And I did make an error on the half a win over the length of the contract part.

So, if I concede that there is a decline for free agents that sign with other teams, can that decline be solely (or primarily) attributed to teams having inside medical information on their players? Could it be a change in ballparks or leagues (though I don't know if VORP accounts for that). Maybe something more obscure like clubhouse chemistry or a change in time zones causing jet lag or an inferior video room for seeing replays? Or in more general terms, is the theory about team insider information support the evidence, or just fit it?

Apr 11, 2010 13:54 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

VORP does account for league adjustments so that should not be an issue. I doubt that clubhouse chemistry or a change in time zones is the explanation specifically because the players who change teams are the ones who do worse and worse over the rest of the contract. The players who re-sign tend to underperform their VORP the first year, but then they don't really decline much at all. The players signing with new teams underperform their projections and then decline even further than that. That's why I think it has more to do with medical information as well as simple workout tendencies. For instance, I think the Phillies knew that Chase Utley was a workaholic and probably would have a longer peak than other second basemen so they signed him for longer.

Apr 11, 2010 15:02 PM
 
Richard Bergstrom

It might be interesting to see if there is a difference in performance for players who are eligible for freeagency, get traded midseason then re-sign with their new team. If medical information is a reason for the dropoff in performance, then I would imagine that those players traded midseason would do slightly better than those who switched teams because a pre-trade physical was given and the re-signing team thus had some medical information and had the player for a half a year to see their work ethic, etc.... but may not do as well as those who stayed with one team throughout their career and re-signed with the same team because those teams would have a more complete medical/work ethic/nutrition history. Make sense?

Apr 11, 2010 16:25 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

I don't think it's medical information in the way you're thinking. Players take physicals in all of these scenarios in general, so it's not really that. When I say medical issues, I mean just knowing how the player's body holds up day-to-day. Something like knowing how well a reliever's arm bounces back is something the player's old team just knows better.

Apr 13, 2010 22:03 PM
 
Ivan Bezdomny

Matt,

1. Why do you not control for age? Calculate average age-based incline/decline for all free agents (new team or not) for each year, and show values adjusted based on that?

2. Maybe I'm missing something, but why do you say "projection systems" when you are only looking at PECOTA? What other projection systems did you consider?

Very interesting study! Thanks!

Apr 11, 2010 18:57 PM
rating: 0
 
Ivan Bezdomny

OK I see that you specifically mention CHONE projections as well as PECOTA. However if these systems systematically over-estimate VORP for a general class of player, than this both a problem, and something that can be corrected (by decreasing projections for this class of player to minimize error).

I only looked at pitching projections, but I found that PECOTA and VORP both over-estimate IP for pitchers by large factors. PECOTA also systematically over-estimates pitcher VORP, both for veterans, but especially for young pitchers (CHONE does not project VORP directly). Maybe there are projection systems out there that have better results?

In any case, I'm not sure that projected VORP has much to do with value of free agents? But certainly one should not put too much emphasis on the expected vs actual value, based on expected estimates that tend to be 16% high on average...

Apr 12, 2010 07:19 AM
rating: 0
 
Richard Bergstrom

I thought PECOTA is based on pre-created depth charts, not that PECOTA generates the playing time/depth charts itself... so wouldn't the "over-estimate IP for pitchers" issue be a problem with the depth charts and not with PECOTA?

Apr 12, 2010 08:35 AM
rating: 0
 
Ivan Bezdomny

Not quite. Pitchers' expected IP can be predicted quite well without looking at depth chart or other team considerations (assuming that teams don't stockpile pitchers in inferior roles, which they rarely do). Depth charts can be (and are used, in the case of PECOTA, as far as I know) to adjust these IP projections.

However IP should not be set by role primarily. CC Sabathia should be expected to throw more innings than Cory Lidle, even if both are guaranteed slots in the rotation. Giving them the same IP projection because they have the same role would not work very well. Looking at both the IP and VORP projections from PECOTA, they consistently over-estimate these values. I assume these are not lineup-adjusted? Does anyone know this? I'm very curious :-)

Apr 12, 2010 11:03 AM
rating: 0
 
Richard Bergstrom

Um... I hope you were picking Cory Lidle as an extreme scenario because he did pass away in 2006 and I doubt there's been a PECOTA projection for him since.

Apr 12, 2010 12:42 PM
rating: 0
 
BP staff member Matt Swartz
BP staff

I didn't use depth charts. I actually just used PECOTA spreadsheets. I'm sure that they overestimate IP on average but probably not for players who got multiyear deals since those players were less likely to be on the borderline between being in the minors and majors the first year. If PECOTA runs a little hot, that's another issue, but I don't think the CHONE system was running all that hot when it happened and Sean Smith didn't seem to see that as the issue in the article I linked.

Apr 13, 2010 22:00 PM
 
BP staff member Matt Swartz
BP staff

I did look at age originally but it didn't seem to be the source of the bias, so I dropped that from the analysis. Players did age worse as they were older, but that just didn't seem to be what was going on at all.

Apr 13, 2010 21:56 PM
 
Ivan Bezdomny

Yes he did pass away. I meant Ted Lilly. I tend to confuse the two, for some reason.

Apr 12, 2010 16:58 PM
rating: 0
 
You must be a Premium subscriber to post a comment.
Not a subscriber? Sign up today!
<< Previous Article
Premium Article Contractual Matters: R... (04/08)
<< Previous Column
Premium Article Ahead in the Count: Pr... (03/25)
Next Column >>
Ahead in the Count: La... (04/15)
Next Article >>
Premium Article Future Shock: The Top ... (04/08)

RECENTLY AT BASEBALL PROSPECTUS
Every Team's Moneyball: Cincinnati Reds: Go ...
Every Team's Moneyball: Chicago White Sox: T...
Premium Article Some Projection Left: The Moran Mystery
Notes from the Field: Seven Days and 32 Pros...
Spring Training Notebook: Cactus League
Premium Article Rubbing Mud: The Demise of the Two-Out Rally
Premium Article Some Projection Left: Matuella has Tommy Joh...

MORE FROM APRIL 8, 2010
Premium Article Under The Knife: Sneaky Recoveries
Premium Article Manufactured Runs: April is the Cruelest Mon...
Premium Article Future Shock: The Top 101 Prospects Update, ...
Premium Article Contractual Matters: Rule 5 Update

MORE BY MATT SWARTZ
2010-04-23 - Premium Article Ahead in the Count: Methodology of The New M...
2010-04-22 - Premium Article Ahead in the Count: Reintroducing MORP
2010-04-15 - Ahead in the Count: Labor Market Discriminat...
2010-04-08 - Premium Article Ahead in the Count: Projecting Free Agent Pe...
2010-03-25 - Premium Article Ahead in the Count: Predicting BABIP, Part 3
2010-03-24 - Premium Article Ahead in the Count: Predicting BABIP, Part 2
2010-03-23 - Premium Article Ahead in the Count: Predicting BABIP, Part 1
More...

MORE AHEAD IN THE COUNT
2010-04-23 - Premium Article Ahead in the Count: Methodology of The New M...
2010-04-22 - Premium Article Ahead in the Count: Reintroducing MORP
2010-04-15 - Ahead in the Count: Labor Market Discriminat...
2010-04-08 - Premium Article Ahead in the Count: Projecting Free Agent Pe...
2010-03-25 - Premium Article Ahead in the Count: Predicting BABIP, Part 3
2010-03-24 - Premium Article Ahead in the Count: Predicting BABIP, Part 2
2010-03-23 - Premium Article Ahead in the Count: Predicting BABIP, Part 1
More...

INCOMING ARTICLE LINKS
2010-12-16 - Premium Article Transaction of the Day: A Tiger Beat Addendu...
2010-04-27 - Premium Article Ahead in the Count: Ryan Howard and the New ...
2010-04-23 - Premium Article Ahead in the Count: Methodology of The New M...
2010-04-22 - Premium Article Ahead in the Count: Reintroducing MORP