February 6, 2015
PFM vs. the 'Experts,' Part Two (Pitchers)
Last week, I took a look at how Baseball Prospectus’s PFM did against the CBS, LABR, and Tout Wars expert leagues. Today, I will shift over to the pitchers.
Last week, I concluded that the expert leagues pay too much for the top hitters while the PFM pays too much for the guys at the bottom of the pile. I thought that if you are going to redistribute your money, the best way to do this would be to redistribute it toward the middle, not at the top. I was amazed at how reliable the PFM was at “predicting” the earnings curve for hitters and wondered if the same thing would apply for the pitchers.
Table 1: 2014 American League Pitcher Earnings and Salaries in Groups of 10
It probably seems confusing as to why all of the columns at the bottom of the chart have vastly different numbers. The PFM uses a $960 budget for pitchers ($80 per team) while my formulas use $1,020 ($85 per team). The reason that the CBS, LABR, and Tout columns are so different is because the PFM’s Top 120 pitchers and the market’s top 120 pitchers are not a 100 percent match. The PFM has the advantage of knowing that Nate Jones and Jarrod Parker are hurt so doesn’t place a bet on them. Likewise, since the $1,020 is derived from the Top 108 pitchers purchased by the expert leagues, the PFM manages to squeeze $23 more out of value out of its top 108 pitchers than the market does.
Gaps or no, the PFM does not have the same accuracy at predicting what the market will earn as it did with the hitters. In the first four groups, the PFM actually spends slightly more than the market… with fairly terrible results until Group 31-40. The PFM gets hyper-aggressive with players 41-60 and nails it, hangs back on Group 61-70, but then gets another final hurrah in Group 71-80 and once again nails it. The $49-89 gap between the PFM and the expert leagues makes this a difficult exercise in terms of predictive value, but does show that the PFM isn’t particularly good at gauging the market’s value curve.
Table 2: 2014 National League Pitcher Earnings and Salaries in Groups of 10
The PFM comes a little bit closer to “predicting” the actual earnings of the NL pitchers, but still isn’t particularly close in any of the tiers. The NL expert spending is more aggressive than it is in the AL in the top tier and the best pitchers—based on their earnings—are more predictable. The drop from 31-50 is precipitous, and unfortunately is where the PFM flexes some of its muscle.
So unlike with the hitters, the PFM isn’t particularly adept at figuring out the earnings curve for the pitchers. How does the PFM do on the best pitchers on a pitcher-by-pitcher basis?
Table 3: Top 20 Salaries, 2014 AL and NL Pitchers with PFM
For a description of what each column in Table 3 (and all subsequent tables) means, please read the descriptions from my retrospective player valuation series.
In the hitter table, the PFM annihilated the expert market, coming closer to the actual earnings 14 times, compared to four times for the experts (with two ties). For the pitchers, this is turned on its head, with the market “beating” the PFM 11-6-3, with ties on Cain, Darvish, and Wainwright.
In this case, this isn’t a particularly instructive exercise. The message on the corresponding hitter table was loud and clear: the market spends wildly on the most expensive hitters compared to the PFM and the actual results. For the pitchers, while there are some differences of opinion, there are more similarities. Nine of the 20 pitchers on Table 3 come within two dollars of their PFM dollar value and the expert’s average bid. 16 of the 20 hitters on the corresponding hitter table have a $6 or greater average bid than the PFM’s recommended price.
The market is paying for optimal performances from the hitters and is not factoring in regression. However, for pitchers the market is factoring in regression similarly to the PFM. As a result, two things are happening:
This second point is significant. We are conditioned to believe that the best pitchers disappoint and that the market is foolish to reach for the elite arms. However, at a minimum most of the elite arms returned or came close to returning their investment. Only five of the 20 most expensive arms lost more than three dollars. What gives the market pause is that those five losing pitchers all lost $14 or more. However, this isn’t a satisfactory explanation, since six of the top 20 hitters also lost $14 or more. The market isn’t willing to take the risk on the top pitchers despite the fact that the bust rate on the top pitchers isn’t any greater than the bust rate on the top hitters.
This is a Catch-22. The bust rate on the top pitchers isn’t as high because the market is unwilling to push them past their regression-oriented projections and is attempting to redistribute the money across the board. Since the market is ripping off the pitchers at the top (relative to the hitters), it stands to reason that the pitchers at the top would lose less money. The best pitchers are not more reliable than the best hitters, but rather less is expected from them on the whole.
With less variance on the top pitchers between the market and the PFM, what kind of pitchers would we expect to reside on the next table?
Table 4: 20 Least Favorable PFM Projections vs. Expert Bids, 2014
Only three of the pitchers from Table 3—Darvish, Kershaw, and Scherzer—make a repeat appearance on Table 4. Unlike with the hitters, there are a lot of pitchers in the middle where there is significant differentiation with the PFM and the market. It is a split decision on who “wins” this battle. The market comes closer in 11 cases, PFM comes closer eight times, and Darvish is the tie.
Given the diversity of the price points and the types of pitchers on Table 4, it is not a good idea to get bogged down in trying to find a definitive pattern. All of the closers were more a matter of timing than anything else; since the PFM uses the Friday before the regular season to generate a dollar value, it has an advantage in theory over the expert leagues, which drafted much earlier, particularly CBS and LABR. However, these aren’t all wins. CBS was the only expert league that purchased Chapman without knowing about his injury and came closest to his actual earnings. Similarly, the market wins on Feliz because of the early CBS/LABR bids.
As you might expect, what jumps out about the non-elite/non-relievers is that PECOTA is awfully conservative about this group. Some of this is likely hindsight talking, but a seven dollar average projection on a group of pitchers that earned $15 in 2013 seems awfully pessimistic. There were a couple of prices I didn’t like, but the PFM seems to be saying it hates everybody, go away, leave me alone.
Since the bids add up to $960 ($80 per team), we know that the PFM must have love for some players in its mechanical heart.
Table 5: 20 Most Favorable PFM Projections vs. Expert Bids, 2014
The PFM gets a gold star for keeping the faith in Felix, but on the whole it is putting a lot of money into some pretty questionable pitchers. Why the PFM is recommending paying $4 more on average per pitcher than what this group earned in 2013 while shaving $8 per pitcher off of the pitchers in Table 4 is something of a mystery to me.
Something the PFM has trouble doing is betting on continued, sustained decline. Hudson, Jackson, Kennedy, McCarthy, Peavy, Sabathia, Verlander, and Weaver all get raises from what they did in 2013. Only Kennedy really had a big bounce-back year, while Hudson, McCarthy (in the AL, not included here), and Weaver improved a little bit. Using a weighted model, the PFM is not just looking at 2013 but 2012 and 2011 as well and factoring these into the projection. Granted, a few pitchers are always likely to bounce back, but for some arms once they are diminished or done, this is the new normal.
The market isn’t necessarily going to be right, but last year is much more important than anything that happened before that. The market and the PFM both want to mitigate risk when it comes to pitchers, but the market is going to hedge even more on last year’s bums than the PFM.
The hitter scorecard was a virtual push. It would seem based on the charts above that the market did better the PFM with the pitchers.
Table 6: PFM and Expert Market: Cost and Earnings on Pitchers with +/- $3 differential
Huh. The market wins, but it isn’t the blowout you might expect from the first few tables. And the average salary of these pitchers and the PFM predicted earnings is only one dollar apart.
It isn’t this simple, of course. The PFM and the market nearly split on the pitchers the PFM strongly prefers. The NL experts get clobbered by the PFM and its pitchers of choice, while in the AL the experts blow out the PFM.
It is difficult to infer too much from the data, but in the AL 15 of the 29 pitchers the expert leagues are bullish on earn $15 or more. In the NL, only nine of the 28 pitchers the expert leagues love earn $15 or more. Because the average ERA and WHIP are lower in the National League, it is harder for the best pitchers to earn a lot of money relative to the amount of money spent on pitching in auction leagues. So the big investment is kind of a waste in the NL, and a wise way to spend money in the AL.
I am at even more of a loss with the PFM when it comes to the pitchers than I am with the hitters. Because the market doesn’t “overspend” on the best pitchers, it really is a battle of preferences between the market and the PFM. And on the whole the market wins, albeit barely.
One of the biggest challenges with the PFM is when it places big bets on pitchers. Even though Felix at $33 was an excellent call by the PFM, when there is so much variability at the bottom of the food chain (and the PFM itself has so much trouble specifically predicting the variability), it makes sense to push the top pitching prices down even further, lest you wind up with Verlander instead of Hernandez. The money has to go somewhere, but I prefer the market’s idea of an even distribution. Take the money from Wood and Jackson and give it to some of the pitchers who the PFM puts a $3-4 bid on, even though those pitchers earned $8-10 last year.
In other words, when the results are close, I prefer to do what I have been doing for years: make educated guesses and simply avoid chasing non-elite pitchers into double digits when I can.