Notice: Trying to get property 'display_name' of non-object in /var/www/html/wp-content/plugins/wordpress-seo/src/generators/schema/article.php on line 52
keyboard_arrow_uptop

Let’s play a game called “Which one of these is not like the others?”

Culled from Fangraphs, Baseball-Reference, and yours truly, defensive ratings for Brett Lawrie:

Metric

RAA

DRS

21

UZR

6.6

TotalZone

7

FRAA

7.0

So.

Let’s say that UZR, TotalZone, and FRAA all give Lawrie credit for seven runs. That’s 14 runs saved that DRS credits Lawrie for that none of the other metrics do. The difference between DRS and the field is double the number of runs that the other metrics assign to Lawrie.

The two metrics that are most comparable in this case are DRS and UZR, as they both rely on the same batted ball and hit location data provided by Baseball Info Solutions. So how do the two metrics come to such radically divergent views of Lawrie’s defense?

The new “new Moneyball” is putting your fielders in really weird places. In Lawrie’s case, the Toronto Blue Jays are employing a new kind of shift, where the third baseman is the only fielder to shift, and boy does he shift—all the way into short right field. An example:

We don’t have to speculate that this shift is affecting his defensive rating—Baseball Info Solutions confirms as much in a May 11th article (which, for reasons I cannot fathom, calls DRS DSR and BIS BSI and Jedlovec Jedlovic):

A significant factor in the Jays' top DSR rating is their frequent use of an over-shift against notorious pull hitters. Through May 3, Toronto had used a shift 47 times and saved eight runs as a result, [BIS research analyst Ben] Jedlovic [sic] said.

Five of Lawrie's 12 runs saved came when he moved to short right field in a shift against a left-handed batter such as David Ortiz or Carlos Pena.

Those three bunt singles that Pena dropped toward third base against the shift on April 17 at the Rogers Centre? They hurt Lawrie's plusminus rating. The DSR formula credits him for being in the right spot on the shift, but penalizes him for abandoning his post when Pena foils the shift with a bunt.

Another respected defensive metric – Ultimate Zone Rating (UZR) – also uses BSI's data but does not give credit for runs saved by shifts.

Jedlovic said some defensive experts believe good coaching and strategy are more responsible for runs saved by shifts than player performance. But BSI believes in crediting the player.

"We've elected to leave it in there because it shows that when guys are shifting, they're saving runs for their team," he said.

So the plays Lawrie make “save runs for the team,” by DRS’s reckoning. But do they really? And (more importantly) are they saving as many runs as DRS says they are? (And why are FRAA and TotalZone, metrics which do not exclude shift plays, not crediting Lawrie for the extra shift plays that DRS seems to?)

First, we want to know how many plays Lawrie has made while in this shift. BIS does not, to my knowledge, publish such information through Fangraphs, Baseball-Reference, or Bill James Online. BIS gets their data by watching telecasts of games, though, so anyone with an MLB.tv account and a goodly chunk of spare time can replicate their methodology. In order to make the task manageable, I used our database to produce a list of all at-bats by a left-handed hitter where the ball was fielded by Lawrie. Then I watched them. And I counted how many times Lawrie fielded the ball in short right field (I ignored any plays where a more conventional shift may have been employed, although I believe there were only one or two of those anyway). And now, the results of that trial:

Batter

Shifts

Shift Plays

IF Shift Plays

Adrian Gonzalez

2

2

1

Carlos Pena

3

2

2

David Ortiz

1

1

1

Erick Aybar

1

1

1

Kendrys Morales

2

2

1

Luke Scott

6

4

3

Matt Joyce

3

3

2

Ryan Doumit

1

1

1

Grand Total

19

16

12

Shifts is number of at-bats where Lawrie fielded a ball from the shift, shift plays is the number of plays made (defined as a putout or assist) on those balls, and IF shift plays is any time a play was made on a ground ball. Maybe BIS’s totals are a little different than mine, based on how they classify “shifts” and “ground balls” and such, but it should be pretty close.

So depending on how BIS counts those non-GB shift plays, that’s roughly 12-16 plays for Lawrie in the shift, plays we know that UZR is ignoring. Since DRS is denominated in runs and not plays, Lawrie would have to be getting credit for darn near a whole play above average for each of his shift plays for this to explain the bulk of the difference between DRS and UZR (which, again, gives Lawrie zero credit for any of his shift plays). Do we think that DRS could be doing this? And do we think that DRS should be doing this, if so?

Here’s what The Fielding Bible Volume III has to say about the methodology behind the Plus/Minus ratings, the part of DRS that deals with fielding balls in play (it’s on page 441, if you want to follow along):

Let’s say we’re watching a Dodger’s game, and Matt Kemp is patrolling center field. We might estimate that a particular play is made by the league’s center fielders 42 percent of the time… If Kemp makes this play, it’s a nice play. A sizable 58 percent of center fielders did NOT make this play. We reward him for this; specifically, we reward him 0.58 Basic Plus/Minus points (or “plays”). If Kemp does not make the play, we penalize him according to the proportion of fielders who did make the play, 0.42. In this case, Kemp would get -0.42 Basic Plus/Minus points on the play.

Please note carefully—the estimate is made of what percentage of center fielders would make the play on the ball in play, given the input parameters. The Fielding Bible also notes that:

A play is considered “not made” when the fielder doesn’t get a putout or assist on the play AND no other fielder had the first putout, assist or error. For example, if the third baseman fields a groundball towards the third base/shortstop hole, we don’t count this play for or against the shortstop. We assume that he didn’t have an opportunity to make the play and award no credit or penalty.

So let’s play around with this basic framework and see how far it takes us. It helps to have some sample data to play around with. Again, we don’t have access to the raw data underpinning DRS. For the purposes of illustrating this point, however, any zone data should do, and the most convenient source of that is from Project Scoresheet. This is what the Project Scoresheet field diagram looks like:

Project Scoresheet field diagram

I have consolidated the “S” and “D” zones with the primary zone, so for instance 5D is lumped in with 5—this way we pay attention only to the horizontal spray angle of the data. So here are third baseman play made rates on ground balls for each of those zones, from 1995 (no particular reason for that year):

LOC_TX

PM_RT

5

0.5889

56

0.3506

6

0.0220

6M

0.0031

4M

0.0017

4

0.0013

34

0.0022

3

0.0055

Shockingly, you see most of the plays made in the zone where the third baseman stands, fewer in the hole, and then almost no plays in the remaining zones. (We can talk about how range bias affects this, but for the purposes of this illustration, it shouldn’t matter much.)

So let’s say that there’s a grounder hit through the 34 zone, and Lawrie makes a play. Given these estimates, Lawrie would get credit for one play made, less the 0.0022 probability that a third baseman would make that play, or 0.9978 “Plus/Minus points.”

What this ignores is that Lawrie is not the only infielder. What’s the probability that any infielder would have made that play?

LOC_TX

PM_RT

5

0.7381

56

0.5124

6

0.8968

6M

0.5702

4M

0.5217

4

0.9246

34

0.5555

3

0.8493

In the particular shift the Jays are playing, you may note that Lawrie can get to only balls that have already passed by the other infielders. This is true, but the other infielders are positioned based upon knowing that Lawrie is backing them up in short right field. (And on one play, I noticed the first baseman giving up on a ball and heading back to cover first because Lawrie was going to be able to field it as well). Those balls that Lawrie is fielding from short right field are not the nearly-certain hits that you would assume from comparing Lawrie to a baseline of all third basemen, most of whom aren’t fielding balls in that position not because of a lack of skill but because they’re standing by third base.

This analysis would suggest that, instead of measuring how many additional plays made at the team level the shift is contributing, it’s measuring how many additional plays are contributed to Lawrie by the shift, crediting him for some plays that other fielders would have made in a more traditional defensive alignment. How many extra plays the Blue Jays are making as a team due to the shift is a more complicated question, which I’ll attempt to examine at a later point. It seems much simpler to see how the shift is distorting the Plus/Minus ratings.

There are, I think, two larger points at work here. One is that seemingly trivial details can have large impacts on the results of “advanced” fielding metrics. Even if we had the BIS data at the root of DRS and UZR publicly available, it’s not clear that we’d be able to duplicate either metric. And without some sort of standard to determine the validity of fielding metrics, there are cases in which it isn’t clear which fielding metric (if either) has made the correct decision.

The other is that what we should be attempting to measure with fielding metrics is not a player’s extra plays relative to his positional peers, but how many extra plays he is contributing to his team. This is an extreme example, but there are many more subtle interaction effects between fielders, and it is not clear that hit-location-based systems have any advantage in recognizing those effects (in this particular instance, it seems to put them at a distinct disadvantage). Obviously, comparing a fielder to his positional peers is one important part of such metrics, but we need to take care to reconcile such things at the team level, as well.


As a special bonus, raw data on the plays examined for the article. Enjoy.

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
Mooser
5/30
I never understood Dewan's reasoning for measuring shift plays the way the do. However, how does FRAA do it. I think you say that FRAA does not simply leave these plays out similar to UZR. Do you simply add it in to the pool of plays a third baseman makes in comparison to other third basemen. Does FRAA compare it to plays a second baseman makes given the positioning is closer to their typical zone?
cwyers
5/30
FRAA doesn't do anything different for shift plays than it does for any other plays - it counts all his plays as plays made, and figures his expected plays made based on team fielding performance. There's no location data, so it doesn't treat a play made over by the third base line any differently than one made in short right field.
canada
5/30
Interesting. I'm somewhat of a novice on defensive metrics. Is there a primer somewhere on the differences / pros and cons of each of the four methods you used?
blcartwright
5/31
The Hardball Times Forecasts has Lawrie at +5.8, right in line with all the non-BIS ratings. Although there are differences in the details, everything Colin just said about FRAA applies to my defensive metrics.

I'm of the opinion that for this kind of shift, with the third baseman in short right field and everyone one else in fairly normal positions (the Pirates have also been doing this) it might be better to not include these plays in the third base ratings, instead creating a new position. I don't want to ignore the shifts, but the calling the guy in short right a third baseman is really making things difficult.
nschneider
5/30
Excellent article. I look forward to your further article on how it affects overall team defence. Given how often I've seen Lawrie field these balls in short right field, he'll obviously have a somewhat inflated range factor. But as you point out, SOME of those balls would have been taken by the second (or first) baseman in a more traditional shift. It would be nice to know how many bunts have been successfully placed down the third-base lines in these situations, as a direct counterpoint to extra plays made due to the shift.
However, the Blue Jays are near the top of MLB in Deffensive Efficiency to this point.
canada
5/30
Shouldn't the "true" (word used very loosely) runs saved fall somewhere between 7 and 21?

While DRS is obviously giving him too much credit, by ignoring the shifts, the other metrics are not giving him enough credit. The increasing commonplace of shifting is clearly breaking the advanced defensive metric systems. FWIW, I have watched the majority of Jays games, and Lawrie has showed serious range when playing in shallow right. There are definitely some fractional RS to be awarded to him for plays in shallow RF that are not picked up when the defensive metrics ignore the shift.
doctawojo
5/30
I don't follow the NBA stathead debates as closely as I once did, but my understanding of one of the big problems / areas of study in NBA stats is in rebounding. Even just watching a game, it's easy to see that many rebounds are completely uncontested, and most are rebounds that any competent person of relevant size would grab. Separating the guys who get a lot of rebounds because their teammates let them have them from the guys who get a lot of rebounds because they're really good at fighting against the opponents for said rebounds is probably quite difficult, and at the same time quite important.

I don't really have a point, except to note the parallel to fielding given the issues Colin raises here.
jimspostmen
5/31
Great article. It was timely for me too. I was recently on baseball-reference.com, and I stumbled across leaders in defensive WAR. On the active career list, I was very surprised to see Brett Lawrie on the list. That turned to outright disbelief when I looked more closely.

http://www.baseball-reference.com/leaders/WAR_def_active.shtml

Lawrie is currently #88, at 3.8. Omar Vizquel tops the active list, with 28.2 - in over 2,900 games (1.6 over a full season). Even considering that per-game is a poor way to compare a youngster to a 20+ year veteran, Lawrie has 89 games coming into today - his 3.8 becomes 6.4 over a 150-game season. Nevermind that the best ever season on BR is "only" 5.4 - Terry Turner, 1906. At this rate, Lawrie would need about 4.5 full seasons to overtake Vizquel!

I have not watched Lawrie play much, but from what I read, he's a fantastic defender - the kind of guy I would buy a ticket to see when he's in town. But over 4 times more productive than Vizquel? I'm skeptical, to say the least. Perhaps Colin partially explains BR's WAR calculation?
blcartwright
5/31
The B-Ref list is Defensive WINS above replacement, not defensive runs above average.

Each win is worth about 10 runs, so Vizquel's 28.2 wins would be about 282 runs, assuming defensive replacement is equal to average. Over 24 years, that's about 12 runs a year
jimspostmen
6/01
I figured as much. I was wondering if it was less than 10, because 38 runs saved by Lawrie seems like a lot for a 90-game career.

But the ratio still doesn't change the fact that Lawrie is on pace to eclipse Vizquel's career number in less than 5 years. But if you subtract 14 runs saved due to the shift this year, and some more from last year, the result begins to make more sense, if B-Ref's WAR incorporates runs saved due to the shift.