“Defense to me is the key to playing baseball.”
Willie Mays

Accurate evaluation of defense remains one of the holy grails of performance analysis, mostly because, from a historical perspective, the way defense has been measured has been stagnant; the same six official fielding stats (games played, total chances, putouts, assists, errors, and fielding percentage) adopted by the National League in 1876 are the same six in use more than 100 years later.

Fortunately, baseball is a game where its structure naturally allows for more rigorous defensive quantification, even if that nascent characteristic was not fully exploited in the past. By assigning players areas of responsibility on the diamond through the codification of positions, each fielder is accountable within his domain when the ball enters it. A good defensive player is then one who is successfully executing plays within his area, or even expanding that area.

In the past quarter of a century, analysts have begun to take advantage of those simple ideas, leading to the development of Bill James’ Range Factor (RF), Pete Palmer’s Fielding Runs (FR), our own Clay Davenport‘s Translations (DTs, which include FRAA and FRAR), Mitchell Lichtman’s Ultimate Zone Rating (UZR), ESPN’s Zone Rating (ZR), David Pinto‘s Probabilistic Model of Range (PMR), Michael Humphrey’s Defensive Regression Analysis (DRA), Shane Jensen’s Spatial Aggregate Fielding Evaluation (SAFE), Baseball Info Solution’s Plus/Minus system, and even Tom Tango’s Scouting Report for the Fans.

With that cornucopia of acronyms to choose from, the discussion within the analytical community has shifted from whether and how defense can be more accurately quantified, to the strengths and weaknesses of the various systems, including combining the metrics to create an ├╝ber-fielding statistic that incorporates the strengths of various metrics. This week–spurred on by some thinking about the Twins/Rays trade involving Delmon Young–we’ll make a small contribution to the ever-growing body of work on the subject of defense.

Armed and Dangerous

In the excellent Prospectus Challenge column that Joe Sheehan and Kevin Goldstein contributed in the wake of the Delmon Young deal, Joe made the point that despite a credible rookie campaign, Young’s game still contains persistent flaws. Among other more obvious problems–such as his lack of plate discipline–Joe mentioned that he is not a good defensive player. This caught my eye, since I recently penned an essay titled “Expanding the Cannon: Quantifying the Impact of Outfield Throwing Arms” for this year’s annual, in which Young was a bit player. Although I’ll leave the details for the annual’s publication, suffice it to say that the ability of an outfielder to prevent runners from advancing–as well as throwing out runners attempting to advance–can be quantified in much the same manner as baserunning.

In Young’s case, while FRAA has him at -8 for 2007 and -11 for his career, he came out at +9 runs due to his throwing in 2007, and +13 in his two seasons (which cover a total of 155 adjusted games, or equivalent nine-inning games, in right field). Covering 2005 through 2007, that total ranks him fourth among major league right fielders, behind only Jeff Francoeur (+25), Alex Rios (+19) and Michael Cuddyer (+15), but first in terms of runs saved per opportunity for right fielders playing in 100 or more adjusted games. Lest you think that this happy result and 20 career assists may be due to runners not being aware of his ability, he also ranks third in lowest advancement percentage (a measure of his reputation) for right fielders who’ve played in 100 or more adjusted games at 32 percent, trailing only Shane Victorino (30 percent) and Alex Rios (28 percent). The end result is that when you factor in the runs he’s saved with his arm, he comes out as an above-average defender,although not by much. Whether he can continue to put up those kinds of numbers in Minnesota remains to be seen.

But thinking about Young’s qualities led me to wonder how difficult it would be to construct a fielding system for outfielders that integrated throwing with range and sure-handedness to get to a composite fielding metric (yes, many of us love “the one great number,” although I realize there is real value in keeping the components separately available). Although not generally interesting to you the reader, as a software developer that led me to considering in just how few lines of code a credible system could be created. So last Friday night I started down the path towards doing just that using our available play-by-play data for the 2005 through 2007 seasons. Although I didn’t get as far as completing the system for outfielders (a task left for a future column), I did complete a first draft (or beta version, if you will) for infielders. For the remainder of this column, we’ll walk through the system, show a few results, and discuss how it stacks up against other systems.

Simple Fielding Runs

As sabermetric-savvy readers will be well aware, the available fielding metrics are generally based either on zone and vector data from the likes of STATS, Inc. or Baseball Info Solutions (BIS) or else on conventional fielding statistics. Examples of the former include UZR, ZR, PMR, and SAFE, while RF, FR, DT, and DRA make up the latter. While the advantage of the zone- and vector-based statistics is obvious–because they consider more precise hit location and speed data–metrics based on more traditional statistics have the advantage of being able to be computed over the entire history of baseball, and in the case of the DTs, between different levels of competition, major, minor, and foreign. In comparison, the more complete data from STATS and BIS only go back a handful of years.

I’m not aware of any system based solely on Retrosheet-style play-by-play data minus the zone information. Play-by-play records typically include hit type (line drive, fly ball, grounder, and popup) as well as the fielder who fielded the ball, but not necessarily the zone in which the fielder was operating. Developing a system that works using this kind of data has the potential benefit of extending it further into the past than zone data can reach–especially as that data becomes more complete–and would allow for analysis of minor leaguers for whom we have play-by-play data but no zone and vector information. In addition, a system like this has the advantage of being based on data that is more widely available, and so could be tweaked by others in an open-source model. In the end, both the analyst and the coder in me were curious as to whether a coarse-grained data set could deliver a decent fielding system.

In essence, the system is based on two fundamental calculations. First, for the season and league in question we develop a matrix that specifies how frequently and at what cost balls of different types hit “near” a fielder were turned into outs. So for example, as shown in the table below, for shortstops during the 2007 season ground balls were turned into outs (hits and errors are treated identically, with the final destination of the batter included in the calculation of total bases and total bases per runner) at a rate of 61.8 percent; each ball not fielded cost the team 1.11 bases. As you can see from the table, ground balls (which also include bunt grounders) and line drives to third base that were not fielded were more costly than those hit towards shortstop:

Positional Area of Responsibility for 2007
Pos    HitType     Balls  Runners      TB    Out%  TB/Runner
Third     G        14773     5427    6264   .633       1.15
Third     L         7399     6672    8747   .098       1.31
Third     P         1888       33      40   .983       1.21
Short     G        20358     7768    8597   .618       1.11
Short     L        10515     9663   12459   .081       1.29
Short     P         2595      106     126   .959       1.19

The key problem faced by a system built on such coarse-grained data is in determining just what constitutes a ground ball or line drive that a fielder is said to have responsibility for. Since we’re not using actual zone information, this has to be done through the proxy of the fielder who fielded the ball, combined with the hit type. In other words, although the play-by-play data only records which fielder ultimately picked up the ball, we can include in the virtual “area of responsibility” balls fielded by other fielders in the vicinity of the fielder we’re interested in. For shortstops, this means that all ground balls fielded by the shortstop and left fielder are considered as well as half the ground balls fielded by the center fielder. Similarly, all line drives fielded by the shortstop, those that are recorded as hits fielded by the left fielder, and half of those recorded as hits and fielded by the center fielder are assigned to the shortstop. In this way, what we’re doing is approximating rough zones that can stand in for us without the benefit of actual location information. In this way, we partition all the balls put in play, excluding home runs, based on fielder and hit type using a simple set of rules to produce data, as in the table above.

The second step is to make the same calculation for each fielder and compare it to the matrix. It should be noted that since the matrix represents the performance of all fielders in 2007, there are no park factors created for infielders which, incidentally, is one of the reasons outfielders are not yet included, since creating factors is mandatory to get decent numbers.

For a pair of examples, let’s consider this year’s top pair of rookies in the National League, Colorado shortstop Troy Tulowitzki and Milwaukee third baseman Ryan Braun:

Troy Tulowitzki, Area of Responsibility for 2007
Pos       HitType  Balls Runners ExRunners    Diff   Ex2B    Ex1B     SFR
Short     G          737     248       281      33      3      30      26
Short     L          338     312       311      -1      0      -1      -1
Short     P          104       4         4       0      0       0       0
                    1179     564       596      32      3      29      25

Using the simple partitioning rules, the table shows that we’ve assigned 737 ground balls to Tulo’s area of responsibility; of those, 248 resulted in the runner reaching base. When compared to the matrix, we would have expected 281 runners to have reached base, a difference of 33. Again, according to the matrix, of those 33 we would expect roughly 30 of them to have resulted in the equivalent of a single and a little more than three of them to have been worth the equivalent of a double. To assign run values to these numbers we can consider each “single” saved to be worth +0.47 runs per Linear Weights’ Batting Runs, and each double to be worth +0.80 runs. But in preventing a runner from reaching base, the fielder is also creating an out, so we need to further credit him with the opposite run value of an out, which we’ve set to -0.27. When we multiply the “singles” by the resulting +0.74 and the “doubles” by +1.07 and add them together, we get the final result, tentatively dubbed Simple Fielding Runs (SFR), in the far right column. Total them up for each hit type, and I find that Tulowitzki was worth +25 runs on defense in 2007. Since the comparison we’re making is to a matrix that includes all shortstops in 2007, those 25 runs are above and beyond what an average shortstop would have done. For the sake of in-house comparison, it just so happens that FRAA has Tulowitzki at +24 runs.

Although this system certainly creates implicit dependencies between fielders (a second baseman who goes to his left well, for example, will cause fewer ground balls to make it to right field, thereby reducing the virtual area of responsibility for his first baseman), even an excellent shortstop like Tulowitzki can’t necessarily hide a weaker infield partner, such as Rockies third baseman Garrett Atkins; Atkins came out with an SFR of -12 runs. In addition, some readers will immediately see that in addition to accounting for hit type, the additional context of batter handedness would also be good to include since the expected outcomes for ground balls from left-handed hitters differs from that of right-handers, depending on the direction in which the ball is hit. The same argument can be applied to differentiating bunt grounders from other ground balls.

Let’s move on to Braun.

Ryan Braun, Area of Responsibility for 2007
Pos    HitType     Balls Runners ExRunners    Diff   Ex2B    Ex1B     SFR
Third     G          315     158       116     -42     -6     -37     -33
Third     L          157     149       142      -7     -2      -6      -6
Third     P           48       2         1      -1      0      -1      -1
                     520     309       258     -51     -8     -43     -40

At third base, Ryan Braun’s area of responsibility included 315 grounders, 158 of which were not turned into outs. We would have expected only 116 to have resulted in a runner, so Braun is debited for 43 baserunners. Since more balls not fielded by third baseman end up as extra-base hits, he is given 37 single-equivalents and six double-equivalents, which adds up to -33 runs. Sum the results, and Braun is debited 40 runs for his efforts in 2007, a whopping difference of 65 runs between himself and Tulowitzki. While not as close as in Tulowitzki’s case, Braun’s SFR compares reasonably well to the -25 value calculated by FRAA.

Before making aggregate comparisons of this metric to others, let’s take a quick peek at the leaders and trailers at each infield position in 2007 according to both SFR and FRAA. In the tables that follow, AdJG is adjusted games, and Balls are the number of balls in the SFR system we’ve assigned to the fielder:

Name                AdjG  Balls  FRAA   SFR
Tim Hudson          25.1    42     6     3
Livan Hernandez     23.1    38     4     3
Steve Trachsel      17.8    36     6     3
Sergio Mitre        16.8    30     4     2
Jeff Francis        24.0    30     1     2
Fausto Carmona      23.9    38     5     2
Shaun Marcum        17.9    27     3     2
Nate Robertson      20.0    27     1     2
Joe Blanton         25.9    28     1     2
Saul Rivera         10.5    26     4     2
Scott Linebrink      7.8    14    -1    -3
Scott Kazmir        23.5    22    -2    -3
Lenny DiNardo       14.8    21    -1    -3
Chuck James         18.0    22    -2    -4
Jose Contreras      21.3    30    -2    -4

The magnitude in terms of runs here for pitchers is very similar in both FRAA and SFR, although SFR tops out quicker. Because catchers and corner infielders typically have primary responsibility for balls hit in the direction of the pitcher, the area of responsibility for pitchers is restricted to balls actually fielded by the pitcher.

Name                  AdjG  Balls  FRAA   SFR
Carlos Ruiz          101.4    47     6     2
Ivan Rodriguez       117.9    43    10     1
Russell Martin       140.1    63    10     1
Jesus Flores          44.3    21     2     1
Brian Schneider      117.8    42     0     1
Yadier Molina         97.2    34    19     1
Miguel Montero        57.4    19    -7     1
Johnny Estrada       107.8    38   -10     1
Chris Iannetta        55.0    20    -1     1
Chris Snyder         100.2    28    12     1
Chad Moeller          12.8     9    -2    -1
John Buck            104.3    34     1    -1
Wil Nieves            18.9     8    -3    -1
Victor Martinez      115.5    25     5    -1
Jason Varitek        119.9    35     8    -1

Here the differences between FRAA and SFR are pretty extreme, although it should be remembered that SFR includes only a catcher’s fielding of batted balls, and not his ability to throw out opposing runners or block pitches.

First Basemen
Name                  AdjG  Balls  FRAA  SFR
Albert Pujols        149.6   636    22    26
Todd Helton          148.2   609    10    15
Casey Kotchman       116.6   420     9    13
Kevin Youkilis       123.2   448    17    13
Adrian Gonzalez      160.6   617    13    12
Justin Morneau       142.0   533    14    11
Lyle Overbay         108.8   427     5     9
Ryan Klesko           89.7   376     5     7
Mark Teixeira        123.5   494    -7     7
Darin Erstad          19.5    80     1     6
Ryan Garko           118.2   444    -1   -10
Richie Sexson        112.1   441    10   -10
Prince Fielder       150.1   581   -15   -13
Dmitri Young          99.1   361    -7   -15
Mike Jacobs          101.4   411   -11   -15

For the most part, the numbers here appear to be quite similar, with Albert Pujols on top in both systems. From 2005 through 2007, Pujols also had the second-highest SFR score (+20 in 2006). However, SFR rates Mark Teixeira as a much better first baseman than FRAA does, and thinks Richie Sexson is much worse. Although not shown, Lance Berkman is also inverted, as he rates a -8 in SFR, and a +7 in FRAA.

Second Basemen
Name                AdjG Balls   FRAA   SFR
Mark Ellis         147.9  1022    27    33
Aaron Hill         157.7  1066     0    32
Kazuo Matsui        95.6   614    14    28
Dustin Pedroia     128.6   767     2    17
Robinson Cano      157.3  1029    26    16
Marcus Giles       104.1   692     6    14
Brian Roberts      149.7   928     1    12
Alex Cora           33.5   183     1    11
Geoff Blum          54.3   347    -2    10
Ian Kinsler        128.8   906     3    10
Danny Richar        55.3   346    -8   -14
Craig Biggio       103.6   604   -17   -15
Freddy Sanchez     142.4   806    -7   -17
Rickie Weeks       110.4   680   -13   -24
Dan Uggla          155.3   981    14   -37

Once again, things generally line up, but there are a couple of substantial differences, particularly Aaron Hill, who gets a +32 in SFR but a 0 in FRAA, and Dan Uggla, who comes in at -37 in SFR but +14 in FRAA; Joe Sheehan noted how other systems tend to rate Uggla lower than the DTs. After Uggla, Jorge Cantu had the second-lowest score from 2005-2007 (-29 in 2006).

Third Basemen
Name                AdjG  Balls  FRAA   SFR
Pedro Feliz        136.1   668    14    29
Ryan Zimmerman     160.3   846    21    17
Scott Rolen        105.6   552    16    15
David Wright       158.2   777     5    14
Aramis Ramirez     122.2   638    17    14
Mike Lowell        149.2   684    14    12
Nick Punto          93.4   430     1    11
Abraham Nunez       66.0   384     8    10
Mark DeRosa         32.2   148     2    10
Joe Crede           43.7   227     8     9
Wilson Betemit      45.7   215    -5    -8
Garrett Atkins     146.2   678   -17   -12
Miguel Cabrera     147.2   735     5   -15
Edwin Encarnacion  130.6   662   -11   -15
Ryan Braun         106.1   520   -25   -40

Defenders with good reputations such as Pedro Feliz, Ryan Zimmerman, and Scott Rolen rise to the top in both systems, and it’s interesting to note that Aramis Ramirez had a superb season after a -10 SFR in 2006 and a 0 in 2005. Meanwhile, Braun can barely see Edwin Encarnacion off in the distance, while the systems disagree on new Tiger Miguel Cabrera. The only other SFR rating for a third baseman in the 2005-2007 time period that comes close to Braun’s -40 was the RoyalsMark Teahen at -31 in 2005, before they mercifully shifted him to right field.

Name                AdjG  Balls  FRAA   SFR
Omar Vizquel       135.9   941    10    38
Khalil Greene      153.3   990    -8    25
Troy Tulowitzki    152.3  1179    24    25
Jason Bartlett     134.7   900     8    21
Jose Reyes         159.8  1019     4    14
John McDonald       89.4   602    12    14
Tony Pena          143.6   988    12    11
Jimmy Rollins      160.1  1088     8    10
Bobby Crosby        91.0   635    -1     9
Rafael Furcal      135.3   945    16     7
Josh Wilson         51.3   336   -11   -13
Carlos Guillen     120.3   862   -12   -15
Derek Jeter        147.4   978    -6   -19
Brendan Harris      85.2   579   -13   -21
Hanley Ramirez     146.1  1045    -8   -24

At shortstop the two systems violently disagree on the virtues of Khalil Greene, and more than a little on Omar Vizquel, but are otherwise fairly consistent. With the long-running debates about the defensive contributions of the Yankees‘ shortstop, many readers will no doubt want to know how Derek Jeter turns out, and we find him at -19 runs in 2007. In 2006 SFR had him at -8, and in 2005 at -10. In the period 2005 through 2007 the highest single-season SFR rating at shortstop belongs to Adam Everett in 2006, when he scored a +41. On the other end of the spectrum, Michael Young scored a -38, and former Rookie of the Year Angel Berroa a -36, with both of those execrable seasons coming in 2005.

Get in the Zone

What we’ve seen in the above tables is that while in many cases SFR creates a similar rating to that of FRAA, in other cases it does not. But since FRAA is based on more traditional statistics such as putouts and assists and is not zone-based, we’d also like to systematically compare SFR to a metric that includes this more granular information. To do this, I ran a regression for SFR against the 2005 and 2006 UZR data for all infield positions except catcher and pitcher. The correlation coefficient was a healthy 0.75, with a sample size of 988 player seasons, a result that is easily statistically significant at the 99 percent level of confidence. The scatter plot below illustrates how strong the linear relationship is between the two measures:

chart 1

Quite frankly, the strong correlation surprised me somewhat, since SFR makes very simple assumptions regarding the zones or virtual areas of responsibility for each position. And when compared with the correlation table that blogger Justin Inaz put together for many of the defensive metrics, SFR would rate second behind only Zone Rating (also based on zones from STATS, Inc.) in its similarity to UZR, although the correlation here considers only infielders.

Even so, correlation isn’t the only tool by which you would compare the metrics, since characteristics of the distribution such as the range and standard deviation are also important to consider. For example, SFR has the highest standard deviation of the three metrics, by about a run, and a greater range than FRAA but more comparable to UZR. It should also be noted that the correlation coefficient remains at 0.75 when we look only at the 437 players for whom 100 or more balls were assigned to his area of responsibility.

The plot above can be contrasted with plots here that show how FRAA stacks up against SFR, and how FRAA compares to UZR:

chart 2

chart 3

All three comparisons show substantial agreement, and the correlation coefficient for the regression of SFR against FRAA was 0.52, while that between UZR and FRAA was a little lower, at 0.41. So we can interpret this to mean that SFR is more similar to FRAA than UZR, but that SFR is closer to UZR than FRAA.

Check Please

The end result is that it would seem my creation of a reasonable defensive metric (at least for infielders, and given the caveats mentioned for catchers) with comparatively little effort was a success. When all was said and done, this first tentative version required less than five hours of development time, and fewer than 950 lines of fairly repetitive code. Stay tuned for the sequel, where we’ll take a look at outfielders.