I’ve been thinking about the way I use advanced performance metrics such as Michael Wolverton’s Support-Neutral pitching statistics. In evaluating a team, I’ll often quote aggregate data such as their relievers’ Adjusted Runs Prevented or their starters’ Support-Neutral Value Added as a way of showing how effective they’ve been, and as shorthand for what we can expect them to do going forward.

This can be deceiving. Of course I know that past performance isn’t an exact predictor of future results, and that a group of players can be over or under their heads in the short- or medium-term. That’s not necessarily the problem; the problem is that the aggregate totals have been compiled by a group of players who are not necessarily representative of the team at the present time.

For example, just last week I mentioned that the Expos’ starters ranked sixth in the National League in SNVA. That’s true, but it’s past performance, and it includes six wretched starts by four pitchers who aren’t going to be used again, as well as the good pitching of Claudio Vargas, who may not take the mound again this year due to a shoulder injury. The SNVA figure is useful in letting us know how the Expos’ rotation has performed to date, but to get an idea of its current quality, it’s best to look at the pitchers taking the ball every fifth day. Isolating the performance of the current five Expos starters yields an SNPct of .569, 20 points higher than the team’s seasonal .549 mark.

The Expos aren’t special in this regard. Twenty-five of the 30 major league teams have current rotations with better SNPcts than their seasonal numbers, which you’d expect in a game so fiercely Darwinian. The pitchers whose stats have soiled the aggregate numbers are back in the minors, on the disabled list or running for governor, and they’ve been replaced by better performers. The only notable exceptions to this are two teams who each lost their best pitcher: the A’s, with Mark Mulder‘s hip injury, and the Orioles, who traded Sidney Ponson. The Angels have virtually identical seasonal and current SNPcts, while the Braves have gotten just three extra-rotation starts and the Mariners none.

Who looks the best once you adjust for this? The following chart ranks teams by SNPct based on the five pitchers currently being used. In a few cases, four starters were used in calculating current rotations, usually because it was unclear who #5 is. (The perils of early-September performance analysis.)

Through September 1

                  SNPct       Rank      Season SNPct    Rank
Diamondbacks      .632         1           .596           2
Giants            .617         2           .539           9
Cubs              .600         3           .542           8
Royals            .590         4           .549           5
A's               .589         5           .598           1
Dodgers           .579         6           .551           4
Yankees           .578         7           .548           7
White Sox         .578         8           .562           3
Expos             .569         9           .549           6
Indians           .561        10           .501          14
Marlins           .560        11           .537          10
Astros            .536        12           .519          13
Mariners          .526        13           .526          12
Braves            .526        14           .527          11
Phillies          .524        15           .501          15
Red Sox           .508        16           .489          17
Mets              .508        17           .456          23
Cardinals         .506        18           .487          18
Pirates           .501        19           .495          16
Reds              .495        20           .393          29
Blue Jays         .493        21           .469          21
Twins             .483        22           .481          20
Orioles           .479        23           .482          19
Devil Rays        .471        24           .405          27
Angels            .463        25           .463          22
Padres            .461        26           .415          25
Rockies           .460        27           .450          24
Brewers           .456        28           .413          26
Rangers           .433        29           .382          30
Tigers            .427        30           .404          28

As I mentioned, everyone looks better this way, but look at the relative changes. The Giants, having added Sidney Ponson and Jerome Williams in-season, have the second-best rotation in the game. I cheated a little with the Cubs, who make their leap on the basis of Juan Cruz effective Thursday, a move that has yet to be confirmed. The A’s slip without Mark Mulder, the Orioles slide without Ponson and the Pirates move down without Jeff Suppan. The White Sox are third overall, but they haven’t improved the back of their rotation, while the teams around them are no longer using the likes of Chris George, Andy Ashby and Jeff Weaver.

I think looking down the list can provide some interesting information. The Indians have used the 2003 season to sort through their young starters, and appear to have been successful in finding ones who can pitch in the major leagues. The Mets, Reds, and Devil Rays are all getting better work from their rotations than their overall numbers would indicate. I wonder if this information–the teams whose rotations improved the most during the season–might be an indicator of future success, of teams who made the transition from using warm bodies to starting pitchers capable of contributing to winning teams.

This method isn’t without its problems. For one thing, the Royals’ placement on this list is out of whack, largely because their rotation for these purposes consists of three guys with 10 starts for them combined, and the good work of Darrell May. (I couldn’t discern their fifth starter, and I’m not sure Tony Pena himself knows.) Take that information with a grain of salt, but also consider that it illustrates a key point about the Royals: it’s hard to say what they’ll get out of their starters over the next four weeks. The data for the Reds, Indians, Rockies and Orioles all have a similar problem, as all those teams have made significant changes to their rotations in the past month.

I also ran into the question of what to do with multi-team pitchers. While it largely comes out in the wash–only Jeff Suppan‘s performance has been significantly different across two teams–whether to credit a pitcher’s current team with his past performance, to increase the sample size and more accurately reflect ability, was an issue. I elected not to, so as to avoid double-counting two-team pitchers, but I can see an argument both ways.

The key point, however, is that the aggregate data I’ve been using can be misleading. It’s not enough to cite a team’s rank in some metric–even well-crafted ones like Support-Neutral stats–as evidence of that team’s level of ability, not when rosters and roles are so fluid. The Cubs, with a huge difference between their rotation with and without Shawn Estes, are perhaps the best example, but the list above provides plenty of other reasons to do more than just run a finger down a list in doing analyses. It’s an important lesson for me, one of the biggest things I’ve learned this year.

Tomorrow, I’ll run this same analysis for bullpens. I think the differences will be even greater because of the damage a few bad outings by staff fodder can do to overall numbers, and therefore, the evaluation of a team’s relief corps.