There’s been a goodly amount of grousing on the subject of bullpen performance in the early going, and for good reason. Setting aside standard-issue Bronx hysteria, several projected contenders-and not just the Yankees-are getting horrifically bad, hope-crushing setbacks from their high-profile relief crews. In some cases, as in Cleveland, this is a repeat of past problems; in the case of the Angels or the Cubs, it’s a matter of seeing deliberate off-season changes from one successful closer to someone else blow up in their faces in the early going. A few high-profile relievers struggle in high-profile situations, folks get angry, and we think we know what’s going on. But in evaluating bullpen performance, we end up leaving a few things out: specific, critical situations, how managers adapt to them, and the pitcher usage patterns that skippers fall into.

I would suggest that, in assessing what’s amiss with a bullpen, on some level we’re conditioned as fans to interpret performance in a particular light. Perhaps that’s by box scores, and perhaps that’s the product of the selective ownership pressures that come with fantasy baseball, but we can tend to evaluate bullpen performance on the level of individual relief performances: saves earned or blown. Perhaps some middle-innings scapegoat singled out for special attention by repeated self-immolation on the mound; holds represent leads held and handed off as a proxy for saves to reward the set-up men who, before the ninth inning, can only blow saves, not log many/any of their own.

Perhaps another rung up in looking at pens is looking at unit-wide performance with more advanced aggregating metrics. At Baseball Prospectus, we like to use context-dependent stats like WXRL or Fair Run Average or Adjusted Runs Prevented for relievers, since those end up capturing how well or badly a bullpen does in adding or preventing runs and (by extension) wins. That can lead to a more involved MLB-wide snapshot, where we can tear out something specific, like how the pens in the AL East are doing:

Bullpen        IP     WXRL     FRA     ARP
Red Sox       110.1  4.834    3.43    21.8
Blue Jays     110.2  1.907    3.92    16.5
Orioles       119.2  1.404    6.09   -11.5
Rays          103.1  0.614    4.55     9.9
Yankees       104.2  0.571    6.60   -13.5

Now, that might seem a bit confusing: we know that the Yankees’ pen is struggling, but what does it mean that the Orioles aren’t all that much better, but have a much better WXRL mark? How is it that Tampa Bay is doing worse than Baltimore in some ways? Any explanation is in danger of being only as good as the tools you use to illustrate these problems of performance. These tools are good at boiling down certain issues: Yankee relievers have been awful at preventing inherited runners from scoring, so their FRA is terrible; the Rays’ middle men have generally done well enough, but a large number of blowouts in their ballgames have limited opportunities for the pen to rack up results in terms of WXRL.

These facts reflect that opportunities and situations are not distributed equally, and they’re mute on the subject of the manager who has to respond to them. This seems strange, because pitching staff management in general and bullpen management in particular is one of the most important responsibilities a skipper has, often in the most critical situations. We look generally at individual or unit performance, and ignore situational management, to see how a manager and a unit resolved a specific situation. One way to do that is to address how a team handles what I refer to as “transition innings,” or those innings when the starter starts a frame but has to come out of it, handing off to the bullpen. This might seem relatively simple as a concept, but keeping the other team from putting up a crooked number on the scoreboard in this kind of high leverage situation is as much a matter of skill as the relative talent of the relievers involved-anticipating the situation, having the right men already up in the pen, and hooking the starter before the bleeding gets too bad.

So, let’s take a look at another chart, the best and worst teams in the division in transition innings, reflecting the disparate opportunities and performances:

Bullpen     G     RA     R/9   AL Rank
Rays       21     28    12.0     1
Red Sox    12     16    12.8     2
Blue Jays  12     22    16.5     6
Orioles    16     35    19.7     9
Yankees    17     53    28.1    14

What comes through here is a reflection of certain managerial preference. As we all saw last October, Joe Maddon’s not afraid to go to his middle men mid-inning, and he gets exceptional work from them; one can debate which is the chicken and which is the egg, but that’s just how the Rays roll, and it’s been working for them. In contrast, Terry Francona and Cito Gaston haven’t liked bringing their relievers into these sorts of situations, instead letting their starters finish frames that they’ve started; they’ve gotten good enough results. Even Dave Trembley’s selections fall short of disastrous, however bad the pen has been overall. The polite thing to say about Joe Girardi’s group is that they’ve been awful, but this isn’t very different from what we saw from his team in this situation last year: just as the Rays were the best team in the league in Transition Inning performance by allowing only 10.4 runs, the Yankees were 11th and allowed 20.5 runs.

How a manager adapts to a situation-and if he consistently fails to handle it well-is something that can lead to something worse than just high-profile failure and a tough post-game press conference. It can provide the sort of obvious, individual events that get guys canned, as GMs hunt for someone who might have a better sense of how to use the pieces in a pen to better effect. Last year, at the very end of a pennant race, Ned Yost wasn’t fired for his staff’s WXRL, he was fired for the perception of how he used his bullpen; his team’s mediocre performance in Transition Innings (ninth in the NL) reflected that. In defining both team and manager performance, we don’t always need numbers that are cumulative, but ones that are more specific, talking about specific opportunities-and specific results.

A version of this story originally appeared on ESPN Insider Insider.