BP360 now on sale! A yearly subscription, '23 Annual & Futures Guide and t-shirt for one low price

I give it one year until there’s a fundamental change to Major League Baseball’s new replay system.

I’d give it even less time if we weren’t less than three weeks from baseball season. While MLB certainly isn’t opposed to going seat-of-the-pants on this, the league probably won’t want to make another major change so soon.

But in a year, I’m counting on it.

What we’ve learned from watching the first two weeks of replay review trials in spring training is that the lack of a deterrent is going to be a problem. “Don’t overanalyze spring training stats” might apply here too, but we already saw this coming. Tuesday’s Mariners-Angels game saw the 21st challenge of spring training and the end to a streak that you could see as either embarrassing for the game or good for the umpires, who were 20-for-20 in upholding the other 20 challenged calls.

And why not issue challenges that are unlikely to be upheld? In their January piece at Baseball Prospectus, Dan Brooks and Russell Carleton showed that there’s almost no reason to hold back from challenging anything. The only deterrent is running out of challenges, which would require three close calls in six innings to work against you, and that never happens.

If baseball was looking for an encouraging soft launch of its product, this hasn’t been it. Not even so much for the sideshows or for Joe Maddon already trying to beat the system, because you knew there’d be some of that, but for the managers’ 0-for-20 start, which was also somewhat predictable.

So I will begin here where Dan and Russell left off in January and examine the possible deterrents to challenging every play that there’s even a minuscule chance of getting reversed.

In football, which was the first major American sport to institute challenge-based replay, there is a clear deterrent. Lose a challenge, lose a timeout. You get three per half, and they can be extremely valuable on a final drive, saving up to 40 seconds apiece. But there’s no real equivalent in baseball, so one has to get creative, and that’s what Dan and Russell did in their last paragraph.

We suggest an out, either added to the current inning or added to the next one. Even this penalty would strongly favor managers challenging plays that they were convinced had been called wrong, correcting obvious mistakes, which was the intended point of the challenge system in the first place.

An out is a lot. According to our stats, it works out to be worth, on average, 0.28 runs, which is definitely a deterrent. The obvious issue is how to handle the out in box scores and such. Baseball is prepared for something like this. There is a batting-out-of-order penalty of an out, which is credited to a pitcher who didn’t really record it.

But here, do you give the out to the next batter up and make him then go to the end of the line? Would that change whether a manager would challenge a play, not wanting to take a chance at burning a good hitter but calling an out for a bad hitter expendable? Or is it just a phantom out that leaves the batting order intact?

And would something worth a quarter of a run or more be too punitive, especially for a play where the only thing preventing a reversal might be a camera angle?

So here are some deterrent alternatives less drastic than a full out.

A ball or a strike
The natural place to go if you feel an out is too much would be to one ball or one strike on the next hitter. Sure this would unfairly punish a hitter or an individual pitcher, but not by putting him into a hole so deep that he couldn’t climb out and make it look like nothing happened.

But is it enough?

Again, Harry Pavlidis came through with some numbers for the ball and strike in addition to his stat on the value of an out. On average, going to 1-0 costs the pitching team 0.036 runs, while going to 0-1 costs the hitting team 0.043 runs, though this could fluctuate in specific cases depending on who the hitter is, how he does with two strikes, etc. (As an aside, these numbers are put together for some of the work we’ve done on catcher framing, hence the .08 maximum value for framing a ball into a strike in Table 2 of the Pavlidis/Brooks must-read on the subject.)

There’s a small bit of asymmetry there, which would make different sides slightly more or less interested in challenging, but the overall takeaway is the penalty. It’s about one-seventh that of an out, with much less messy housekeeping. I’d argue that if all you’re looking for is some deterrent, this is some deterrent without being overly punitive.

Losing a player
No, we’re not talking about a power play situation where you have to play with eight fielders, which would be something straight out of the Effectively Wild emailer dreamscape. But as Russell suggested, lose a challenge, and your opponent could pick a player on your bench who wouldn’t be able to come into the game. Is making the challenge worth perhaps losing your closer if it’s a close game?

My idea in this category comes from the not-too-distant future of the universal designated hitter, where the unsuccessful challenge would cost you your DH spot for the rest of the game. As the game got later, that would mean less and less, but it would still be a large deterrent, especially if you feared extra innings.

These things, like the timeout in football, are much less measurable in terms of run expectancy. But there is a problem you don’t run into with changing the count or the innings: Somebody could get hurt. The second you enact a move that leaves a pitcher in for an extra couple of batters because the team is minus its LOOGY, and the guy forced to stay in winds up on the operating table a few months later, it will be Hot Take City. And the second a reliever is batting because the DH has been challenged away and that reliever goes Chien-Ming Wang on the basepaths, that will be the end of accountability for your challenges.

Use the third category of challenge results
We consider a challenge to be of dual outcomes—you win it or you lose it. But the language and MLB’s system of keeping records don’t classify them that way. There are actually three possible results, and therein could lie another way to enforce this.

We obtained from MLB a copy of their records from this spring’s challenges through the close of play Tuesday, and they are actually divided into three categories: “Overturned,” “Confirmed,” and a third category—“Stands.”

Of the 23 challenges that got off the ground (one was nullified due to a power failure in the machinery), one was overturned, seven were confirmed with visual evidence to support the umpire’s call, and the other 15 simply stood, meaning that there was a lack of evidence to overturn it.

Within this separation is a chance to punish what we want to punish. Perhaps leave it as no penalty or just a ball/strike for a call that simply stands—you don’t want to be screwed if you’re right and there’s just no good angle to show that you’re right. And make a severe penalty like an out for plays on which the replay clearly shows confirmation of the call.

This would allow for the deterrent without being punitive when a play really is close and worth challenging.

Remove one challenge
This is the simplest possible deterrent. Right now, the thinking goes that the disincentive is the possibility of running out of challenges, but this is an extreme scenario that’s not really worth fearing, as the research showed. But if instead of two in the first six innings, it went to one that you got to keep if you were right, there really would be some second thought about frivolous challenges.


There is some very good news in the spring training data that we have: Instant replay hasn’t been much of a time-suck on the game. From the MLB document, we have good time data on 21 of the 24 challenges (omitting one with no timing because of broadcast problems, one where the timing was an outlier because of an ongoing pitching change, and the aforementioned power failure). The challenges so far have averaged:

35 seconds from call to challenge (standard deviation 19 seconds)

1:20 on the headset, (s.d. 31 seconds)

2:59 to the next pitch, (s.d. 37 seconds)

The ubiquitous comparisons of a challenge time to the argument time are extremely silly without making the leap of a 1-to-1 correspondence between times we have replay and times we would have had the argument. But we do know that if we assume that 2:59 to the next pitch represents a delay of about two minutes, we’ve lost about 48 minutes to challenges so far this spring. That’s over the span of exactly 200 games from which the data was pulled. So we’re losing between 14 and 15 seconds per game to replay, which is nothing.

One would think that in spring training, we’d see challenges only when it’s most egregious (except for the time when a manager was begged to do it). After all, in the regular season, challenging is no risk, high reward, and in spring training it’s no risk, no reward. Yet managers are 1-for-23, and while this is probably just a hot streak from umpires or some experimentation by the skippers, there will be downward pressure on the percentages caused by higher potential gains in the regular season, and still no consequences.

In the NFL, the coaches’ success rate in getting calls overturned is roughly 40 percent. Baseball’s doesn’t need to be that high to be effective, especially if umpires can get more efficient. But if it’s down in the single digits or the teens, and we get even short amounts of time wasted for what feels like nothing, then it will be time for one of those deterrents very soon.

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
I think the out option is way too extreme. I don't want to see a 26-out perfect game with a free out from a stupid challenge.
I don't think anyone would challenge anything.
You'd challenge if you thought the play was:

...called wrong with a high enough probability that the game needs to be stopped to get it fixed and
...important enough that the game needs to be stopped to get it fixed...

which is ideally the point of replay review.
I agree. Make it terrifying to fail on a challenge and keep them to a blessed minimum.
I'm not sure if I'm reading it correctly, but you list the following:
1 out ~= 0.28 runs
1 ball ~= 0.36 runs
1 strike ~= 0.43 runs
Either ball or strike ~= 1/7 of an out.

Did you mean ball ~= 0.036 runs and strike ~= 0.043 runs?
Fixed, thanks.
Yes, 0.28 for an out, 0.036 for a ball and 0.043 for a strike. Thanks. -Z
It's taking way too long for the challenges to be reviewed. In the NFL fans watching the game almost always know the correct call before it goes to commercial. The NFL doesn't care that it takes two minutes because they're the NFL and they get that much more commercial time. Baseball - that's not gonna cut it. Make. A. Decision.
ThisThisThisThisThis. A 'challenge' commercial in the NFL just replaces a commercial that would otherwise have been run after a punt. A commercial instead ran during a challenge, now instead they can start up play right away. Baseball you can't do that.

The second replay, I guarantee many casual TV viewers will click away just to avoid it. And a few will then stay with whatever they click to.
Yes, I hate waiting for the challenge in the NFL and I have really, really hated it in spring training. The game is slow enough already. This needs to be done with close to zero seconds wait.

But I guess I'm also not of the opinion that it's such a crime against humanity to have human error and its consequences out on the field. Especially when you are talking about 162 games. It comes out in the wash. If you can't stand winning or losing a game here or there on bad calls you probably don't have the patience for baseball anyway.
I hate the idea of winning/losing a game on a bad call, and I'd wait all day if it meant the umps got the call right.
2 simple solutions:

1), once you lose a challenge, you're out of them. The point is to appeal a play you've reason to believe they got wrong, not double check anything you want to.

2), every wrong challenge costs the franchise a thousand bucks, or some amount more if you think that's not enough. Billionaire owners became billionaires in the first place by really really caring about money. No, they're not going to just wave away the onfield manager pissing it away. Everybody's on some kind of budget when it's their money.
Is there any possibility MLB has asked managers to just use the challenge whenever there is a very close play just to exercise the system and possibly work out any kinks in the process? It would make sense for them to do that.
This apparently has been happening:
The Maddon article is fascinating. (Example: Runner on second, two outs. Batter grounds out on a close play. Maddon is drilling the runner to run hard and try to score while the defense is napping in case he wants to challenge the call at first.)

Of course, the counter-Maddon strategy has defenses running around tagging random runners after the third out has been signalled by an umpire, just in case.

These are clear incentives inherent to the replay system, but it also raises the risk of an injury (to either team), looks unsportsmanlike, and it will be confusing to fans.

Under the current system, a "close play" can be defined pretty loosely, because you lose almost nothing for trying. Plus, the runner is probably not going to get much of a glance, so he may try this even if the runner was out by a full step.
Why not a given number of challenges for the season? Do some decent analysis of how many calls were actually argued last year, add 10%, and call that the number of challenges you get. Or come up with some other method for figuring the number, but limit the number pretty carefully. Bet no one uses a single challenge in the first half of the year, and yes there might be some egregious challenges at the end (you can still of course limit the per game challenges in September, or add an umpire's discretion similar to the 7th through 9th innings now), but you'd accomplish what the per game limit is suppose to accomplish without creating the incentive to slow down the game stupidly all season.
Award a ball or strike to the other team, but allow them to use it whenever they like. Or during the next inning.
I guess I'm missing the reason for needing a deterrent. At most there's going to be four challenges a game, and as you point out, they're happening rather quickly. So...what's the problem? Just because the managers started off 0-20 doesn't really mean anything either. Everyone's just getting adjusted to the system.

Running out of challenges is deterrent enough. If there's a close call at first base in the first inning I bet many managers would hold off challenging that for fear of not having one available for a close call in a close game in the 8th.
No challenges after the 6th inning, so the "threat" of running out in the late-game is not a concern. The reason is that without a deterrent, there are clear incentives for managers to make spurious challenges that waste everyone's time and detract from the game.
re: the Maddon article, while I applaud him instructing his team to play hard to their fullest advantage, wouldn't the umpires apply a form of "dead ball" reset to the game play? Same way a ground-rule double isn't, in fact, automtically a double, but the umpires awarding base-runners bases using their judgement of where they would have normally advanced (not that they ever award a speedster a triple, but still - per rules it's their judgement to place runners).

So, for instance, runner on second, two outs, upon review batter is safe on what had been ruled a ground out at first, runner from second awarded third, not home.

By Maddon's logic EVERY time a base-runner is called out at first to end an inning you should not just challenge the call, but the runner should continue to second (or beyond) if the defense leaves the field. Just keep running Forrest Gump style.

That falls a lot closer to the "cluster-f&*$" side of things than it does to being good ol' fashioned hustle.
I can see where this is going: first people can't bear all the incorrect calls, then they cant bear to sit through the replays. Perhaps the solution is keeping them to a minimum, not to save time but so that the plays being reviewed are important enough to keep people's interest.
Challenging a call also seems like a good way to get your reliever a few extra warmup tosses in the bullpen.
No challenges after the 6th so that shouldn't be much of an issue.
Maybe, maybe not. Two-thirds of MLB starts last year lasted fewer than seven innings.
I say award bases. I haven't done the thought experiment out to imagine the ways it could be abused, but then, the leading cause of problems is solutions.
I think punishing with outs or even balls or strikes is going too far. We'll start having to adjust metrics for players that happen to be involved whenever there's a challenge that goes wrong. And as much as i don't want the decisions of games to come about as a result of blown calls, I don't want them coming about due to erroneous challenges, either.

No, I think the best method is to reduce the number of challenges available if they fail. Or make it so none of that team's players can call for time in the next inning.

That's the beauty of the challenge system in football - while losing a TO does impact your ability to win in a small way, the reality is that it doesn't alter or change any of the actual performance on the field or change the score or components of the score.

Baseball's deterrent system, should it choose to implement one, should operate similarly.