Among the attendees of the second annual SABR Analytics Conference, which took place in Phoenix this past Thursday through Saturday, were statistical analysts from several clubs; some whose names you’d know from Baseball Prospectus or other sabermetric sites, and others who’ve kept a lower public profile. But with the exception of Bill James, whose stature is such that he can continue to play a public role even from the inside, the team statheads weren’t at SABR to take part in panels or present PowerPoint slides. They were there to keep their eyes and ears open for any ideas or developments that might give their employers an edge.
They sat silently in the back rows of conference rooms, or clustered together outside the exits with other delegates from their own clubs, talking quietly or sending messages back to base with their omnipresent phones. Occasionally, one team’s cluster would meet and merge with another’s, chatting amiably like less athletic versions of opposing players crossing paths before first pitch. But even (or especially) among their own kind, their words were guarded: they talked shop without citing specifics. As Zachary Levine wrote last week after returning from the MIT Sloan Sports Analytics Conference, team employees tell few tales.
Team employees in baseball operations, that is. And for the most part, Baseball Ops employees are the only ones that the hundreds* of fans who flocked to Phoenix care about, or are even aware of. But they’re not the bulk of a baseball team’s front office. As Bill James described it during his keynote address on Friday, “A modern baseball front office has two almost entirely separate divisions: there is the business side and the baseball operations side…there are a lot more of them than there are of us.” As far as most fans (and maybe even most members of Baseball Ops) are concerned, team employees in the business development, corporate sponsorships, and ticketing and sales departments might as well be working in some less-exciting industry for all the connection their jobs share with the glamorous profession of evaluating, acquiring, and developing players. As James admitted, “I actually don’t have any idea what all they do.”
*According to SABR President Vince Gennaro: 430, to be exact, up from 275 last year.
So it came as a surprise that the most memorable address at SABR (in my mind) was delivered not by a former, current, or aspiring Baseball Ops employee or consultant, but by a member of a marketing department. It had an inscrutable title (“Business of Baseball Analytics: Marketing-Mixed Optimization: Agent-Based Modeling Approach”), and I stayed to see it in part because it was scheduled between the Brian Kenny/Bill James/Joe Posnanski “Analytics Super Panel” and Vince Gennaro’s presentation about “The Big Data Approach to Baseball Analytics.” Had it been held at some other time, or in some other room, I might have wandered off to see something sexier (or what passes for sexier at SABR). That would have been a bad choice.
When you think of the Cleveland Indians’ front office, historically one of the earliest and most open in its embrace of sabermetrics, some familiar names might come to mind: former BP authors Keith Woolner and Jason Paré, former writers/analysts from elsewhere on the internet Sky Andrecheck and Victor Wang, or former Cleveland consultant Russell Carleton. You probably wouldn’t think of Alex King. You probably should.
King has been the Indians’ Vice President of Marketing and Brand Management since July 2011; before that, he was a brand manager at Procter & Gamble and a product manager at The Bradford Group. On Saturday, King addressed the audience at SABR after an introduction by Damon Ragusa, the Chairman and Chief Strategy Officer for ThinkVine, a company that specializes in marketing planning and simulation. Despite the talk’s intimidating title, its subject was as simple as it was appealing to an analytics-inclined crowd: bringing sabermetrics to the business of baseball.
Think back to the days before Moneyball, when the state of player evaluation and the popular perception of what made teams win was much as it had been decades before, a murky mix of tradition, guesswork, and mistaken assumptions. Since then, we’ve seen front offices place an ever-greater emphasis on efficiency, accuracy, and the pursuit of sound process in putting teams together.
Baseball marketing hasn’t had its Moneyball moment, at least until now. Many teams decide which promotions to run (bobblehead or Beanie Baby?), how to advertise (radio or TV?), and when to advertise (shortly before a big series, or well in advance?) based on anecdotal evidence and their incomplete knowledge of what’s worked (or seemed to work) before. As one might imagine, inefficiencies are out there, ready to be identified and exploited by a data-driven approach.
There probably won’t be a movie about baseball marketing, and King won’t be played by Brad Pitt. But with ThinkVine’s assistance, he’s applying some of the same principles to marketing as the game’s new breed of progressive GMs has to Baseball Ops. And the payoff could be just as impressive.
The first obstacle to allocating marketing funds more efficiently is that it’s difficult to tell which tactics work, and to what degree. Say you run a particular promotion, and attendance is especially strong. Can you be confident that the increased attendance is tied to the promotion? Or might there be other factors at work, obscuring the promotion’s actual effects and potentially causing confusion between correlation and causation?
That’s where ThinkVine, which announced in January that it had landed the Indians as a client*, comes in. The Indians gave the company five years of historical data (2007–11) on attendance, ticket sales, and promotions, and ThinkVine used it to generate an agent-based model (as opposed to a more traditional econometric, or regression-based approach) that would help the team develop a “base probability” that an Indians fan would attend any given game. That base probability can then serve as the foundation of a forecast for attendance with one or more additional draws. In theory—and, King is confident, in practice, too—the model allows the Indians to quantify the effects of “exogenous factors” (team performance, weather, competing events, the Cleveland economy, etc.) and isolate the impact of marketing techniques.
*According to King, the decision to work with ThinkVine was prompted in part by former Indians General Manager and current Indians President Mark Shapiro, who was bumped up to that role before the 2010 season. I’ve always wondered why teams decide to promote good GMs out of a position where they’re already making a major impact. Presumably, it’s because it allows them to focus on both the baseball and the business sides of the game and make more of an impact on the club’s overall operations. Free from the day-to-day pressure of constructing a roster, Shapiro can seed other areas of the Indians with the sabermetric approach, producing even greater gains.
Armed with that information, the Indians are attempting to maximize revenue by optimizing their promotional schedule. King showed a bar chart displaying the effectiveness of each promotion, broken down by year. “Dollar Dog” days, for instance, were the biggest draw, better than bobbleheads or Kids Fun Days. But they weren’t necessarily the most dependable; Dollar Dog days did better when the Indians were winning, whereas fireworks nights were highly resistant to team performance: people apparently want to see fireworks whether their team is winning or not.* Nor were Dollar Dog days the most profitable; that distinction went to hat/cap giveaway days, presumably because the low cost of caps relative to the extra attendance generated offers the best return on investment. Not only can the Indians determine which promotions to schedule and when, but they can also arrive at the correct quantities for giveaway days: at the 15,000-item level, only hats are profitable; at 10,000, almost everything is.
*The fireworks finding was my favorite, because of my fondness for Hall of Fame former Indians owner Bill Veeck. Veeck was writing about the positive effects of fireworks over 50 years ago, and putting his ideas into practice even earlier. Using what passed for a scientific approach to promotions at the time, Veeck learned that fireworks would bring fans to the park even if a bad team was playing. He’d have loved to see the data I saw on Saturday, and he’d be even happier that the Indians were the source.
Veeck believed that good promotions had a compound benefit: they’d not only pump up attendance, but also increase concession sales. The impact of promotions on concessions wasn’t mentioned during King’s presentation, so I asked Ragusa via email whether they were considered. “We built the model around ticket sales but did all the backend ROI math considering merch and concession,” Ragusa said. “We are working on a project to build in more sense of lifetime value of different consumer groups in the model. This will include more direct/indirect impacts on both merch and concession simultaneous to ticket sales.”
We can already see the effect that information has had on the Indians’ schedule. The team will always vary its promotional offerings, both to keep things fresh for the fans and to provide more exposure for their corporate partners. But now the Indians are emphasizing the ones that work. King didn’t say how they’d make use of their new awareness of the profit-generating powers of headwear, but it’s easy enough to see. The team’s 2013 promotional calendar includes no fewer than four cap/hat giveaways. Last year’s didn’t have any.* That’s satisfying in the same way that seeing Jose Molina play in a career-high 102 games at age 37 is satisfying: it’s a decision we can explain from afar, because we know it’s driven by recently discovered data which even we outside observers are aware of. And to take the parallels to the sort of sabermetrics we’re used to even further, King presented a “heatmap” of the optimal days of the week for each promotion.
*Another tidbit: King revealed that it’s tough to break even on concerts. That surprised me, since the Rays run a summer concert series at Tropicana Field, and the Rays generally don’t do things that aren’t backed up by the data. Cleveland is putting on only one concert this summer.
The analytical benefits also extend to advertising. King found that 80 percent of visitors to Jacobs Field buy tickets within three days of the game. If that’s the case, it might not make sense to start promoting a series well in advance of its start. Digital spending is easy to change on the fly, so the Indians (who use dynamic ticket pricing) can ramp up their advertising right before a series to maximize their return. King showed two graphs of projected advertising spending over time, overlaid on average ticket price for each series. On the graph for 2012, there was little overlap between the two—spending didn’t correspond closely to potential ticket revenue. On the 2013 graph, the two lined up almost perfectly.
All of this optimization adds up. In the past, King believes, the Indians have earned $1.05 for every $1.00 they’ve spent on promotions, and broken even on their media dollars. But now that they’ve fine-tuned their promotion schedule and media spending, they’re projecting massive gains for 2013: a return of $1.85–$2.50 for every dollar spent on promotions, and $1.15–$1.45 for every dollar spent on advertising. Imagine what that would mean for the Indians’ bottom line.
The Indians’ commitments to player payroll have raised eyebrows this offseason: first they inked Nick Swisher (on the same day that ThinkVine put out its press release), then Brett Myers, then Michael Bourn. The general consensus among internet analysts seemed to be that Cleveland had locked up both Swisher and Bourn at reasonable rates, but that the timing of the expenditures on over-30 outfielders was perplexing, given that the team appears to lack the pitching to compete.
So why were the Indians willing to spend now? Well, maybe because they had (or expected to have) the money. I don’t know whether the marketing department’s projected profits played into those moves, but baseball teams do have budgets that dictate signings, and King’s sabermetric approach to marketing might have made Cleveland’s budget a bit bigger. As Bill James noted on Friday when describing the Baseball Ops/everyone else divide, “We don’t have any control over them at all. They, on the other hand, have some control over us. And it absolutely has to be that way, because teams have to make money.” It’s not far-fetched to think that using sabermetric methods to eke out extra attendance dollars might have permitted the Indians greater payroll flexibility, just as using sabermetric methods to identify an undervalued player would allow for more spending on the rest of the roster.
Ragusa told me that ThinkVine hasn’t worked with any other sports franchises, and that their work with the Indians was a way to “determine the potential value creation for professional sports teams.” Having completed that work, they’re now “actively speaking with other MLB teams, NHL and NBA.” I don’t doubt it. Right after I finished talking to King, a marketing executive from an NL team approached him and asked if they could set up a time to talk. He mentioned that his team doesn’t do anything like the Indians did with ThinkVine, but that they’d like to. If I’d been a marketing executive in the audience, I would’ve felt the same way.
Now, a couple of caveats. First, while those projected profit increases are compelling in PowerPoint, they haven’t yet come to pass. If we told you that we’d made PECOTA more accurate than ever, and that it would predict team and player performance this season much more precisely than it has in the past, you might want to wait and see whether it actually happened before you bought in. Similarly, we might want to wait to assess the Indians’ marketing results for 2013 before we start congratulating them on their on-paper profits.
Second, it seems to me that there’s some risk of overfitting. The ThinkVine model might accurately describe the data that was used to train it, but we don’t know for sure whether it will work as well out of sample (in other words, this season, though out-of-sample data from 2012 was used for testing and validation). I asked King after his presentation whether this was a concern, and he stressed that the heterogeneity of the data decreased the risk. He also mentioned the Indians’ wide range of team performance over the five-season sample, from 96 wins in 2007 to 65 in 2009, which fortuitously helped the team determine how the effectiveness of each promotion held up under differing conditions. And because ThinkVine was modeling only the Cleveland market, not national ticketing trends (because the Indians believed that other teams’ results wouldn’t be as applicable), it took less data to produce forecasts for the Tribe than it would for a larger company that operates in many local markets. However, King did acknowledge that additional data might improve the model’s predictive powers.
I directed the same question to Ragusa, who offered a longer response:
We aggressively validate this model in both hold-out periods and within the marketplace (frequent updates) to assure the model is highly generalizable (not just effective in the years we trained it). These are all (in math terms) largely under-determined problems, which classically leads to overfitting. Because we build and calibrate the model across many dimensions and in consideration of people’s behavior heterogenously, we reduce the under-determined nature, thus dramatically reducing the risk of overfitting.
In English, the answer goes like this: the more realistic the system we build is to the actual marketplace the less likely we are to overfit. So if we can simulate realistically how different people carry out their lives differently; use different media, etc. then the subsequent simulation is generalizable.
As Ragusa added, “We’ll get good reads on how effective it is within a short period.” Ultimately, it won’t matter that much to us whether the model works as well as advertised (though it will matter to ThinkVine and the Indians). What matters is that the Indians made the attempt to do something different. Let me go back again to Bill James, as seemingly every speaker at SABR did. “The key is to find the questions,” James said. “Once you find a question that is interesting and compelling, it actually makes very little difference to the world as a whole whether you get the answer right or wrong. … Because if you don’t get it right, somebody else will. Every interesting question becomes the basis for sequential research done by a lot of different people, and the first take on the answer is always wrong, somewhat.” As James pointed out, it’s getting harder to come up with these questions, but Alex King and the Indians are asking some.
And best of all, for fans accustomed to teams not letting anything slip, they’re giving us some of the answers. I asked King how he thought his department’s process compared to that of other teams. He was hesitant to put down anyone else’s methods, but he did say that he thinks ThinkVine’s agent-based approach puts the Indians ahead of the curve. The desire to get ahead of the curve, of course, is what brought all of the aforementioned Baseball Ops employees in the crowd to the conference. It’s impossible to imagine any of them taking the stage to tell other teams about their latest innovation. If King is already ahead of the curve, why did he come to SABR to tell other teams about his ideas?
I asked King whether he worried at all about other teams adopting the same methods and catching up to Cleveland. As you might imagine, given the content of his talk, the answer was “no.” King doesn’t see other teams as competitors, from a marketing perspective. If the White Sox were to start using sabermetric marketing and selling more tickets in Chicago, it wouldn’t directly decrease demand for Indians tickets from fans in Cleveland. He’d worry more about another entertainment company in Cleveland optimizing its promotions than he would a baseball team doing the same in some other city.
There were some limits to what King was willing to share, which is understandable: many dollars died to bring the Indians this information. Plus, while teams might not be competing directly when it comes to tickets, more efficient marketing means increased revenue, and increased revenue can translate to improved performance. Steeper competition for Cleveland might mean fewer wins, and less on-field success means fewer fans in the seats. All of which is to say that we should consider ourselves lucky to know as much about the sabermetrics of marketing as we do. It’s a rare treat to hear a team come this close to telling the truth.