July 9, 2013
An Attempt to Measure Makeup
Time to put on my other hat. Let's leave the wonderful world of sabermetrics behind for a few minutes and venture into the world of psychometrics. In one of my former lives, I conducted (neuro)psychological testing, mostly on children and adolescents, as part of my graduate school requirements. I'm proud to say that I administered the Rorschach only once. Because they made me.
Psychological testing isn't nearly as scary as it sounds. Most of the time, it was a question of finding out more information about a diagnosis (is there a learning disorder present?) or differentiating between two possible diagnoses (is this ADHD or is the problem with attention due to anxiety?) Generally, we stuck to tests that are very well-grounded in research and that measured specific behaviors or cognitive skills. In fact, the process of writing up an evaluation of a psychometric battery of tests has the same feel as writing up a sabermetric view of a baseball player. It's very numerical and grounded in previous research—and if you don't know how to use those numbers properly, you'll come to some very wrong conclusions.
In my professional work, I did a lot of evaluations around issues of learning and attention. Often, the patient was having trouble with learning in school, and it was my job to figure out why. By extension, it was also my job to figure out what could be done to make life easier for the parents, the teachers, and most importantly, the kid.
But like so many things in life, I found myself asking "How can I apply this to baseball?"
Majoring in Baseball
Young baseball players are an altricial species. It's rare that a player is ready to go directly from high school (or even college) to the majors without a little bit of instructional time, and often several years’ worth. That means that baseball scouts have a much harder job than scouts do in other sports. They have to look at an 18-year-old and, sometimes devoid of context, figure out what sort of talent he will have at age 27. That's like asking someone to predict what a room full of fourth graders will look like at their high school graduation. A gifted athlete can succeed on pure athletic skill at the high school level, mostly because he's going up against competition who might be there because they didn't make the bowling team. What will happen when he faces guys who know what they're doing?
What has always struck me as something in need of repair was the fact that the language of scouting is heavy on vocabulary for describing physical build and playing skills, but relatively light on what might be going on under the surface. The story of baseball is littered with guys who had immense physical talent and promise, were never hurt, and yet did not make much of an impact. Look through the recap of the first round of any MLB draft, starting about five years ago. You'll pick them out pretty quickly. What happened?
The word that's most often trotted out in this situation is "makeup." It's one of those vague words that probably has as many definitions as it has people who use it. There are connotations of players making poor decisions, lacking desire, and just plain being a jerk rolled into the term, mixed with a small dash of moral disapproval. All of the above are probably true from time to time.
What about the guys who honestly try hard, but still just don't make it? There's something of an assumption that players are all equally teachable and that guys who start out at 18 with the most skills will be the guys who have the most skills at 27. It's not a bad way to bet, but it never works out that way. Maybe there's something that we're missing? Some guys who have trouble learning? Some who have other issues going on? Maybe there's more to makeup than just a bad attitude. What if there were ways to at least screen for some of the issues that might pop up?
Recently, some teams have become more aware of these issues, and some have started screening potential prospects in their own ways with their own proprietary methods. In what follows, I'd like to take a slightly different approach. I propose to develop a battery of tests comprised entirely of measures that are publicly available and can be purchased (or are free!) that might be useful in explaining some of the slack in the system. Many of them are tests that I've used in my own work in a psychological testing clinic and all are well-known and well-researched in the psychological community. (They're admissible in court!) In the process, I'll highlight a couple of issues that we don't often think about that might explain why first-rounders end up as busts and fifth-rounders end up as All-Stars.
Attention, Reaction Time, and Impulse Control
In the clinic, one of my favorite tests was the Conners' Continuous Performance Test (
If a player can't sustain attention, it will affect him not only on the field, but during instructional time. When the coach is showing him a different way to position his hands or some trick to use to get a better grip on his changeup, will the player be able to concentrate long enough to hear the coach's message? It's not that teams shouldn't draft these guys. It's that they might have to make a few extra accommodations to get the best results out of them.
Self-Correction and Cognitive Flexibility
One off-the-shelf assessment that might give us some idea is the Wisconsin Card Sort Test. There's a version of it that includes actual cards, but this one is better done on a computer as well. The
Like most psychological testing, this isn't an exact replica of what a player will be asked to do in his actual baseball work. He might pass the
Verbal, Visual, and Kinetic Learning Styles
There are several ways to get at this one. There are memory tests, such as the Wechsler Memory Scale (
Potential Mental Health Strengths and Challenges
Dealing with Life
There are methods of coping that show better results (active problem solving) than others (ruminating about the problem), and prospects who seem to use the latter might end up unable to deal with the pressure of becoming a major leaguer or life more generally. There are several ready-made coping inventories (including this one) that are quick to use. In addition, teams might want to know whether a player has the skills to function as an adult in society. Does he know how to complete a rental/lease agreement? Can he balance his bank account? Does he understand how to have safer sex? If a player has trouble in these other areas, it might spill into his play on the field. Some teams have programs to help young players with these skills, but it might be nice to know in advance whether a player will need that help. The Ansell-Casey Life Skills Assessment might help here.
Does He Like Baseball?
Prospects are generally chosen because they have a great deal of natural talent. But the key question is whether the prospect has the will to actualize his natural talent. (Maybe it would be best to shorten that to some sort of acronym?) No, there isn't a test for this one. It's something that you have to get some field reconnaissance on from someone who is skilled in having these types of conversations. Someone like… a scout.
The question "Is he a jerk?" might be good information to find out as well.
Tying it All Together
The other message that I want to convey is that development (of any kind) is not a nice, steady linear progression. It's hard work, it happens in fits and spurts, and it isn't just about what a player looks like from the neck down. There are a lot of issues that can impact a player's development. Sure, teams bear some responsibility to screen for those issues (and some need to do more), but before you trash your favorite team's player development department or curse the name of that hot prospect who looks like he's a bust, remember that it's not as easy as it looks.
It's not that teams don't recognize this, and again, some are already addressing the issue. (If only they would release their databases!) There will be limits to how much outsiders can truly measure these effects, although I think we do ourselves a disservice if we don't at least try. But make no mistake, there is plenty to be discovered here (and I left even more issues that I wanted to discuss on the cutting room floor). If you're a baseball researcher who’s begun to believe that everything about baseball that can be discovered has already been found, may I suggest that you poke around this area?