Submit chat questions for Jeffrey Paternostro (Fri Dec 4 at 12:00 pm EST)

Time to put on my other hat. Let's leave the wonderful world of sabermetrics behind for a few minutes and venture into the world of psychometrics. In one of my former lives, I conducted (neuro)psychological testing, mostly on children and adolescents, as part of my graduate school requirements. I'm proud to say that I administered the Rorschach only once. Because they made me.

Psychological testing isn't nearly as scary as it sounds. Most of the time, it was a question of finding out more information about a diagnosis (is there a learning disorder present?) or differentiating between two possible diagnoses (is this ADHD or is the problem with attention due to anxiety?) Generally, we stuck to tests that are very well-grounded in research and that measured specific behaviors or cognitive skills. In fact, the process of writing up an evaluation of a psychometric battery of tests has the same feel as writing up a sabermetric view of a baseball player. It's very numerical and grounded in previous research—and if you don't know how to use those numbers properly, you'll come to some very wrong conclusions.

In my professional work, I did a lot of evaluations around issues of learning and attention. Often, the patient was having trouble with learning in school, and it was my job to figure out why. By extension, it was also my job to figure out what could be done to make life easier for the parents, the teachers, and most importantly, the kid.

But like so many things in life, I found myself asking "How can I apply this to baseball?"

Majoring in Baseball
Baseball develops its young players a little differently than the other "major" sports, or at least football and basketball. There is no "minor league" system to speak of in the NFL, and a limited one in the NBA. Players emerge from each league's respective drafts and are generally expected to contribute immediately, even in some limited capacity. It will be a couple of years before most of even the highly regarded MLB draftees who had their names called a few weeks ago sniff the big time. In fact, in an odd sequence of events, a baseball player might be drafted at the age of 18, given a half million dollars, and—despite the fact that he's never lived apart from his parents—sent to a fourth-tier city a thousand miles from his home. If he's an international signee, it's entirely possible that he's 16 and being sent somewhere where no one else speaks his language. Now, he gets to learn how to play big league baseball. And become a man.

Young baseball players are an altricial species. It's rare that a player is ready to go directly from high school (or even college) to the majors without a little bit of instructional time, and often several years’ worth. That means that baseball scouts have a much harder job than scouts do in other sports. They have to look at an 18-year-old and, sometimes devoid of context, figure out what sort of talent he will have at age 27. That's like asking someone to predict what a room full of fourth graders will look like at their high school graduation. A gifted athlete can succeed on pure athletic skill at the high school level, mostly because he's going up against competition who might be there because they didn't make the bowling team. What will happen when he faces guys who know what they're doing?

What has always struck me as something in need of repair was the fact that the language of scouting is heavy on vocabulary for describing physical build and playing skills, but relatively light on what might be going on under the surface. The story of baseball is littered with guys who had immense physical talent and promise, were never hurt, and yet did not make much of an impact. Look through the recap of the first round of any MLB draft, starting about five years ago. You'll pick them out pretty quickly. What happened?

The word that's most often trotted out in this situation is "makeup." It's one of those vague words that probably has as many definitions as it has people who use it. There are connotations of players making poor decisions, lacking desire, and just plain being a jerk rolled into the term, mixed with a small dash of moral disapproval. All of the above are probably true from time to time.

What about the guys who honestly try hard, but still just don't make it? There's something of an assumption that players are all equally teachable and that guys who start out at 18 with the most skills will be the guys who have the most skills at 27. It's not a bad way to bet, but it never works out that way. Maybe there's something that we're missing? Some guys who have trouble learning? Some who have other issues going on? Maybe there's more to makeup than just a bad attitude. What if there were ways to at least screen for some of the issues that might pop up?

Recently, some teams have become more aware of these issues, and some have started screening potential prospects in their own ways with their own proprietary methods. In what follows, I'd like to take a slightly different approach. I propose to develop a battery of tests comprised entirely of measures that are publicly available and can be purchased (or are free!) that might be useful in explaining some of the slack in the system. Many of them are tests that I've used in my own work in a psychological testing clinic and all are well-known and well-researched in the psychological community. (They're admissible in court!) In the process, I'll highlight a couple of issues that we don't often think about that might explain why first-rounders end up as busts and fifth-rounders end up as All-Stars.

Attention, Reaction Time, and Impulse Control
I've written extensively about issues of attention and focus, and how baseball demands a great deal of both. Baseball, more than any other sport, requires constant vigilance often combined with limited action, and in amounts that strain even typically developed young men.

In the clinic, one of my favorite tests was the Conners' Continuous Performance Test (CPT). I primarily used it to screen for ADHD, but it can be used for much more than that. It lasts 15 minutes and consists of a series of letters that flash on the screen. The person being tested is supposed to push the space bar when a letter comes on the screen, except on one special letter. Over those 15 minutes, the rate at which the letters are presented varies, and the main outcome of interest is actually the person's response time. Does it get longer when the time between letters goes from one second to four? (Attention is wandering.) Does it get significantly longer as you get into the 12th minute of the test? (Sustained attention might be an issue.) Does the subject have trouble stopping himself from pushing the space bar in response to the forbidden letter? (Is there an impulse control problem? Will he swing at anything?) Of course, no one will be perfect at the test, but the rates are compared to the league… er, population norm. In addition to yielding information concerning attention, the Conners’ Test is also a decent indicator of simple reaction time.

If a player can't sustain attention, it will affect him not only on the field, but during instructional time. When the coach is showing him a different way to position his hands or some trick to use to get a better grip on his changeup, will the player be able to concentrate long enough to hear the coach's message? It's not that teams shouldn't draft these guys. It's that they might have to make a few extra accommodations to get the best results out of them.

Self-Correction and Cognitive Flexibility
One reason that a player might flame out is that he's not willing to make adjustments to his game. Playing against the bowling team's rejects in high school, he might have hit .570, and he might have a chip on his shoulder. Let's see what happens when he faces decent (much less really good) pitching. Will he drop the chip and accept coaching?

One off-the-shelf assessment that might give us some idea is the Wisconsin Card Sort Test. There's a version of it that includes actual cards, but this one is better done on a computer as well. The WCST starts off with purposely limited instructions. The person taking the test is given four cards with different shapes, colors, and numbers on them and then asked which of the cards best matches a fifth card. He gets no feedback on being right or wrong until he makes his choice. Once he gets it right, he generally tries the same strategy again (match the color? match the number?), and for a while it works. Part way through the test, mercilessly, the rules change and again, the subject is not told anything. Will he stubbornly persist with what worked before? Will he show a longer response time (thinking about it more?) when things change? Will he be willing to try a new strategy?

Like most psychological testing, this isn't an exact replica of what a player will be asked to do in his actual baseball work. He might pass the WCST with flying colors but still flame out because he stubbornly refuses to try something new in real life. But the WCST screens out those who lack the underlying cognitive flexibility to begin with.

Verbal, Visual, and Kinetic Learning Styles
Some people learn best when you show them how to perform a task. Others learn best when you explain it to them verbally, and still others learn best by doing. Some are fine hearing something in isolation. Some need context. When I assessed learning issues, one thing I always tried to get at was how the particular child I was working with learned. (This, by the way, is why one-size-fits-all developmental systems can be very counter-productive, and why a coach who can explain things in multiple ways is probably worth a couple of wins per year.)

There are several ways to get at this one. There are memory tests, such as the Wechsler Memory Scale (WMS), which present a person with verbal information (a list of words) or visual information (an abstract design) to remember and later reproduce. There are also tests of memory for various hand gestures (although not on the WMS). The full WMS takes a long time to administer, but it's possible to administer specific subtests to get just the information you need. In doing so, you can get a good idea of how a person best encodes and remembers newly presented information. That might prove handy when deciding how to present lessons on the training field.

Potential Mental Health Strengths and Challenges
Men are notoriously hard to get through the door of a psych clinic, and an untreated mental health problem can easily get in the way of a promising career, so maybe it's best to screen for any as-yet-identified mental health concerns. There are a number of good screeners both comprehensive (Adult Self-Report, BASC-2 System), and brief (Strengths and Difficulties Questionnaire). Ditto for alcohol abuse and other substance use problems (and there are several screeners from which to choose). Teams would want to make sure that appropriate services are available to the player both as a matter of player development and as a matter of being decent human beings.

Dealing with Life
On the flip side, teams might also ask questions about a player's strengths. When things get rough, how does he cope? Make no mistake, things will get rough. The player will deal with failure on the field, perhaps on a level that he's never even thought possible. And then there's the simple fact that the minor league experience is a giant transition in life, as Twins vice president of player personnel Mike Radcliff points out in Parker Hageman's BP guest piece today. We often forget that these are 20-year-old guys away from home for the first time, sometimes taking care of meal planning and schedule-keeping for the first time on their own, and perhaps entering into their first mature romantic relationships. In addition to becoming major league players, they are transitioning into the role of adults within society. That isn't easy, no matter what job you're working.

There are methods of coping that show better results (active problem solving) than others (ruminating about the problem), and prospects who seem to use the latter might end up unable to deal with the pressure of becoming a major leaguer or life more generally. There are several ready-made coping inventories (including this one) that are quick to use. In addition, teams might want to know whether a player has the skills to function as an adult in society. Does he know how to complete a rental/lease agreement? Can he balance his bank account? Does he understand how to have safer sex? If a player has trouble in these other areas, it might spill into his play on the field. Some teams have programs to help young players with these skills, but it might be nice to know in advance whether a player will need that help. The Ansell-Casey Life Skills Assessment might help here.

Does He Like Baseball?
Here's a question that doesn't often get asked by fans. Do the players on your favorite team enjoy playing baseball? There's a difference between being good at something and enjoying it. When I was preparing this article, I talked to BP's own Jason Parks, who had an interesting take on the subject. The player development process requires a great deal of commitment to be done right. There's a lot to learn, and many teams are very strict in their teaching methods, with some, according to Jason, approaching military strictness. Will the player be willing to put up with that in order to get a shot at a major league career? He'd better like baseball.

Prospects are generally chosen because they have a great deal of natural talent. But the key question is whether the prospect has the will to actualize his natural talent. (Maybe it would be best to shorten that to some sort of acronym?) No, there isn't a test for this one. It's something that you have to get some field reconnaissance on from someone who is skilled in having these types of conversations. Someone like… a scout.

The question "Is he a jerk?" might be good information to find out as well.

Tying it All Together
Maybe I'm a little biased, but I believe that a full understanding of the mental side of the game—more than just the "rah-rah, you've got to work hard!" motivational piece, but issues of neurology and psychometric assessment—represent a great unexplored area in the study of baseball. There are real differences between people in these areas, and it's fairly easy to understand how a problem in one of them might derail a promising career. My goal here is to show that these soft factors aren't big black boxes. They are actually well understood within a broader context. We just haven't had the chance to look too deeply into them in baseball.

The other message that I want to convey is that development (of any kind) is not a nice, steady linear progression. It's hard work, it happens in fits and spurts, and it isn't just about what a player looks like from the neck down. There are a lot of issues that can impact a player's development. Sure, teams bear some responsibility to screen for those issues (and some need to do more), but before you trash your favorite team's player development department or curse the name of that hot prospect who looks like he's a bust, remember that it's not as easy as it looks.

It's not that teams don't recognize this, and again, some are already addressing the issue. (If only they would release their databases!) There will be limits to how much outsiders can truly measure these effects, although I think we do ourselves a disservice if we don't at least try. But make no mistake, there is plenty to be discovered here (and I left even more issues that I wanted to discuss on the cutting room floor). If you're a baseball researcher who’s begun to believe that everything about baseball that can be discovered has already been found, may I suggest that you poke around this area?

Thank you for reading

This is a free article. If you enjoyed it, consider subscribing to Baseball Prospectus. Subscriptions support ongoing public baseball research and analysis in an increasingly proprietary environment.

Subscribe now
You need to be logged in to comment. Login or Subscribe
So, basically looking for a way to put a grade on #want.
Will to actualize natural talent. W.A.N.T.
Bravo! Great article!
On topic, I enjoyed this Posnanski article on Jeff King:

He basically had the opposite of #want.
That article was actually in the back of my head when I wrote this.
Did he really? He seems to have had a pretty successful career in any case. I mean, it's obviously difficult to tell whether he would have been significantly better if he loved playing every day or not, but maybe that tells us that, for some people, the enjoyment may be less important than some of the other issues that Russell raises.

Also, just wanted to say that this was another excellent article.
Great stuff as always, Russell, although I did cringe a bit, remembering my grad school days and trying to administer and score some of the tests you mention to patients with schizophrenia. A rewarding process, albeit a time consuming one...

I would be interested to hear you thoughts on the utilization of some of other reaction time/working memory driven tests - the Stroop and it's close relative the Sternberg Item Recognition Paradigm, things we used in our functional MRI studies quite extensively. I wonder if they would be useful tools in helping determine not only cognitive flexibility, but also item recognition; it would be interesting to see if hitters who are better at pitch recognition would score better at these tests than those who do not, and also have quicker reaction times.

For those interested in reading more about the Stroop and SIRP, here are some links to peruse:
I had a section on the D-KEFS ready to go, but chopped it for space. I'm fascinated by the idea of executive functioning assessment and what it could do for both scouting and development. If nothing else, Stroop will give you a good idea of parallel processing and response inhibition, both of which would be good to have in the middle of a game where you have multiple strategic concerns to balance.
Actually, wouldn't that be even more directly related to the manager's job? Any inkling of organizations utilizing any of these tests as part of the managerial hiring process?
What are your thoughts on ADHD and hyperfocus? I've read that one of the "issues" with ADHD is not that there is a lack of ability to focus - but a lack of ability to focus on things that the person with ADHD does not find interesting.

I was diagnosed with ADHD last year (I'm 27) - and I remember that baseball was the only thing I could concentrate on. I would throw a ball against a wall for hours and hours and work on different ways to field a ground ball, my throwing motion, simulate an entire game... etc. But that was because I found baseball really interested and enjoyed it.

As a side note - I wanted to share this documentary with you as well on Andres Torres and overcoming ADHD at the Major League Level:

Attention and focus are going to be affected not just by frontal lobe activity. You are correct that one problem seen in ADHD is that people have a hard time turning on "the filter." There are a billion things going on around us at one time and we don't notice them, because we don't attend. And we can't... it would be too distracting to process all that.

But attention is also a function of whether something provides a level of stimulation that matches the cognitive resources that a person has available at the time. Something that is interesting is often high stimulation, and if baseball matched up with your frontal lobe's desire for stimulation, it probably was something that you enjoyed and that you could properly process. That's a wonderful thing.
I found the test that you referenced (Conners Continuous Performance Test) and took it just to see what my results were. ( I found a couple of studies to compare my results to; but nothing that was laid out plainly as normal ranges.

Statistic ISI: 1000 2000 4000 Pooled
Correct Trials 111/120 114/120 112/120 337/360
Correct Targets 106/106 110/110 105/106 321/322
Correct Foils 5/14 4/10 7/14 16/38
Target Acc Rate 1 1 0.991 0.997
Foil Acc Rate 0.357 0.4 0.5 0.421
Commission Errors 9 6 7 22
Ommission Errors 0 0 1 1
Correct RT Mean 343.61 363.97 389.21 365.5
Correct RT SD 82.65 65.76 79.1 78.28
Error RT Mean 296.89 326.17 320.29 312.32
Error RT SD 17.84 17.66 14.08 21.17
Sensitivity (d') 0 0 -2.348 -2.537
Bias (beta) 0.746 0.726 0.694 0.719

Anyways - it looks like my reaction time was pretty quick - but there was a ridiculous amount of commission errors. Apparently, a high amount commission errors is correlated with impulsivity. As the test was going on, I was trying to think of the X's as curveballs but would still just hit the space bar. (And I've had to do a lot of work to identify breaking pitches and am still not very good at it.) It would be REALLY interesting to see if there is a correlation between high impulsivity scores and walk rates, O-swing %, ability to the curveball, etc.
As an industrial psychologist, I often find that many employees/managers/companies, across a range of fields develop a certain amount of skill/competence that gets them to very good job performance.
However, in order to make the leap from "good" to "great" often requires faith that learning new methods, trying new things will be worth it in the long run, even at the expense of short-term performance. Many are unwilling to do this, holding them back.
(for example, some folks can type 25wpm using two-finger huntandpeck, but if they used proper keyboard technique they would suck at typing for a months, but then be able to type 60wpm. Many will not persevere through the month of suckitude, especially if their performance is being tracked and their compensation is based on it.)
I guess this is my long way of suggesting that a player's willingness to make adjustments, refashion swings and pitching motions, and really commit to them despite early setbacks should be really important to their long-term success.
However, it would take a very confident AA player to do this (or an organization that makes it ok for prospects to fail for a while), considering the scrutiny and competition he faces.
Other things to consider (or at least that I've considered)

1. Conscientiousness
2. Cognitive ability tests that work more effectively cross-culturally/linguistically (like the Raven's Progressive Matrices given that a large percentage speak English as a second language)
3. Measuring a players' implicit person theory (Carol Dweck's work). That is, do players view abilities as fixed or malleable and what that means for their work ethic/preparation.
4. Role of emotion (some research suggests that we're more able to draw information from short-term memory when experiencing positive affect.

I also think a lot of psychology research should look at managerial styles. Our research right now is extremely limited to their tactical decisions but I'd imagine a lot of what makes a good manager happens outside of gametime.
Raven's was one of the things that I left on the cutting room floor as well, and you are right that the issue of culture and language is something that would need to be considered (wrong place to have that discussion here, but certainly a good issue to think about). The issues around self-efficacy would make for some interesting research though.
The only thing I will say is I'd caution between conflating self-efficacy and implicit person theory - they're both theories of motivation but may make very different predictions (especially if you subscribe to Vancouver's work with control theory suggesting that higher self-efficacy leads to higher goals, but then less effort).

At any rate, great piece. I remember someone once telling me that at the professional level players are so similar physically that what really separates them is the mental side of things.

Also, your background is in clinical psych, yes?
Clinical in a program focused on child and adolescent. I ended up working mostly with adolescents, oddly enough. I had planned on working school age. Sometimes weird things happen in life.
Seems to be the trend in psychology- get the degree and then see where life takes you.
Great article. Thanks for this. One would imagine these kind of things would have been par for the course. Sadly not.
I wonder how many high school players had a high school coach kill the joy of the game for them and the players took that attitude to affiliated ball with them?
Bravo. I wouldn't be surprised to see a Major League team hire Russell next.

Q. Are these sorts of tests subject to more wildly varying results by the same subject on different occasions than, say, an aptitude test or almost any physical test? I think some days I am in a rut and can't think creatively or with strong "self correction", while other days I am more open minded and clear.
That's a great question.
Depends on the test, and specifically the reliability of the test.

Creativity is kind of a different ballgame but things like cognitive ability and personality tend to be a bit easier to measure (though still difficult).
There probably is some state-dependency that goes with it. For example, I once took the Conners attention test myself at 4:00 pm after an exhausting day. It pretty much diagnosed me as having ADHD. In general, we would do this sort of test first because if a kid was well-rested and not tired from other tests, but still showed limited attention, then that was an indication that there was a problem.
Good stuff Pizza! I wonder how much teams use this sort of stuff. Seems like it would be very worthwhile.
Thank you for this look into a side of professional baseball I rarely thought about in the past. It provided an excellent, informed look (and a bunch of cool tests and places to look at as a teacher, too).