Human performance, psychometry, and baseball statistics I

This is the first in a series of three posts on the topic of human performance, psychometry, and baseball statistics. In the process of writing, it outgrew single-post size.

Human ambition for achievement in modest measure gives meaning to our lives, unless one is an existentialist pessimist like Schopenhauer who taught that life with all its suffering and cruelty simply should not be. Psychologists study our achievements under a number of different descriptions--testing for IQ, motivation, creativity, others. As part of my current career transition, I have been examining my own goals closely, and have recently read a fair amount on these topics which are variable in their evidence.

A useful collection of numerical data on the subject of human performance is the collection of Major League Baseball player performance statistics--the batting averages, number home runs, runs batted in, slugging percentage--of the many thousands of participants in the hundred years since detailed statistical records have been kept and studied by the players, journalists, and fans of the sport. The advantage of examining issues like these from the angle of Major League Baseball player performance statistics is the enormous sample size of accurately measured and archived data.

The current senior authority in this field is Bill James, who now works for the Boston Red Sox; for the first twenty-five years of his activity as a baseball statistician James was not employed by any of the teams. It took him a long time to find a hearing for his views on the inside of the industry, although the fans started buying his books as soon as he began writing them.

In one of the early editions of his Baseball Abstract, James discussed the biggest fallacies that managers and executives held regarding the achievements of baseball players. He was adamant about the most obvious misunderstood fact of player performance: it is sharply peaked at age 27 and decreases rapidly, so rapidly that only the very best players were still useful at the age of 35. He was able to observe only one executive that seemed to intuit this--a man whose sole management strategy was to trade everybody over the age of 30 for the best available player under the age of 30 he could acquire.

There is a fair amount of more formal academic research on this issue. It is described in the literature as the field of Age and Achievement. The dean of the psychologists studying Age and Achievement is Dean Simonton. A decent overview of their findings is here. This is a meta-study of hundreds of individual studies. Many fields and many metrics are sampled. There is one repeated finding. Performance starts low at a young age and steadily increases along a curve which bears a resemblance to a Gaussian bell-shaped curve, peaks, and then declines. The decline is not as rapid as the rise (it is not a symmetric bell shape; it is steeply inclining from the left to the peak and gently declining form the peak to the right), but it is inevitably seen everywhere. The age of peak achievement varies, depending on the field. Baseball players peak at 27 (the curves from the psychology publications look exactly like the curve published by Bill James in his Abstract), business executives peak at 60, and physicists peak at age 35. Shakespearian actors peak late and rock stars peak early. These are statistical results and individual outliers abound. You, the individual physicist, may not be over the hill at 40, but this is the way to bet.

My hometown major league baseball franchise, the Houston Astros, recently had this empirical law verified for themselves in real time, and the hard way. They invested the bulk of their payroll budget on three players: Miguel Tejada, Carlos Lee, and Lance Berkman. All three were over the age of 30, i.e., definitely into their decline phase. When their performance declined more rapidly than expected, the team lost many more games than they were planning for. They had a contending team's payroll and big plans, but now Tejada and Berkman are gone and they are rebuilding. In an attempt to cut losses, they traded their (prime-age) star pitcher for young players.

A recent post on Hacker News, Silicon Valley's Dark Secret: It's all about Age, generated 120 comments of heated discussion about institutional age discrimination and the unappreciated value of experience. The consensus view expressed there is young programmers have to advance into management or become unemployable near age 50.

It could perhaps be viewed as a biological ecosystem. We are in an ecosystem. The ecosystem selects for fitness. What is sometimes misunderstood is the ecosystem does not select for absolute fitness, but for fitness specific to a niche. If the available niches in this "ecosystem" are for 40 years-old brains, and there aren't any niches for 50 years-old brains, then some fully fit brains (in an absolute sense) are going to be out of employment opportunities. Faced with a system like this, the job seeker may have to be clever at finding ever narrower niches to squeeze into.

One of the moderators at Hacker News, Paul Graham, is a software venture capitalist. He is accused in the thread of unconcealed age discrimination--that he will not invest in entrepreneurs over 38, and claiming that nobody over 25 will ever learn Lisp. If you are a forty-year-old physicist and you want to learn Lisp and get venture capital funding for your business plan--well, good luck with that!