Five Minutes With...

Five Minutes with Keith E. Stanovich, Richard F. West, and Maggie E. Toplak

Today, Keith E. Stanovich, Richard F. West, and Maggie E. Toplak are featured on our Five Minutes with the Authors talking about their book The Rationality Quotient. The books explores how to assess critical aspects of cognitive functioning that are not measured by IQ tests. 

Why is rationality commonly seen as a part of intelligence, yet excluded from tests that measure it—IQ tests being the primary example?

The idea that IQ tests do not measure all of the important human faculties is not new. This point deserves elaboration, because a common misinterpretation of our work is that we are trying to improve intelligence tests. Not only is this not our goal, but it is a serious misunderstanding of what we are trying to achieve.

Unlike some writers, we do not see the usefulness of labeling every human cognitive skill as intelligence—particularly when there are readily existing concepts (both scientific concepts and folk concepts) for some of those things (rationality, creativity, wisdom, critical thinking, open-minded thinking, reflectivity, sensitivity to evidence).  For example, theorists such as Howard Gardner and Robert Sternberg define entities such as practical intelligence, bodily-kinesthetic intelligence, emotional intelligence, etc…

We would argue that these theorists are adopting a permissive conceptualization of what intelligence is rather than a grounded conceptualization.  Permissive theories include in their definitions of intelligence aspects of functioning that are captured by the vernacular term intelligence (adaptation to the environment, showing wisdom, creativity, etc.) whether or not these aspects are actually measured by existing tests of intelligence.  Grounded theories, in contrast, confine the concept of intelligence to the set of mental abilities actually tested on IQ tests.  Adopting permissive definitions of the concept of intelligence serves to obscure what is absent from extant IQ tests.  Instead, in order to highlight the missing elements in IQ tests, we adopted (like most scientists) a thoroughly grounded notion of the intelligence concept in our book.

Our goal always has been to give the concept of rationality a fair hearing— almost as if it had been proposed prior to intelligence.  Had that happened, what would a comprehensive test of rationality have looked like?  We are running, if you will, an experiment on theoretically grounding and operationalizing an aspect of mental life that is important in its own right.


What is the danger in inflating the concept of intelligence?

What you are referring to with this question is the fact that permissive theorists put into the term intelligence more than what the IQ tests measure.  One very strong tendency among permissive theorists is to use adjectives to differentiate the more encompassing parts of their intelligence concept from the “IQ test part”.  Major theorists such as Sternberg and Gardner talk about practical intelligence, creative intelligence, interpersonal intelligence, bodily-kinesthetic intelligence, etc.  In such usages, the word intelligence becomes a marker for “optimal or expert behavior in the domain of”.  So, for instance, when Sternberg discusses high practical intelligence it can be translated to mean “optimal behavior in the domain of practical affairs” or when Gardner talks about high bodily-kinesthetic intelligence he means little more than high functioning in the bodily-kinesthetic domain.  The word intelligence is actually superfluous.  It is there merely to add status to the domain in question.  The strategy seems to be something like the following: Because intelligence is a valued term and we want bodily-kinesthetic talent to be valued too, we’ll fuse the term intelligence onto it in order to transfer some of the value from intelligence to bodily-kinesthetic talent.  Indeed, this is why educators have been so enthusiastic about the “multiple intelligences” idea.  Its scientific status is irrelevant to them.  They use it as a motivational tool—to show that “everyone is intelligent in some way.” The same is true for the coinages of social intelligence or emotional intelligence. 

However, there are unintended consequences—some of them quite ironic—of this strategy, consequences that have been insufficiently appreciated.  Labeling different mental entities with the same name will encourage just the assumption that many permissive theorists want to attack—it will inflate the esteem given to IQ tests in the public’s mind.  In a sense, permissive theorists seek to break a rule of construct validity—and of common sense: things that are named the same should go together.  If these things really are separate mental faculties, and we wish to emphasize their separateness, then we should not be calling them all “intelligences.”  However, by their profligate use of the term intelligence, the permissive theorists subvert their very purpose of isolating “the IQ test part of intelligence” as only one aspect of many cognitive virtues that we may wish to value (spatial ability, creative ability, fluency in practical affairs; and, of course, rationality).

By inflating the word intelligence, by associating it with more and more valued mental activities and behaviors, permissive theorists will succeed in doing just the opposite of what many of them intend—cutting “the IQ test part of intelligence” down to size.  If you inflate the conceptual term intelligence you will inflate all its close associates as well—and 100 years of mental testing makes it a simple historical fact that the closest associate of the term intelligence is “the IQ test part of intelligence.” 

One of the goals of the permissive theorists is to emphasize that there are aspects of cognitive life that are important outside of those measured on IQ tests.  However, I do not see why everything in human nature, cognitively speaking, has to have the label intelligence—particularly when there are readily existing labels for some of those things. For example, we have scientific valid labels such as rationality, creativity, wisdom, critical thinking, open-minded thinking, reflectivity and sensitivity to evidence.

This tendency to label all of these competencies as intelligence has several bad effects.  For example, rational thinking skills vanish under permissive definitions of intelligence.  Rationality assessments become part of intelligence if the latter is conceptualized broadly.  And again, intelligence test producers gain from these permissive definitions because people will continue to associate the broad concept of intelligence with these tests.  How could they not?  The tests carry the label “intelligence” and the producers of the tests are not eager to discourage the association with permissive theories.  For example, it took real chutzpah for David Wechsler to define intelligence in his book as “the aggregate or global capacity of the individual to act purposefully, to think rationally and to deal effectively with his environment” (p. 7, 1958) despite authoring an IQ test with his name on it that measured no such thing!

Our strategy in The Rational Quotient has been different.  We adopt a grounded view of intelligence, and then go on to point out that there are legitimate scientific terms as well as folk terms for the other valued parts of cognitive life that are not assessed on IQ tests.  With the CART, we have proven that the domain of rational thinking is measurable.  The strategy we have used in The Rationality Quotient was to open up some space for rationality in the lexicon of the mental and, in doing so, tame the intelligence concept.  We have coherent and well operationalized concepts of rational action and belief formation.  We have a coherent and well operationalized concept of intelligence.  No scientific purpose is served by fusing these concepts, because they are very different.  To the contrary, scientific progress is made by differentiating concepts.  Although IQ tests do not actually encompass all of the important mental faculties, we often act (and talk) as if we have forgotten this fact.  Where else does our surprise at smart people doing foolish things come from if not from the implicit assumption that rationality and intelligence should go together?  The CART, in operationalizing the concept of rationality, should help to attenuate our surprise at this phenomenon and to create conceptual space in which we can value other cognitive abilities not assessed on IQ tests.

Subsections of the CART test involve questions on subjects such as conspiracy theories, superstitious thinking, and antiscience attitudes. Such beliefs often evolve and change over the course of someone’s life. Does this mean that someone’s rationality score can change over time?

Most definitely the score can change over time.  This is because of the inclusion of subtests just like those you have mentioned.  The framework for the CART is unique  in that it provides a mix of process considerations and knowledge considerations.  Critical thinking assessment, for example, is often criticized for its overemphasis on process issues and for its failure to acknowledge that no thinking can be rational or critical without requisite knowledge.  In contrast, the CART assesses critical knowledge bases (numeracy, financial literacy, risk knowledge) while also measuring the tendency to acquire problematic knowledge (superstitions, anti-science attitudes, conspiracy theories).  Of course, the CART also assesses processing issues related to rationality under the general rubric of miserly information processing.  But the knowledge components of the CART might be particularly prone to change over time, as your question suggests, because people are continuously acquiring and updating information.  This is equally true of facilitating knowledge bases (e.g., financial literacy, risk knowledge) and inhibiting knowledge bases (e.g., superstitions, conspiracy theories)—the latter what we call contaminated mindware.

Is rationality something that is innate? Can people train to improve their rationality skills should they perform poorly on the CART test?

There is no way to train for rational thinking across the board.  What I mean is that there is no training program that will generalize to all aspects of rational thinking.  This is because there is no g-factor of rational thinking.  Rationality is a multifarious concept.  This is reflected in the fact that the CART has 20 subtests and four thinking dispositions scales (the latter are not part of the total score but just for informational purposes). Collectively they tap both instrumental rationality and epistemic rationality.  In cognitive science, instrumental rationality means behaving in the world so that you get exactly what you most want, given the resources (physical and mental) available to you.  Epistemic rationality concerns how well beliefs map onto the actual structure of the world.

The CART assesses epistemic thinking errors such as: the tendency to show incoherent probability assessments; the tendency toward overconfidence in knowledge judgments; the tendency to ignore base rates; the tendency not to seek falsification of hypotheses; the tendency to try to explain chance events; the tendency to evaluate evidence with a myside bias; and the tendency to ignore the alternative hypothesis.

The CART assesses instrumental thinking errors such as: the inability to display disjunctive reasoning in decision making; the tendency to show inconsistent preferences because of framing effects; the tendency to substitute affect for difficult evaluations; the tendency to over-weight short-term rewards at the expense of long-term well-being; the tendency to have choices affected by vivid stimuli; and the tendency for decisions to be affected by irrelevant context. 

My purpose here in digressing to describe the CART is to point out that given the number and complexity of rational thinking skills, it is unlikely that a single training program would affect all of them at once.  Nevertheless, many of the specific components of rational thinking have been the target of training programs, and these programs have shown moderate success.  We do not review this literature in The Rationality Quotient, but in an earlier book, Rationality and the Reflective Mind, I pointed the reader to the literature on this topic.  In a table in the last chapter of that book, I addressed the issue of the potential malleability of the skills in our framework for rational thinking (as it existed at the time of that book).  In that table, for each experimental paradigm that operationalized rationality in our framework, we listed at least one study that demonstrated that the skill is possibly malleable via training, education, or experience.  The types of evidence listed, however, differed in terms of the strength of the inference that can be drawn.  Some were true training studies using random assignment to experimental groups.  Such studies of course provide the strongest evidence that training does result in measurable increases in a rational thinking skill.  We also included some studies showing that levels of a rational thinking skill are correlated with educational level and/or experience.  Such correlational evidence is of course suggestive, but it is not definitive evidence that the skill is teachable or trainable.  Nonetheless, it is impressive that, given the many different areas of rational thought I listed above, there is at least some type of evidence suggesting the trainability of each one of them.

I should also mention that in Chapter 15 of The Rationality Quotient we include a similar table showing that rational thinking tendencies are linked to real life decision making.  In that table, for each of the paradigms and subtests of the CART, an association with a real-life outcome is indicated.  The associations are of two types.  Some studies represent investigations where a laboratory measure of a bias was used as a predictor of a real-world outcome.  Others are reports of real-world analogues of biases that were originally discovered in the lab.  Clearly more work remains to be done on tracing the exact nature of the connections—that is, whether they are causal.  The sheer number of real-world connections, however, serves to highlight the importance of the rational thinking skills in our framework. 

Detractors might argue that what counts as rational thought is subjective, dependent on which camp one is arguing from. What would you say to them?

There are two reasons that this is not an issue for the CART.  The first is that the tasks used to operationalize instrumental rationality in the CART are relativistic in a certain sense.  These tasks are designed to accept the goals of the subject as given.  Recall from above that instrumental rationality means behaving in the world so that you get exactly what you most want, given the resources (physical and mental) available to you.  Somewhat more technically, economists and cognitive scientists have refined the notion of optimization of goal fulfillment into the technical notion of expected utility.  Utility theory as a normative model takes a person’s goals as they are and simply asks if the preferences the person displays satisfy certain consistency relationships (the so-called axioms of choice). 

Take a very basic principle of rational choice, transitivity (not tested in the CART, but many similar axioms are).  Transitivity says that if you prefer A to B and B to C, then you should prefer A to C.  If, instead, you prefer A to B and prefer B to C and prefer C to A, you have violated the transitivity axiom and (it has been shown) you cannot be maximizing utility.  You are not instrumentally rational.  The content of A, B, and C do not matter to the axiom.  It is relativistic in that sense.  It doesn’t care what the content of the options are, only that you stay consistent.  The principle tolerates differences in the valuation of options because it only assesses consistency.  Many of the rational principles assessed in the CART are like this.  They measure consistency and responsiveness to context, but they do not make an overall judgment about the valuation of options.

There is also a second reason why the CART is not vulnerable to criticisms that responses on it should be scored another way.  It has been found that subjects who have missed an item will most frequently, when the error is pointed out, agree that they have violated a principle of rational thinking that, in general, they would want to follow.  In short, when presented with a rational choice axiom that they have just violated in a choice situation, most subjects will actually endorse the axiom.  That subjects endorse the strictures of rationality when presented with them explicitly suggests that they acknowledge the normative force of the axioms of rational choice.  We have only used items in the CART for which the rational thinking norms are uncontroversial.