What happens when a whole body of research has relied on a flawed measurement tool? Williamson Smith et al.’s study focuses on a popular research measure, The Passion Scale, with this and other questions in mind.
The concept of “passion” has been used for a while by CDPs and others. “Passion” tried to capture the idea that people could love their work and be inherently motivated to do it when it aligned with an important part of their identity. This notion was supported by research, but soon two kinds of passion were found: one built on intrinsic motivation (harmonious passion) and one derived from external pressures and outcomes (obsessive passion). The behaviours each creates are similar, but the outcomes are very different. Individuals with harmonious passion are positive about and engaged with their work whereas those with obsessive passion are likely to be negative and exhausted. These are important findings to CDPs: Helping people find work they love will lead to work satisfaction and engagement than more so than helping people focus mostly on external rewards.
Here’s the rub: The research findings were sometimes mixed. Williamson Smith et al.’s search to explain the inconsistencies led them to zero in on the most popular measure of work passion, The Passion Scale, a well-tested instrument that had checked many of the important boxes used to assess psychometric tests (e.g., passing tests of predictive, and discriminant validities). Williamson Smith et al. noted, however, that it had not undergone tests for content validity [1] (i.e., does it have the items in it needed to measure the thing it’s trying to measure?).
The approach to assessing content validity started with a review of definitions used in the research regarding work passion as well as related concepts (e.g., “vigor” is a core part of work passion but “work engagement” (which includes vigor) is not – it is what the researchers refer to as an “orbital” construct).
Williamson Smith et al. then had 528 individuals rate the items in three versions of The Passion Scale using a process too complicated to explain in this summary. They found very few content validity strengths overall and no evidence of content validity in the test’s sub-scales (general passion, harmonious passion, obsessive passion).
In short, The Passion Scale has not been measuring what researchers thought it was measuring (but each version got better), which means all research using the scale may have to be re-interpreted.
The takeaway is that even sophisticated research can be flawed. The upside is that the scientific method will uncover the flaws and correct them. One of H.B. Gelatt’s decision-making paradoxes applies here: Be aware and wary! CDPs need to keep up with the evidence-base informing their work and always be ready to adapt when new evidence emerges.
[1] One would think all psychometric tests would have been checked for this but, at least in organizational behaviour research, the majority have not.
Responses