ResolutionsRTK

"It depends."
Testing the Relevance of Psychometrics

Tom Pietkiewicz
Principal Consultant

I have spent much of my career working in assessment and development, and have worked on and managed a number of large projects where a range of behavioural and psychometric testing has been utilised. As such, I am often asked to recommend or set up measures for recruitment or talent management.

In most cases I am asked by a client what particular tests I would recommend. The answer is not that simple. It depends. I hope this article provides a few insights.

Many assessments are off the shelf and providers set up a battery of verbal, numeric and abstract tests combined with a personality measure. The person who scores higher on these and returns personality scores that sounds ‘nicer’ gets the job or is considered ‘talent’.

There are a few issues with such an approach. First of all, you could be measuring things that are completely irrelevant for the job. Are you testing numeracy or literacy, or reasoning and thinking skills? Which ones are relevant to the role? How did you determine this? What do we want to know about the candidate?

In many cases I have found that the factors that predict success in a role are often counterintuitive. So rather than advocate particular psychometric brands or tests, in my experience organisations gain more value from psychometric assessments that have been:

Correlated with indicators of high-performance for that particular role;
Measured over time; and
Periodically reviewed, particularly the assumptions underlying the assessment and recruitment decisions.

In one organisation I oversaw over 500 psychometric assessments comprised of various measures, and learned a number of interesting things by constantly comparing candidate scores with their resulting performance scores in the role. Here are a few examples.

Critical Thinking – not universal
The tests measuring higher skills like verbal ad numerical reasoning and critical thinking were linked to success in roles starting from middle management and above. In these middle management roles we found that ‘Critical Thinking’, or the ability to think objectively, probe information, make judgements, had a 0.4 correlation.

Just to be clear, correlation is the relationship between one score and another. A correlation of 1 is a perfect correlation where an increase of one score returns a comparable increase in the other.

Our correlation on the Critical Thinking measure suggested a fairly strong relationship between this skill and a candidate’s likelihood of returning better performance scores in middle management roles or above.

More interestingly, the Critical Thinking measure had no predictive validity at the lower tiers in the company and junior management roles. Yes, some people scored higher than others, but this did not impact on their resulting scores. We found that other abilities and personality factors were key.

In other words we would not rely on the Critical Thinking measure at those levels to make a recruitment decision. However we might use this measure if we wanted to find some individuals who might be successful when promoted to those higher roles.

Numerical Reasoning – limited predictive validity
We used numerical measures fairly widely, but again found that their predictive validity was limited. Through analysis we found that scores on the advanced Numerical Reasoning tests were very predictive for success at executive roles, with a correlation of 0.5 between scores on this measure and resulting employee performance. This is a strong relationship and it does make sense. Decisions and thinking at that level need to be grounded in solid economics, considering budgets, statistics and data at all times. People with lower scores had a much lower chance of success in those roles.

Abstract Reasoning – hit and miss
Abstract Reasoning is another popular measure which we found to be very hit and miss in predicting success. After analysing scores across the organisation it was discovered that it had some relevance in IT, Engineering and some technical functions, but we mostly stopped using this measures for mainstream roles.

Personality – counterintuitive
Again we compared personality factors and scores against performance scores and were able to build a success model for various roles and tiers within the company. Many findings were not what one would expect.

For example, high performing frontline Customer Service staff were consistently lower in resilience. How could that be? Interestingly, it was the stress they felt from an upset customer that motivated them to act and resolve a complex issue, follow up or do additional work to help. Highly resilient staff were less likely to change approach and continued with the usual script, “Yes, customers get upset all the time… That’s the nature of our job... So what?” Performance scores reflected this. The danger is that at selection some candidates with lower Resilience scores might have been rejected.

So, collect data and assess your assessments
There are many other counterintuitive factors and examples that were discovered, too many to list here. In many cases scores that we might have seen as unfavourable at face value were actually what was needed for success.

A large data bank of assessments and scores can take a long time to build. An approach I like using is to profile employees that are performing well. Recently I looked at some complex data entry, information management roles. We picked out a dozen staff who were high performers and tested them with a variety of measures. From the results we looked at common factors, similarities and differences. This allowed us to build a selection model that we could then use to assess new employees. This model used only the tools that were relevant and was therefore targeted, simple and inexpensive. It helped us build an understating of the role and also assisted with management of staff in the role as we now knew more about them.

The key point these case studies illustrate is the need for a tailored approach that is flexible and builds knowledge of roles and contexts that are being measured. Only then can you make informed decisions based on psychometric assessments.

It is less important what brand of measures you use, or how much the assessments cost. It is the knowledge that is built that is invaluable. Ultimately, a good assessment program should aim to not only select better staff, but also assist in managing talent, engagement, retention, culture and ensuring better business outcomes. These are the potential benefits of an approach to psychometric assessment that has been prefaced by, “It depends.”