Testing By Any Other Name


I’ve written about testing of employees before (see here and here), and it seems like a topic that just won’t go away. The other day, I made a comment on Matt Mullenweg’s blog about hiring and age discrimination and such. This prompted an email from someone on the cutting edge of employee testing.

Seeing that I used to be the VP of HR at Microsoft, Bryan Kennedy of Pairwise sent me an email asking for comment on his company’s new testing tool (you can see it here). They have developed a personality test that relies on images rather than words, a high tech Rorschach test. They are trying to sell it to companies like Microsoft and Google for their recruiting efforts. As Bryan sees it working:

An employer, such as Microsoft, asks their current employees to take the 5-10 min test. The test is posted on their web site for candidates to play.

After an interested candidate takes the test, they would see comparative results with current MS employee generalizations. Ex:
“You’re more outgoing than the average engineer.” or “You have a similar work-play ethic.”

It might even suggest for which teams you’d be most interested in applying. At that point, you’d be encouraged to submit your resume. The hiring director would see a report about the findings from the test and could use them at their discretion.

  • It’s far less (if not nearly impossible) to game than other written tests
  • It empowers candidates to do their own evaluation – rather than being a hidden metric of assessment, it’s a tool for figuring out if you’d like it there
  • It’s a way for companies to attract hires in a new and interesting way

When I sent him off to read my opinion on testing (see here and here), he replied with: “I side with the argument that as long as they’re used properly and taken with a grain of salt, their value is probably (!) not null.”

I respectfully disagree. The more I look into testing, the closer the value asymptotically approaches zero. I responded to Bryan with a million questions. Here are some of my concerns:

  • How does one get the 50,000 employees of Microsoft to take the test? Why would an employee want to take the test? Some mandate from above? Wouldn’t it just reinforce the impression that HR is just a bunch of wishy-washy voodoo? If there’s nothing in it for me to take it, except potential bad outcomes, why should I take it? What kind of complete rate would one expect (10% – 20%)? Doesn’t that make the results useless?
  • The best engineers/employees are the ones least likely to complete the test. For tests like Myers-Briggs most people who take it are people who are a) having some difficulty and have turned to the test to help them out, b) are trying to please management in some way, or C) just the “I love to examine my navel” kinds of people. That doesn’t seem like a good healthy cross-section.
  • “The hiring director would see a report about the findings from the test and could use them at their discretion” is extremely dangerous. Doesn’t it risk them making homogenous teams, intentionally or not, just as Matt’s post was commenting on? Doesn’t it even risk lawsuits from candidates challenging the use of the test in the hiring process?
  • They say it’s impossible to game, but I find that difficult to swallow. A picture of a puppy, a picture of a skateboarder… hmmm… let me guess what you want me to like, then I’ll answer that way. Or, “I’m feeling mischievous today, I’m going to answer A or B based on the drum solo in Inna Gadda Da Vidda”. I tried it with my photography background in mind — I picked ones more in focus. The result was gibberish.
They say it’s impossible to game, but I find that difficult to swallow.
  • It raises the “what do I do with this info” question. Let’s say it comes up with “you are an unmotivated, introverted, a&$hole”, what do I do now? Take it again and choose all the other pictures? Try to change? Go see a shrink?
  • How do I know if I’d like it there from the results of this test? Because everyone is like me? And just as the post on photomatt was talking about, don’t I want to be different? Isn’t is a good thing to not be like everyone else? Or should I want to go there just to change the culture?
  • As for enticing people to apply, I find this assertion stunning… If I was asked for any kind of psychological test, I’d immediately withdraw my candidacy. It tells volumes about the kind of company it is, (see this post). It speaks to the kind of company culture there is. They don’t see people as individuals and potential valuable assets, but rather as test subjects, resources, automatons. I think this drives away candidates, not attracts them.
  • Is the testing based on lots of psychological merit, or is this just some guys who have a great idea? Do they have PHDs on staff who are developing these tests? Have the tests received any kind of peer review in the psychology world?
  • Do they charge by the test? Or by the results? Does [Microsoft/Google/hiring company] get the results in some permanent way, or am I tied to their web site? Do they own the results or do I?
  • How do they control access to the tests? Who can see results? My manager? Only HR? The police/FBI (we caught a child-porn king at your company, we want this as evidence)? How do I tell them who controls that access? Doesn’t that mean I basically have to upload a complete company-wide global hierarchy? Do I host the test on my systems? If not, how do I know their systems are secure (aren’t being keylogged, etc.)?
  • Can I stop the test (my phone just rang) and pick it up tomorrow? Doesn’t that potentially change my answers? What’s preventing me from “team-taking” the test, with me and 6 of my drinking buddies sitting around and having fun taking it?
  • There are many stills from movies and television, as well as some familiar looking images. Are they all licensed properly?
  • Last but not least, how does one define “like better”, as in “which picture do you like better”?

That’s a lot of questions/concerns to be sure. But it doesn’t even hit my most important concern: bias. Do people from the US or India or Europe or China all respond to the same pictures in the same way? Do they even know what some of the images are in the pictures?

But it doesn’t even hit my most important concern: bias.

There’s the classic story about an original version of MS Mail that used a mailbox for a symbol (you know the long box with the round top and a flag on the side). Turns out that’s not at all what a mailbox looked like in 85% of the world, and most people thought it was a trash can. How do they control for that?

What about racial/ethnic bias? In some parts of the world black people are rare/unheard of and are symbols of evil, in other parts of the world they are what everyone looks like. And certainly, to some people a skater-boy with baggy pants is an icon, to others it’s a nuisance, to still others it’s a “rich white kid”.

And reviewing the site only makes me more concerned. As I noted there are dozens of culturally specific images. Stills from the Sound of Music, the Lord of the Rings, the roadrunner, the Three Stooges, South Park, Titanic, Gladiator – won’t results differ depending on whether I’ve seen these?

There are snowy scenes, what if I’ve never seen snow? An old dial phone, a marijuana plant, and the Eiffel tower – isn’t it possible I have no idea what these are? Food dishes with meat and shrimp, what if I’m a vegan? Things written only in English on the images (“die” in one example).

And it turns out I didn’t need to be concerned about pictures of black people. All of the people I saw on the test were white. Ouch.

This test looks like just a new take on the long history of testing nightmares. I don’t know how anyone could or would want to use the results to tell them anything they’d want to know. Cute game, but not much more.