Testing Limits

Episode #7
Note: Podcasts on Leading Smart are produced for the ear and designed to be heard. If you are able, I strongly encourage you to listen to the audio, which includes emotion and emphasis that's not on the page. While the text is carefully transcribed, it may contain errors. Please check the corresponding audio before quoting in print.

I had been looking forward to this lunch for some time. She is an old friend and colleague, now a senior executive at one of the world’s premiere tech companies. She leads a team of well over a thousand engineers scattered around the globe through which moves tens of billions of the company’s revenue. If you dwelled on the enormity of the responsibility, it would be debilitating. She handled it almost effortlessly. Or at least made it seem like that.

Our meeting was a great chance for me to see her in her element, and to explore her world. Although we had entertained it, we both only half-heartedly thought there might be a place there for me as well.

We met near her office, and the hustle and bustle outside the windows of the restaurant was invigorating. The variety of people was also somewhat unexpected. I’ve seen lots of tech firms and they tend to look a lot alike – at least with respect to age. It’s usually a fairly narrow band from mid-twenties to late thirties. But here there quite a few people with more than a little grey hair.

She noted that the company respected age, experience, and wisdom, and many of the senior managers reflected that. Having been a respected leader for decades, that was one of the reasons she felt comfortable there.

I noted that it also looked like a fairly diverse crowd in other ways too. She hung her head and muttered “if only that were really true.”

You see, her team was – at least at the engineering level – shockingly homogeneous. It was made up of almost exclusively young Asian men. They were very good programmers, but there wasn’t much diversity of thought. And she feared those lone engineer types presented a very shallow management talent pool.

She told me how it happened. The company was inundated with applications, tens of thousands at a time. They needed some way to winnow the field. So they implemented a series of automated tests for engineering applications. The tests were timed and scored, with only the top making through to even be considered.

It turns out that the company had set up a filter to find not great engineers, but great test takers. People who have spent a lifetime practicing that skill, can recognize trick questions, and can do it at light speed. By screening for this one skill they accidentally removed the creative problem solvers and the great team players. In short, they got precisely what they asked for. And that’s what this is all about.


This is Leading Smart, the show about Managing in the Brainpower Age. It’s a field guide to the joys and challenges of leading and working in the modern workplace. I’m Chris Williams, your guide to the stories and ideas that I hope will inspire you to be a better leader in the world of knowledge work.

In this podcast, we’ll take a look at how people meet the challenge of managing smart people in this Brainpower Age. Each episode, we’ll explore everyday problems and provide practical tools you can use to be a better and smarter leader.

This is the second of our episodes looking at the challenge of hiring brainpower workers. In this episode we’ll look how to find, identify, and filter for junior level talent. This is Episode Seven: Testing Limits.


I was Vice President of Human Resources for Microsoft at the height of the dot com boom at the end of the 1990s. It was a heady time for the tech industry, and especially for Microsoft. There was limitless enthusiasm about what the internet could do for virtually every industry, and Microsoft was incredibly well placed to take advantage.

Yes, Microsoft had been mocked by the tech press as coming a bit late to the internet party. But the company had proven time and again that it was a great tail-light chaser, coming late to a market then speeding past its rivals. With the enormous success of Windows and Office, and a stock price that seemed inexorably on the rise, the company was the hot place to work at the time.

We were deluged with resumes, an average over 1000 a day though it occasionally peaked at many times that rate. All of them arriving in the mail, on paper. We had a team of people, sometimes two shifts during peak periods, handling the tidal wave. They scanned them in, did optical character recognition, tweaked those results (which always needed help), then shepherded them into a massive database. It often seemed overwhelming.

There were far too many applicants for even our 300-person recruiting team to sort through, so we resulted in keyword selection. We filtered for mentions of common computer terms, programming languages, or specific skills. Any way to get the list down to a manageable size.

It seems obvious in hindsight, but this was a broken system. People of a certain age will remember Groucho Marx and his game show where to “say the secret word” won you $50. This was not that different, except the prize was a job. If your resume didn’t use the exact phrase, spelled just so, it never saw the light of day. And you never knew why.

Eventually, applicants figured it out and we started receiving resumes that looked like the pages of a technical dictionary, mentioning every possible keyword. It got ridiculous, and it certainly didn’t filter the candidates in any useful way.

Companies have tried all kinds of ways to filter. Testing is a common way. There are skills tests. Trick questions. Timed quizzes. Even personality tests. As I noted in the first episode, it happened to me in the 1970s, and it was pretty stupid even then. Developing really good and fair tests is a complex art, that even experts fail at. Even if they’re fair, these tests mostly filter for people who are good at tests. There are internet courses to help people practice for the tests, just to make it past the filter. And if you’re a nervous test taker, or a slow, careful thinker, or someone who excels at working in a team, you never even get a response.


In their defense, though, employers are simply sifting through their enormous pile of candidates trying to find the best ones. But to do that effectively they need to carefully define what makes the “best” candidates. And for brainpower workers, that’s not easy.

The best such workers come in a wide variety, and from a broad range of sources. For example, some of the best program managers we had at Microsoft, even the most technical ones, had degrees in the humanities: English, History, or even Philosophy. They were great thinkers and communicators. Requiring a particular course of study, say computer science, would artificially cut off some of the best and brightest.

The best brainpower workers are also voracious learners. They often to live to learn. Even if you could fairly and effectively test them, judging them by the skills they have today is incredibly short-sighted. Great learners will obliterate any skill deficiency in no time at all.

And most importantly, the best employees are passionate about what they do, and throw their entire selves into their work. Over the years, I’ve seen passion and drive win up against skills and technique time and time again. But, I’ve yet to see a test that can measure this any meaningful way.

In short, you can’t test or filter your way to find great brainpower workers. There’s just no fair and reliable way to separate the wheat from the chaff.


There’s also no way to guarantee the best ones will apply. If you’re a smaller organization, you need to figure out how to attract the applicants to filter in the first place. You’re not as well-known as the big companies, but you might have an equally fascinating opportunity. Without the name recognition or a massive ad budget how do you compete for attention?

Even Google, Amazon, or Microsoft can’t be assured that the best candidates will apply and wind up in the bucket for filtering. Maybe the prospect doesn’t want to work for a big company. Perhaps they don’t know about all the opportunities across a huge company. Or maybe they feel they simply won’t make it past all the filters.

And just like we discussed in the last episode, even if you could manage to grab the attention of enough applicants, you have to ask if they are the best and brightest. Are the people trolling Indeed or LinkedIn for jobs really the potential superstars you’re looking for? Or are they perhaps looking for work for some less than ideal reason? It’s hard to tell.


The best recruiters, the ones that find and land the best talent, have long ago realized that finding the best junior brainpower workers isn’t that much different than what we discussed in the last episode for the senior ones. It isn’t an inbound task, it’s outbound. You can’t toss out some bait, then sit back and hope. You have to find them.

Like with senior workers, you want to find people who’ve already shown they can do great work, can learn new skills, and have passion for what they do. Even with first time workers, there are places where you can see that in action.

That’s why the best recruiting we did at Microsoft was on college campuses. I get a lot of pushback when I say this. People cite the many non-college graduates who’ve made it big: Henry Ford, Oprah Winfrey, Steven Spielberg. Tech is rife with them: Michael Dell, Steve Jobs, even Microsoft’s very own founders, Paul Allen and Bill Gates. None of them had a college degree. I get it.

But college is a wonderful place to find the kinds of skills you want in great brainpower workers. It’s a place where learning is a practiced skill and is richly rewarded. It’s a great lab to hone vital communication skills. It helps many narrow their focus and find a passion. And getting a degree is a crucible that tests one’s ability to complete a difficult and complex multi-year project. In short, college is a great hunting ground for brainpower workers.

To be sure, there are other places that can produce similar results. One of the best program managers I ever worked with was a stand-out in the Air Force. He was thoughtful, dedicated, and thorough in every task. I’m sure some of that was learned in the service. I’ve seen people who graduated from rocky startups, or from community colleges, or even tough high schools, who showed the signs of brainpower excellence in their own ways.

From an employer’s perspective, however, you need to go where the hunting is best. And often that’s a college campus. Because the filtering there is done for you, and you find hundreds in one place.


You can start at their career fairs. Those are great ways to get to see a lot of potential candidates quickly. And get your organization some name recognition. But I also strongly encourage you engage the faculty, especially in areas of your interest. They know the students, and they certainly know the stars. That’s pretty much what they work for, to set the ones with the most potential off on great futures.

If you put in some time and energy researching good schools, and especially good programs led by great faculty, you can find some of the best future candidates. If you’re small, start locally. Engage the local college’s business school, computer science classes, humanities departments. Find them by going to events they organize, working through their career counseling offices, or even just calling straight into the department heads. These kinds of connections can find you everything from potential interns, strong graduating candidates, even faculty members looking for work on the side. And these relationships will pay dividends for years as classes turnover and new students arrive.

Also, don’t forget community colleges, trade schools, even veterans’ outplacement services. The pickings here are thinner, as many of these graduates are choosing non-brainpower type fields. But each and every student has shown the ability to learn, work through a course of study, and complete a major milestone. For the potential employer, these basic skills separate them from the random LinkedIn resume by a large margin. In addition, these are likely places where competition from larger, more well-known, organizations is less intense.

For larger firms, I would turn most recruiting operations inside out. If I was building a recruiting team today, I’d build a two-layer team: one focused on the sources, one focused on the destinations. That is, some recruiters should be experts on the schools, companies, and other sources for your top candidates. They should be building relationships, researching for new sources of candidates (especially internationally), and in general finding where the candidates you want are coming from. The other team would be working inside your organization, understanding the kinds of people you need, the opportunities that are most urgent, and the internal leaders who can land the best talent. I would make sure these two groups interacted often to exchange market information, and to work to match the sources with the opportunities.

In addition, one source everyone should pay attention to is right in your building. The great brainpower workers you already have. Smart people know more smart people. They know their alma maters. They know the great professors and programs. Engage these people in the effort to develop these sources.

I’m also a big fan of using recruiting bounties, rewards for referrals of top candidates who end up making a difference. Like all incentives, you certainly have to be careful in designing them. Incentives are a topic we’ll cover in detail in the future, but when thoughtful, they can be very useful. Encouraging your great brainpower workers to find and land some of their peers to join your organizations is one of the best ways to find great talent.

Finally, don’t take all of this as me saying I would never advertise. Or that I would completely discard some attempts at filtration. Advertising and the resulting pool of responses can be useful in understanding the market. And it can help to raise some awareness for your organization. You might even flush out a few good candidates. Perhaps the next Bill Gates is hiding in there. But, like I mentioned in the last episode, this should not be your main focus or the largest of your efforts.

And please, please, don’t use testing to filter 80% of your candidates. Perhaps it can be useful to separate the clear spam from the queue, but if you’re using it to stack rank them, or to eliminate more than half of them, you’re doing it wrong.

In summary, in my experience, I’ve always found that you get the best candidates by going to them. By searching them out. But asking them questions. And that’s what the next episode is all about. Once you’re talking to a candidate, how do you select and land the best of them? But that’s next time.

Until then, remember that testing has its limits, and that finding great brainpower workers is your job, not theirs.


Leading Smart is from me, Chris Williams. You can find out more about the show and discover other resources for leaders at my web site, CLWill.com. That’s C-L-W-I-L-L.com. Or find me on social media as theCLWill.

As a new podcast, Leading Smart could really use your support. Please subscribe and rate the show in whatever podcast app you use. Tell your friends about it as well.

I also hope you’ll participate. Do you have questions about managing people? Do you work in an industry that faces these kinds of challenges? Are you, or do you know, someone I should be interviewing? Let me know. Each episode has a page on my web site, and comments are welcome. Or just send an email to pod@clwill.com. I can’t promise I’ll answer or interview everyone, but I read every email I get.

That’s it for this episode. In the next episode we’ll continue our look at hiring brainpower workers. Next we’ll look at interviewing and landing brainpower workers, and it’s called “Wonder Why”. I hope you’ll listen. Until then, please remember that each of the several dozen decisions you make today are part of Leading Smart.