Survey Says

Note: Podcasts on Leading Smart are produced for the ear and designed to be heard. If you are able, I strongly encourage you to listen to the audio, which includes emotion and emphasis that's not on the page. While the text is carefully transcribed, it may contain errors. Please check the corresponding audio before quoting in print.

STOP, it read. Big, bold, black letters. You could see the rage in the strokes of the oversized marker. It couldn’t have been clearer: the sender really wanted to be removed from our survey sample.

The only problem was, we couldn’t oblige. Not because of some legal or business requirement. Not because we didn’t want to remove them. No, we couldn’t stop sending surveys because we had no idea who they were. The furious sender had carefully removed all traces of their identity. They’d blacked out all clues so thoroughly the marker had bled through to both sides, likely staining their own desk in the process. They’d even scratched away the magnetically coded numbers at the bottom that might have provided any help. I’m sure the follow-up mailings only made them even angrier. I could almost hear the enraged phone call our poor operator was going to receive.

As I noted early in season one, my first full time job was working for National Family Opinion. We did “panel” market research, where people signed up to be peppered with ongoing studies that tracked their attitudes over time and for the long term. They’d get small envelopes with surveys, the shape and size of a computer punch card that had questions on a variety of topics. Each envelope carried up to a dozen small surveys. Each survey had magnetic ink at the bottom, like a bank check, a key to statistical variables like the age, gender, and so on of the respondent. All part of ensuring a representative sample.

This poor person either was tired of the surveys or didn’t fully realize what they signed up for. And now they wanted out. Theirs was not an uncommon reaction, we had even received bricks attached to our postage paid return envelopes. But with this one, their anger swamped our ability to comply.

There are two key questions common to all solicitations of feedback. Whether it’s market research, political polling, or even an interpersonal inquiry, the same two questions come up over and over. Are the answers sincere and do they accurately represent the entire population? Get either wrong and the conclusions are way off. Just look at the last several elections and the polls that preceded them.

Managers who want to understand the health of their organization face the very same issues. To get a true picture of what’s really going on you need to understand how to get good feedback that is both representative and genuine. And that’s what this is all about.

This is Leading Smart, the show about Managing in the Brainpower Age. It’s a field guide to the joys and challenges of leading and working in the modern workplace. I’m Chris Williams, your guide to the stories and ideas that I hope will inspire you to be a better leader in the world of knowledge work.

This episode continues the series on communication as a leader. This time we look at how to ask for input. This is Episode 225 – Survey Says.

Leaders who follow the path we’ve discussed here in the last several episodes will ultimately wonder if all the communication they are doing is working. They’ve carefully defined a clear vision, they have made the case to anyone who would listen, they have echoed it time and again. But, still, the question remains: are the messages resonating with the team?

Finding the answer isn’t easy. Getting good, sincere, and timely feedback is hard even at the personal level. Who among us hasn’t asked a loved one “what’s wrong?” and received quite literally “nothing” in return?

Quality feedback is even more elusive with groups of people. Discerning the broad attitudes of a population has been the challenge of market researchers and political pollsters centuries before those were actually job titles. 

In 1982 Tom Bradley, the long-time mayor of Los Angeles and an African American, ran for reelection. Polls leading up to the election showed him with a sizable lead and exit polls on election day projected an easy win. The San Francisco Chronicle even trumpeted a headline to that effect. 

But once all the votes were counted, Bradley had lost to George Deukmejian. At least one post-election analysis concluded that racism was to blame. That in an effort to not appear racist, white voters had told pollsters they intended to vote for Bradley. Yet when alone in the voting booth, they voted for the white Deukmejian.

While subsequent analysis offered several alternative explanations for the late Deukmejian surge, it still led to the coining of the term the Bradley Effect. This effect notes that survey participants have many ulterior motives and aren’t always honest with pollsters. The Bradley Effect has echoed many times in the last several years, from the overestimation of Obama support among black voters to the specter of the “shy Trump voter” reluctant to voice their support to pollsters. Pundits frequently point the Bradley Effect introducing polling errors to explain their miscues. 

Whatever the source, it’s hard to look at the wide variances between polls and results in the last several US national elections and conclude anything other than “accurate polling is hard”. 

When faced with the challenge of taking the temperature of their team most leaders reflexively turn to their own staff. They’ll ask in a meeting: “How are your teams doing? Do they believe in the vision? Is everyone on board?”

A smart leader is well attuned to the inherent bias in the replies. Even trusted lieutenants will color their responses in ways that benefit them. They will, even if subtly, exaggerate success and downplay failure. It’s simply human nature, completely expected, and entirely undeniable.

In an effort to seek a more objective perspective, the leaders often turn to their Human Resources department. They ask HR for their read, hoping for a fresh perspective. But for reasons we’ll discuss in a future episode, HR too lacks a clear lens into the team. So, HR in turn frequently leans on an old crutch: the employee survey. Alas, employee surveys suffer all the challenges of election or market polling. But they have the added complexities of organizational politics and personal consequences for the respondents.

“100 people surveyed, the top five answers are on the board, here’s the question…”

The first difficulty in any survey is reaching the right people, a representative sample. Knowing the right people to ask, matching that group to the population as a whole, ensuring that the selection of the sample doesn’t bias the result in some subtle yet crucial way. There’s an entire field of study dedicated to the many forms of sampling: random sampling, cluster sampling, quota sampling, panel sampling… Each with their own quirks, biases, and margins of error. Polling a subset of your own organization is fraught with all these risks.

If your universe is small enough, or the need compelling enough, you might decide, “let’s just ask everyone”. Yet even if you solicit the entire organization, quality responses can be difficult to come by. In an era when people are bombarded by messages, ignoring the question is often the easiest response. And when the result might have a direct and unpredictable effect on one’s work life and employment, many simply pass. 

Most often, the only people who respond are those with reason to do so. Either they are deliriously happy and want to share that, or they are aggrieved and want everyone to know. In either case, they hope for some benefit. That their passion and loyalty will be recognized, or that their grievance will finally be addressed. The potentially vast middle of respondents simply goes unheard. 

In larger organizations, voluntary response rates to employee surveys above 50% are exceptional. And if compelled to respond, many of their responses bear all the sincerity of a thank you note to one’s grandmother.

This raises the second and even more challenging problem with broad requests for feedback: determining if the results are genuine. Respondents routinely game the system, as described in the Bradley Effect, carefully crafting their answers. They may exaggerate their sentiments in hopes of amplifying the results. Or they answer in ways they feel their management wants to hear. Some humorously or even nefariously try to spoil the validity of the survey. Hiding in the anonymity of a broad survey allows for all manner of unpredictable behaviors.

A major reason for these Bradly Effect behaviors is that employees don’t know or don’t trust what the survey will be used for. Is it simply for broad measures or with a specific goal in mind? Does the way I answer this survey effect my future job prospects? Could that color my relationship with my manager?

If used for more specific purposes, such as for performance reviews as we discussed way back in Season One, surveys are a minefield of potential issues. Employees worry deeply about retribution and will often soften or even hide their true feelings lest they ignite the ire of their manager. More often, the purpose of the survey will be unclear, it’s consequences a mystery, and the wary employee will follow their mother’s advice: if you can’t say anything nice, don’t say anything at all.

In short, employee surveys done from within an organization are fraught with errors, are subject to the Bradley Effect, and are rarely an effective measure of the organization. To mitigate these issues, many try to use distance to improve the quality of the survey. Distance either organizationally or distance in time.

Some leaders turn outside for assistance in judging organizational health. They use external resources to provide perspective, relying on outside firms for surveys, focus groups, and other input. Yet these firms offer only a veneer over the same surveys done internally and suffer virtually identical fates of reliability and sampling error. Cautious employees, wary of consequences, will be no more transparent with a third party whose independence is unknown.

Some brave managers look to sites like GlassDoor that allow anonymous people to post their employer reviews. Outside of their organization’s control, these sites can offer unvarnished perspectives. But here again, the only posts you tend to see are those at one end of the spectrum or the other. Either they are unrealistically enthusiastic, or vehemently negative, rarely are they a representative sample. And it takes a particularly thick skin to read much of the commentary.

A more effective distancing mechanism, however, is affording respondents the distance of time. It can be an effective way to get less exaggerated and more thoughtful comments on the health of an organization. When I was in HR at Microsoft, we at times skipped the traditional exit interview, which was often fraught with tension and pain. We opted for a delayed post-employment interview, often as much as six-months or a year later. And when on the Board of the Overlake School, we chose to wait until sophomore year of college to query graduates on their secondary school experience.

These interviews, almost always done by phone or in person, offered by far the best results. The delay afforded the respondent time to garner perspective and avoided the immediate pain points that discolored their opinions. The distance also isolated them from the consequences of frankness, and usually led to more open and honest assessments. Their answers were often more measured, and they frequently offered very useful insights into the health of the organization.

Delayed results like this do little to help urgent issues, to be sure. A failing project might well have long since imploded by the time the survey is done, for example. But the terrible manager or teacher could still be there, making life miserable for later generations. A frank responder could highlight a heretofore undetected problem. In addition, assessing the culture that changes only slowly is often still valid many months later. 

In general, we found that the quality of the responses received to this delayed outreach made for extremely useful feedback. It made for at least one way to get survey input less prone to the Bradley Effect.

So, in total, surveys are a poor judge of organizational health. They fall short on the two key metrics of good feedback: they frequently fail to present a representative sample and they rarely allow for genuine, quality responses. This means that, even if they are earnestly accomplished, leaders who rely on survey do so at their own peril. Fortunately, most often employee survey results are simply ignored. By both the employees and the leaders who commission them.

“Survey Says!”

So if you can’t ask your staff, HR can’t help you, and employee surveys aren’t worth doing, how is a leader supposed to assess the health of their organization?

Simply by asking.

I’ve talked about MBWA — management by walking around — several times in this podcast. One-on-one frank conversations are the best way to assess the health of your organization. These conversations evoke quality responses that can be evaluated in real time for their sincerity. Thus, the Bradley Effect can be minimized. And though the sample size may be small, with careful consideration, a leader can be assured that, combined with the input of their staff, it is a true and clear representation of the organization.

Next time we’ll look into a how to do effective one-on-one conversations to evaluate your organization’s health. How to decide who to talk to. What to talk about and what to avoid. How to use the results of the conversation. One surprising factor doesn’t have to do with who, what, when, or how it happens, but rather where it happens. We’ll look at the turf war that can change the conversation entirely. But that’s next time.

In the meantime, remember that what the “survey says” is often skewed and you need to take most of it with a grain of salt.

Leading Smart is from me, Chris Williams. You can find out more about the show and discover other resources for leaders at my web site

If you like the show, please share it with your friends especially on social media. Referrals are the greatest source of new listeners. I’d also love your feedback. I’m “theCLWill” on Twitter, LinkedIn, and Facebook, or send an email to

That’s it for this episode. The next episode is another of my conversations with leaders. We’ll talk with Adam Bosworth, who has chased a goal of making hard things easy for over forty years, at several of the world’s leading tech companies. I hope you’ll listen. Until then, please remember that each of the several dozen decisions you make today are part of Leading Smart.