Every few months I get the same call. A founder, an Insights team leader, a CMO who just spent the better part of a six-figure budget on customer research and ended up with data they can’t trust. The questionnaire looked fine on the surface. The survey researcher gave confident recommendations. But the actual results? Off point and unreliable.
This is survey research’s expertise problem. It looks easy from the outside. Almost everyone has taken a survey, so writing great questionnaires feels obvious. Which makes survey research a textbook case of the Dunning-Kruger effect; the curve where people with the least knowledge of a topic are often the most confident in their abilities within it. Kinda like how people who watch cooking shows say they’re great cooks, then serve you burnt, bland food.
So how do you screen for real expertise when you may not be a survey researcher yourself?
Over the years I’ve hired a lot of survey researchers, as contractors and employees. The strong ones are easy to spot once you know what to listen for. The dangerous ones appear confident and have a few good war stories, but deliver disappointing data quality and weak insights.
Here are the seven questions I rely on most to screen survey researchers (my full list is 21, but some are a bit esoteric and only apply to very specific types of studies such as market segmentation, price sensitivity, and UX Research). None are trick questions. A genuinely experienced survey researcher will answer them comfortably and with examples from their own work.
1. How does the sampling plan impact questionnaire design?
Listen for: the sampling plan defines the target population of interest and shapes everything. It informs reading level, screening logic, length tolerance, mode (mobile vs. desktop, phone vs. online), and which questions you can even ask. A researcher who misses this connection is a red flag.
2. What’s the difference between a Likert scale and a semantic differential scale?
Likert scales measure agreement with a statement (conventionally Strongly disagree → Strongly agree, though there are variations). Semantic differential offers the respondent two opposing descriptors (Very easy → Very difficult, Fun → Boring). Both are valid options, but Likert scales are more susceptible to acquiescence bias. Anyone calling themselves a survey expert should be able to explain this.
3. When would you use a rank-order question instead of a rating question?
When you need prioritization. Ask people to rate ten features individually (each on, say, a 5-point importance scale) and many people will rate all ten features "very important”, which is pretty useless. Ranking forces trade-offs so we learn what actually matters most.
4. What’s the difference between incidence rate and completion rate?
Incidence is the percentage of screened people who qualify (for example, a study on EV buyers might require someone who’s purchased an electric vehicle in the past two years). Completion is the percentage of qualified respondents who finish. Any survey researcher should answer this easily.
5. What percentage of respondents typically give usable answers to open-ended questions?
There’s no single right number, but the answer should be "it varies", followed by why it varies. Blind surveys (sponsor unknown) tend to get unhelpfully short open-end responses. Brand-identified surveys, especially for brands people like, get much better responses. If someone claims they routinely get 70% or better high-quality open-ended responses (people didn’t just type gibberish to progress to the next screen) on blind surveys, that’s a red flag.
6. When reviewing the final data file, what three things do you check to flag bad records?
Strong answers include speeders (faster than humanly plausible duration time), straight-lining in grids (selecting the same answer for every item), gibberish in open-ends, and inconsistencies across related questions. Data quality is a huge risk factor in survey research, and these are the most basic steps real researchers take to mitigate it.
7. What is "weighting" in survey research?
Weighting is a common statistical adjustment that makes the sample collected more accurately reflect the target population when data collection resulted in certain demographic groups being over- or under-represented. Example: a survey about frozen vegan entrees missed its quota for males with college degrees. Weighting can correct for that if we know what the true percentage is in the target population. But weighting is a tool, not a magic wand, and a good researcher will tell you that weighting also has its risks.
Asking Questions that Find Real Survey Research Experts
A few good questions, asked early, can save a six-figure customer insights project from going sideways. If a survey research contractor or employee candidate handles these comfortably and correctly, you’re likely talking to someone who’ll deliver data you can actually trust.
If they don’t? Resist the urge to "be flexible." Real survey research experts exist, and you can find them.






