As researchers, we’ve all encountered the data-skeptical client. You know, the one who was eager for the survey results…until they read the report.
They receive the findings and almost immediately push back: “I don’t think this data is reliable.”
Why? Maybe the results challenged their perceptions of customer attitudes and behaviors. Maybe the data suggests their pet product idea is more likely to flop than fly. Or maybe the findings revealed brand weaknesses they weren’t prepared to hear.
Whatever the reason, it can be easier for clients to challenge the data than to reconsider their assumptions.
That’s where our role as researchers matters. We can’t make people like the results, and we can’t make them use the data. But we can be ready to explain how reliable the survey data is, what factors affect that reliability, and why discomfort with the findings is not the same as a data quality problem.
In a recent Research Rockstar Live Event, instructor Julie Worwa covered several concepts that help researchers do exactly that: evaluate survey data reliability and communicate it clearly to clients and stakeholders.
One note before we dive in: in client conversations, “reliability” is often used broadly to mean “Can we trust this data?” Here, we’re using it in that practical sense while also looking at the statistical concepts that support confidence in survey results.
1. A Census Is Ideal, But Most Research Uses a Sample
Confidence in the data is easier when we’re doing a census because we’re talking to everyone in the targeted group. But in most survey research done for business purposes, we conduct research among a sample, which is a subset of the target population.
That means we don’t survey every single person in the group we care about. And that’s why data reliability matters: we need to know whether the data from our sample can reasonably represent the broader population of interest.
2. Many Types of Error Can Affect Survey Results
Several types of error can affect survey results, including:
- Population-specific error: Surveying users rather than decision-makers
- Selection error: Including only respondents who have strong opinions on the topic
- Sample frame error: Using the wrong source data, such as an outdated customer list or one with incomplete contact information
- Non-response error: When people chosen for the study don’t participate or drop-out
- Sample size miscalculations: Having a sample that is too small to effectively represent the target population
- Sample contamination: Having unwanted or incorrect information in the sample
These are not abstract issues. They shape how much confidence we can have in the results.
3. Sample Size Calculations Matter
Sample size calculations help determine the minimum number of observations needed in a data file to estimate a population parameter with confidence and precision.
Julie described Cochran’s formula, which solves for n, the sample size needed. Cochran’s formula uses your chosen confidence level, your acceptable margin of error, and an estimate of how varied responses will be. There are several commonly used equations for calculating sample size. While some researchers rely on standard calculators, we can often get a more precise sample size when we know the population size or expected variability.
4. Standard Deviation Measures Data Spread
Standard deviation measures the spread of data around the mean.
A low standard deviation indicates that the data clusters close to the mean. In survey terms, this means most respondents gave similar answers. A high standard deviation indicates more variability, meaning responses differed more widely.
In normally distributed data, about 68% of the data falls within plus or minus one standard deviation from the mean, and about 99.7% falls within three standard deviations.
5. Similar-Looking Results May Not Be Statistically Different
Responses to questions may look similar when we compare means or top-two-box results. But if the standard deviations differ, the pattern of responses may differ too.
When two groups have substantial overlap, they may show no statistically significant difference—even if the peaks look different visually. In Julie’s Yummy Donuts example, the male and female means looked slightly different, but the t-test showed the means were not significantly different.
6. Confidence Level and Confidence Interval Are Different
This is an area where colleagues and clients can get confused, so researchers need to be clear.
- A confidence level indicates the probability that the survey would capture the true population parameter if repeated multiple times. At a 95% confidence level, if you repeated the same sampling method 100 times, about 95 of the resulting intervals would contain the true population value.
- A confidence interval is the range that goes with that confidence level: at 95% confidence, it’s the band of plausible values calculated so the method captures the true population value 95% of the time—not a single point estimate.
7. Be Careful Comparing a Subgroup to the Total
In the Q&A, Julie explained that comparing a subgroup to the total breaks statistical rules because the subgroup is already included in the total. That means we’re comparing it partly to itself, which muddies the statistical results.
A better approach is to use a test such as ANOVA to compare mutually exclusive groups to each other. For researchers looking to advance their skills here, it’s useful to understand when to use ANOVA, z-tests, and chi-square tests.
How to respond when a stakeholder questions your data
Survey data reliability is more than a statistics topic. It’s part of how we help clients and stakeholders understand what decisions the data can—and cannot—support.
When someone says, “I don’t think this data is reliable,” we need to be ready with a clear, grounded response. That means understanding sample quality, sample size, variability, confidence intervals, and statistical significance well enough to explain them without turning the conversation into a surprise statistics lecture.
We can’t force stakeholders to accept findings they don’t like. But we can address misplaced concerns about reliability, and help research clients separate “this result surprises me” from “this result is wrong.”
Ready to learn more about conducting professional-level survey research?
