Comparisons
Online Survey Tools for Startup Validation: What They Show
Surveys are fast and cheap, which makes them tempting for validation. They are good at measuring how common something is among people you can reach. They are poor at telling you whether anyone will pay. Here is how to use them well.
Online survey tools help startups measure how common a problem or behaviour is among people they can reach, and compare segments. They cannot tell you reliably whether people will buy, because answers about future behaviour are weak evidence and online samples can contain careless or fake respondents. Use surveys after interviews, to size what you learned, not instead of them.
Pew Research Center, one of the most careful survey organisations in the US, has shown how badly online samples can go wrong. An online opt-in poll in December 2023 reported that 20% of US adults under 30 agreed "The Holocaust is a myth." When Pew repeated the question with probability-based methods in January 2024, the figure was 3% (Pew Research Center).
Key takeaways
- Surveys measure frequency, not willingness to pay. Use them to count, not to decide.
- Who answers matters more than how many. Opt-in panels can include careless or fake respondents.
- Interview first, survey second. Interviews find the right questions; surveys measure the answers.
- Small subgroups have large margins of error. Be careful comparing segments.
- Pair survey interest with a real ask, such as a pilot or pre-order, before treating it as demand.
What can an online survey tell a startup?
How common a problem, behaviour or tool is among the people who answered, and how that differs between segments. That is useful for sizing and prioritising what you learned in interviews.
| Good at | Poor at |
|---|---|
| How many respondents use a given tool or workaround | Whether they would switch to yours |
| How often a problem happens, as reported | What it costs them, precisely |
| Ranking known problems by how common they are | Finding problems you did not know to ask about |
| Comparing segments you can recruit in enough numbers | Predicting purchases at a price |
Why can online survey samples be misleading?
Because the people who answer may not represent your buyers, and some may not answer honestly at all. Opt-in panels in particular can include respondents who answer carelessly to earn rewards.
Pew's analysis found that "bogus respondents", people answering insincerely to earn rewards, can inflate rare attitudes and behaviours, and that they disproportionately claim to be under 30 or Hispanic, possibly to get past screening questions (Pew Research Center).
For a startup, the practical lessons are:
- Recruit from where your buyers really are, such as your own contacts, communities and customer lists, rather than a general panel, when you can.
- Add an attention check and screen out respondents who fail it.
- Be suspicious of surprising results from paid panels until you can confirm them another way.
How many survey responses do you need?
It depends on how precise you need to be and how many segments you want to compare. Small segments carry much larger margins of error than the whole sample.
Pew explains that "survey estimates on subgroups of the population have fewer cases", so their margins of error are larger, "in some cases much larger". In its example, a national poll of 1,067 people had a margin of about ±3 points overall, but a subgroup of about 160 people had a margin of about ±8 points on the same question (Pew Research Center, 2016).
For startup decisions, that means a survey of 100 people split five ways tells you very little about the differences between the five groups. Decide in advance which comparison matters, and make sure that group has enough responses.
Can a survey tell you if people will buy?
Not reliably. What people say they will do and what they do at a price are different things. Treat purchase intent from a survey as a lead, and test it with a real ask.
The Mom Test, as summarised by Sachin Rekhi, advises asking about "specifics in the past instead of opinions about the future", and validating with commitments of time, reputation or money. A survey question such as "Would you pay $30 a month for this?" is exactly the opinion about the future that advice warns against.
Better survey questions ask about the past: "In the last month, how many times did this happen?" or "Which of these tools do you pay for today?"
How should a startup use surveys in validation?
Interview first to learn the right questions, survey second to measure how common the answers are, and then test demand with a real offer.
- Interviews with people in your target role, to find the problem and the words they use for it. See our guide to customer discovery.
- A short survey to the same kind of people, asking about past behaviour and current tools.
- A real ask to the most relevant respondents: a call, a pilot or a pre-order.
Keep surveys short, ask about the past, and include one question that screens for your buyer. If you are a student on a limited budget, our free survey platform workflow walks through the same steps.
What should you look for in an online survey tool?
Logic to skip irrelevant questions, attention checks, export of raw responses, control over who can answer, and pricing that fits the number of responses you need.
Most tools handle basic question types well. The differences that matter for validation are screening logic (so non-buyers are routed out), the ability to export raw data for your own analysis, and whether you can recruit your own respondents rather than relying on a panel. Buying panel responses is fast, but as the Pew findings show, it needs careful quality checks.
Frequently asked questions
Are online surveys good for startup validation?
They are good for measuring how common a problem or behaviour is among people you can reach. They are weak for predicting purchases, so combine them with interviews and a real offer.
How many responses does a startup survey need?
Enough for the comparison you care about. Subgroups carry much larger margins of error than the full sample, so a small survey split into many segments tells you little about the differences between them.
Should I buy survey responses from a panel?
It is fast, but opt-in panels can include careless or fake respondents. Add attention checks, screen for your buyer and confirm surprising results another way.
What questions should a validation survey ask?
Questions about recent past behaviour and current tools, such as how often a problem happened last month and what they pay for today. Avoid hypothetical questions about future purchases.
Can a survey replace customer interviews?
No. Interviews find problems and language you did not know to ask about. Surveys can only measure answers to the questions you already thought of.
Start where you are
Surveys count. Commitments decide.
Sources
- Pew Research Center, Online opt-in polls can produce misleading results, especially for young people and Hispanic adults (March 2024)
- Pew Research Center, Understanding the margin of error in election polls (2016)
- Sachin Rekhi, A Primer on Talking to Customers From The Mom Test
This guide does not rank or endorse specific survey products. Check each tool's current pricing and data-handling terms before using it.