The Classical Survey Design Process

As mentioned in the introduction, it’s the process of designing a survey that's important. The flowchart below shows the major steps in the survey design process. It might look complex or scary, but valid surveys require effort. Let’s walk through the steps of the classical survey design process.

Some of the steps are pretty straightforward, while others require more explanation. The more complex steps are discussed in-depth below, while all of the steps are discussed further in the online guide.

Guide 09: Survey Design Template

‍

Set Survey Goals (or write Hypotheses)

You first have to know what you’re trying to learn. Setting goals is possibly the most important thing you can do when designing a survey. Without clear goals, your survey will always collect meaningless data.

Without clear goals, your survey will always collect meaningless data.

Surveys are a quantitative method, so you can start your survey design process start by writing down what you expect to be true (jump to this Topic for more on deductive reasoning). Convert these into testable and measurable research hypotheses.

Guide 06: A Research Hypothesis Checklist

With hypotheses, you can write not only fewer questions but increase the utility of the data you collect. You can’t add or edit questions after you launch your survey, so you must be confident you’re asking the most critical questions. Below is an example of how hypotheses can influence your survey design.

Survey Research Hypothesis Example

• Let’s pretend that you’re interested in seeing if there was a difference in preference between iOS and Android owners for the latest app redesign. You might write the following hypothesis: Hypothesis: iOS owners rate app redesign more favorable than Android owners.‍
• With this one hypothesis, you know that you have to ask at least two survey questions: a question to understand what type of smartphone people have and another to see how much the app redesign was preferred.
• Each of the questions will collect categorical data (because smartphone ownership and redesign preference are examples of categories), meaning you can also figure out statistical test you would use to test the hypothesis (such as the chi-squared test of association to see if there’s a relationship between smartphone ownership and redesign preference).

Guide 12: Using the Chi-Squared Test

You don’t have to write research hypotheses for a survey. However, with short timelines, knowing that certain hypotheses will be tested using specific statistical tests can help you focus your analysis and speed up your data analysis phase.

Designing your survey becomes cluttered and unhelpful without clear goals or things to validate, test, or disprove. Take the time to speak with your stakeholders about what they believe to be true, craft hypotheses to test them, and then design your survey. Without specification, you’ll spend more time designing and collecting survey data than getting value out of that data.

If your goal is to measure a specific construct (like customer satisfaction), you’ll need to either check the literature for a survey instrument you can use or go through the process of designing and validating your survey. The diagram below shows you the link between your goals for a specific construct (jump to this Topic for more on operationalizing a construct).

Note that you’ll still need to validate your survey, no matter how you operationalize a construct. Recall that valid means the survey is measuring what it was designed to. Whenever you can, try to use a validated survey. Here’s also an article that walks you through the rigorous process of validating a survey.

‍

Conduct a Literature Review

Before writing any survey questions, do yourself the favor of conducting a literature review. You’d be surprised how much data can be found inside your business and outside online. It’s also entirely possible to conduct a literature review and find answers to your survey research questions without actually using a survey.

Other things to look for when conducting a literature review to make your survey are listed below:

Topics and Ideas for Survey Design Literature Reviews

• What internal or product analytics data can help you narrow down who to survey?
• Are there any specific considerations, requirements, or challenges your target survey population has?
• Are there any specific or unique language or terms used by your target survey population to help you write better survey questions?
• Are there any tested and validated survey instruments that you can use or modify?
• How have past surveys for your target survey population performed? Do you need to add incentives or budget more time to get a meaningful survey sample size?

You want to use a validated and tested survey instrument whenever you can instead of making your own. Not only will this be faster, but you’ll collect more valid credible data. Check the literature before writing a single question to see what you can find. The sad part is that validated survey instruments for experience research are (a) hard to find and (b) hard to find for free or at an affordable price. What can you do?

‍

Run Qualitative Research First

The best option is to first run qualitative research around the questions or hypotheses you or your stakeholders have. You want to see what you can learn to help write a shorter, more focused survey. Some interviews with participants from your target population may completely eliminate the need or want for a survey.

Whenever possible, run a focused qualitative study addressing some of the survey goals and questions as it’s a tried-and-true process. You can use your interviews' qualitative themes to help narrow down what survey questions to draft.

‍

Draft Survey Collaboratively

Even if you’re designing the survey only for your use, try collaborating with others. Try to work with senior coworkers or those with expertise on your survey topic. They can point out what topics to ask about, what data is already present, and offer tips for questions or responses to avoid or include.

First, draft questions by volume and slowly eliminate the unhelpful or unnecessary questions. For every survey question, ask yourself: “Will the collected data be useful when addressing your study goals and/or testing hypotheses?” If the answer is no, remove or edit the questions. Ultimately, ending with a 8-12 question survey is ideal because it’s short enough to be tested but long enough to collect meaningful data.

‍

Test/Edit the Survey

This is when you take your draft survey, have respondents from your target segment take it in front of you, and see where the survey is confusing, ambiguous, or challenging. This is known as cognitive testing.

Guide 10: Cognitive Testing your Survey

Why should you cognitive test every survey you design? You need to ensure that respondents interpret and answer the questions as you intended, not what they assumed. Without understanding the gaps, your survey data might appear perfectly fine when your data is biased or distorted in reality.

Build at least three days in your study timeline to test your survey with five respondents from the target population or segment. That way, you’ll have time to make the necessary improvements.

Cognitive testing aims to make changes or edits to your draft survey. There’ll always be some big and small ways to improve any survey’s clarity, focus, and structure. If you’re working on a tracking survey, then you’d want at least two or even three rounds of cognitive testing (with changes between rounds).

And even if you’re working with a validated survey instrument, you’ll still need to cognitive test it. It’s not a guarantee that your specific group of respondents will interpret it as intended. If you test and realize the instrument needs to be changed, you’ll have to make your own survey instrument from scratch. Even small changes can threaten the validity of a validated survey instrument.

If you have access to statistical software, you might also want to test the survey’s reliability. If you’re measuring a construct, then you’ll want to go through the full process outlined in the link below. Note that you’ll require access to statistical software or tools (such as R, Python, or SPSS) to validate a survey appropriately. This article does a great job walking you through the basics of validating a survey.

‍

Pilot Test (at least once)

The last step of the survey design process is to pilot-test everything. This means launching the survey to a handful of respondents, mimicking the real survey launch. The goal is to figure out how the survey is being distributed and how the data is coming back.

If you’re using a survey tool, you want to see if survey links are broken, if your survey logic is faulty, or if the data that comes back is still useful when addressing study goals. If needed, you’d edit and make any final changes before launching the survey. You can read more about pilot testing here.

‍

The Streamlined Survey Design Process

The survey design process is highly structured because it has to be. Without intention, planning, and iteration, your surveys appear effective. If you’re short on time, it’s strongly recommended that you consider another method. But if you have some time, you can follow the shorter survey design process below. It’s not perfect, but it’ll increase the validity and quality of the survey data you collect.

Streamlined Survey Design Process

1. Set survey goals and/or write a research hypothesis
2. Check literature to find a validated survey or data to help write fewer, but more meaningful questions
3. Draft questions and responses (first for volume, then cut out irrelevant or redundant questions)
4. Cognitive test & pilot test with five respondents from the target segment
5. Make changes & launch (cognitive test again if possible before launch)

When you use a survey, you’re using it to collect data for you. The downside is that everyone might interpret the same question differently. Worse, they might rush through the survey just to finish it. When designing a survey, you want to understand how people answer any survey question to avoid these issues.