Common Study Design Elements

Every study you design will be different. However, there are some common elements that’ll you need to include in every study design.

The report you create and share is one example of a research artifact or any observable, tangible output that comes from your research. Examples include your final interview guide, study compensation structure, photos from different study sites, notes in your analysis journal, or early versions of your survey questions.

Guide 03: A practical research Plan

Guide 11: Using an Analysis Journal

While the elements might stay fairly consistent, the odds that every element works as designed won’t. As you conduct more research, you’ll quickly discover an unfortunate but important fact: you have far less control than you think.

‍

To Design is to Risk

No matter where you conduct research, designing a study will always have some unremovable amount of risk. In this context, risk means not having a lot control over the execution and progression of your study. Even in mature research cultures, you can’t make guarantees for what’ll happen after you design and start your study.

You never have as much control in a research study as you might prefer or need.

For example, you can’t guarantee that enough informative people will respond to your survey by the end of the week. You can’t guarantee that you’ll see a specific behavior or interaction during a contextual inquiry. In some situations, you can’t even guarantee your stakeholders will care about your findings even though they were the same people who asked for the research.

Sadly, the less control you have over how specific phases of your study pan out, the more likely you’ll have to pivot or make immediate changes after starting a study. That’s not a good situation because your prep work, planned analysis, and reporting commitments have to be reworked in real-time. Risks make it even harder for you to quickly deliver findings and insights back to your stakeholders.

Every time you design a study, you accept some level of risk. But what exactly could go wrong when designing a study? What are the risks you’ll encounter? Let’s revisit the study design elements from above, but this time, let’s look at the possible ways each element can be risky, unpredictable, or uncontrollable.

The size of the table above might send you into a panic. It seems like you have zero control over how your study will work. But remember that not all of the risks will apply in every study you design. To overcome these potential risks, you have to first predict what’ll go wrong in your study and then make decisions to lower or manage those risks.

‍

First, Predict & Then Decide

If you design your qualitative study to use an offline prototype inside a participant’s home, then the risk of poor/unreliable internet connectivity isn’t a relevant scenario you need to consciously plan for. But you might predict that your laptop might lose battery during the session, so you decide to bring a laptop charger and to make sure to sit near an electrical outlet. Predictions essentially lead to smarter study decisions.

A prediction isn’t only about being aware of any relevant risks but recognizing if and how much those risks will affect your immediate study design. A decision, then, is something you do lower or manage those risks. In some situations, you might choose to accept a risk and do nothing at all. For example, you might decide to study a vague or uninteresting question as a way to build relationships with senior or research-resistant stakeholders. This isn't necessarily a bad idea for a handful of studies in a research immature culture.

First, predict the potential risks in your study design. Then decide how to manage, lower, or remove those risks.

But other times, you’ll need to make an explicit decision on how to design your study to minimize as many risks as possible. If you predict that your study will struggle with recruitment, you’ll need to consciously align and decide on a more practical sample size or ask stakeholders for help. If you predict that you won’t cover all of the necessary topics during a 45-minute interview, you need to decide what topics get removed or make a plan to learn about additional topics in another study.

Based on the design of your study, you’ll need to make different predictions and decisions. What predictions might you implicitly make when you design a qualitative or quantitative study?

‍

Qualitative Study Predictions

While you read more about qualitative research in the next handbook, below are some common predictions you might make when designing a qualitative study. Keep in mind that you might be implicitly or unconsciously making these predictions. If you’re unaware, you’ll likely find yourself making immediate changes or backup plans to keep your study on track. You can also read more about any of these ideas throughout Handbook 2 in Collection 3.

Qualitative Study Predictions

• Your qualitative study design will produce meaningful and helpful answers to your qualitative research questions
• You can reach some level of data and thematic saturation after x number of participants
• You can be sensitive enough to recognize nuance and subtlety in your participants’ behavior and responses
• You can be present and minimize the influence on participants
• You have a meaningful time to build rapport with each participant to lower or manage participant reactivity
• You can analyze all collected data with focus, speed, and quality
• You can collect data about important topics without rushing or forcing a biasing a participant’s responses or behavior
• You can conduct yourself in a credible and dependable manner
• You can arrive at qualitative findings that are confirmable and transferable

If you’re designing a quantitative study, you’re likely making a different but equally complex set of predictions for your study.

‍

Quantitative Study Predictions

Below are predictions you might be implicitly making whenever you design a quantitative study. As mentioned before, not all of the predictions will apply to every quantitative study you design. You can read more about these ideas in Handbook 3 in Collection 3.

Quantitative Study Design Predictions

• Your qualitative study design will produce meaningful and helpful answers to your qualitative research questions
• You can validate that how you’ve designed a quantitative study is studying what you set out to
• You can collect enough data from noticing stable or consistent patterns
• You can produce an accurate, precise, and unbiased estimate of what’s true at the population level
• You can use random sampling and/or random assignment
• You can write research hypotheses that narrow your focus and make the interpretation of quantitative data easier
• You can collect the level of quantitative data you need (such as nominal, ordinal, interval, or ratio) to test research hypotheses or answer certain research questions
• You can structure, clean, manipulate, and visualize any collected quantitative data in a timely and efficient manner
• You can check or assess the representativeness of your sample relative to a known population

If any of the risks and/or predictions described affect a study you’re planning right now, take action to handle them. If you can set yourself up for success, you’ll find that every phase of a research study goes more effectively. That means making smarter, sustainable decisions. With time and experience, you'll realize which study design elements are manageable or modifiable and which ones you'll need to actively plan around.

‍

Smarter, Sustainable Design Decisions

You can make several decisions (listed as behaviors or actions in the table below) to take design smarter studies. But it’s up to you to make certain decisions or not. Your stakeholders might not recognize the potential risks or challenges of conducting any research, but you should. You have to decide what relevant risks to accept and address. Also, remember that not doing anything is in itself a decision. If you don’t reasonably define who the Most Informative Participant (MIP) is, you might find recruiting to be challenging and irrelevant. If you don't consider the complexity and time needed for proper qualitative data analysis, you'll run the risk of being overwhelmed and rushing through it.

There’ll be reasons and situations where you’ll need to break some of these suggested decisions. That’s fine because research is messy. But for every decision you make when designing your study, keep this question in the back of your mind: “How likely will this decision maximize learning for you and your stakeholders without exhausting you or your limited resources?” If your answer is “very unlikely,” have the respect for yourself, your stakeholders, and this craft to make changes before starting a study.

Always ask yourself: “How likely will this decision maximize stakeholder learning without exhausting you or your limited resources?”

All the elements above end up in your research plan. You don’t want to start planning your study without having a research plan and getting alignment from your team. Even worse than poor study design predictions is setting out to study the wrong thing.

Guide 02: Aligning on a study

Guide 03: A practical research plan

You make these predictions every time you start a study. Many of the elements are covered in other chapters, so let’s focus on one of the less risky predictions: the methods you use.