Numbers & the Human Brain

In the first Topic for this handbook, you were asked to choose between a sample size of 10 or 1,000 participants? For many stakeholders, research is seen as valuable, credible, or trustworthy only when it's based on the data of thousands of people.

With quantitative research, the interest or demand for large sample sizes might force you into bad study design choices, making you do whatever you can to fulfill the expectation. But there’s a problem: people — like you and your stakeholders — can’t make sense of even moderately large numbers easily.

The human brain struggles to understand “large” numbers.

For a simple illustration, how many apples are in each group in the illustration below?

One, three, and four apples. You'd be right if you felt it was a fairly easy task. You don't need to manually count for very small, observable numbers — you can recognize how many items or units are present.

But you won’t see such small numbers in your quantitative research. After all, you wouldn’t report “quantitative findings” after talking to two people. Quantitative data analysis is best done on lots and lots of data.

Let's update the illustration to better reflect larger quantitative samples. Now count how many bananas are in each group below.

There are 9, 18, and 30 bananas from left to right. Now imagine each banana as a participant in one of your quantitative studies. In practice, 30 participants isn’t a particularly large number. But as shown in the illustration, you’d have to resort to manually counting each banana.

Now zoom out to your own quantitative research. You’re probably trying to shoot for the largest sample size you can. You’ve might have even had 300, 1,000, or even 5,000+ respondents in your last survey. But the size of the sample doesn't fully capture the value of learning from so many people.

For example, a sample size of 1,000 respondents is only 100 more than 900 respondents. But in practice, the amount of effort and resources needed to get 100 more people to respond isn’t represented at all in the final sample size. In some sense, your stakeholders hear about your sample size while you experience it.

And a modest sample size increase from one study to the next can be valuable. For example, take two samples of 125 and 250 participants, respectively. If you were to visualize sample sizes (as shown above), you’d notice the larger sample has 50% more people, which translates to 50% more data.

With 125 more participants, you’ve greatly increased the opportunity to learn new things, confirm assumptions, and recognize blind spots in your or your stakeholder’s decision-making. The point is simple: every sample size represents people. When you focus on only the size of a sample, you miss out how the complexity and nuance your participants have to offer.

Your goal isn’t to study the most amount of people but be able to explain why the sample size you did learn from is valuable and meaningful. You have to define and defend a practical sample size.

‍

Defining practical sample sizes

A practical sample size is the minimum number of participants your stakeholders find valuable and for you to sustainable study given your limited resources. If your stakeholders are expecting large, complex samples, you need to either (a) be able to fulfill that request or (b) align on more reasonable but still effective sample size. You can jump back to Topic 2 in this handbook for ways to challenge or debunk if your stakeholders demand a large sample size for every study.

Ask yourself: “How does a sample of this size affect your future participant recruitment?”

Practical sample sizes are also sustainable. If you struggle with recruitment and only have a handful of people you can contact, you’ll run out of potential participants. Keep the future in mind no matter how urgent and important a study might feel. If you use all of your recruitment resources in one study, your future study recruitment will suffer.

Be honest about your own recruitment and what you’re able to reliably do. Always ask yourself how the sample size decisions you make in one study affect or change how many people you can learn from in later studies.

The rest of this Handbook is devoted to determining and justifying practical sample sizes. Keep in mind the numbers listed below are suggestions, not a definitive sample size. You'll have to take these numbers and contextualize them in your research environment. Get uncomfortable and start to figure out practical sample sizes for your work. Your numbers might look different if you have a research panel or an internally managed sampling frame. You can read more about creating research panels here.

With that said, the sample sizes listed below are there to offer you a good place to start figuring out what practical sample sizes you should consider. The numbers ahead are cost-efficient, explainable, can be scaled up or down as needed, and possibly most important, sustainable. Use these reasons to explain why your research is more valuable than the number of participants you learned from.

Let’s start with practical sample sizes for qualitative research.

‍

Practical Qualitative Sample Sizes

Qualitative research in a short, fast-paced timeline forces you to work with smaller samples. You can use these two ways to select and justify a smaller sample size: saturation and attention (see this Topic for more).

For a a quick primer, saturation is related to how novel or unique is the data you’re collecting relative to what you’ve already seen, observed, heard, or known. One common way to recognize saturation is when you start thinking to yourself, "I've heard or seen this before."

Attention, on the other hand, has to do with your focus and energy. You only have a limited amount of attention when you're conducting in-depth qualitative research. To practice sustainable and practical qualitative research, you need to reach saturation without completely draining your attention.

To practice sustainable and practical qualitative research, you need to reach saturation without completely draining your attention.

Below are two practical approaches you can take to determine a practical qualitative sample size to reach saturation without draining your attention.

First, if your qualitative research question is studying subtle, abstract, complex, and interconnected variables, you’ll need more participants to confidently reach saturation.

For example, you’d need different sample sizes to reach saturation for these two qualitative research questions: “What do people think about the latest redesign?” vs. “How does gender identity affect the social relationships that young non-binary children (ages 9-12) create and sustain with their peers?” If you have lots of variables or segments to study qualitatively, assume that you’ll need a larger sample size to reach saturation.

Your ability to generate meaningful and credible themes is greater when dealing with simple research questions than complex (see this Topic for more on qualitative themes). With simpler qualitative research questions, smaller sample size can mean saturation can be reached during data collection and analysis.

If your stakeholders want to understand something complex and qualitative, then aim for sample sizes on the larger side. If not, a handful of participants should work just fine.

The other approach is to recruit more participants until saturation is reached. You can do this by using breakpoints in your study design. This is when you take your desired or expected sample size and divide it in half. You collect data from the first group of participants and see if you can reach saturation. If saturation isn’t reached, you would inform your team and continue to study the remaining half and possibly add more participants to your study.

If you reach saturation with the first half of participants, you could run member checks with the remaining half (see guide for more) to see if your early findings reflect their experiences.

Guide 13: Member-checking Qualitative Findings

You can take both of these strategies and use them alongside the sample size suggestions below. Remember to explain why you’re recommending a specific sample size alongside the number.

Now, let's shift focus to quantitative research sample sizes.

‍

Practical Quantitative Research Sample Sizes

Unlike qualitative sample sizes, determining a practical quantitative sample size is more rigid. Quantitative research (and its approaches, techniques, analysis, and interpretation) is based on standardized, mathematical ideas to determine and justify an appropriate sample size.

Similar to qualitative sample sizes, there isn’t one perfect sample size for all your quantitative research. While you can be selective with qualitative samples, there are many more factors that shape how large a particular quantitative sample size should be:

Factors that Affect Quantitative Sample Sizes

• Purpose of research (test hypothesis, estimate population parameter, etc.)
• Sampling technique (non-random sampling will need a larger sample)
• Margin-of-error / confidence interval (larger samples lead to smaller margins-of-error)
• Effect sizes & power (smaller, non-obvious relationships will need a larger sample size)
• Nonresponse bias & coverage bias (over or under sample for sample balancing, read more in the next Topic)
• Number of groups being studied, compared, or measured (more groups, clusters, or strata will need a larger sample size)
• Type of data collected (categorical data will need a larger sample size than numerical data, read more here)
• Requirements for any statistical methods you plan to use (such as at least five observations per cell for the chi-squared test)
• The resources you have (such as time, budget, the size of your sampling frame, etc.)

Given these factors, the table below was created and modified using the Yamane sample size formula (you can learn more here). Without getting into the math, below are a range of sample sizes based on the size of your population in your quantitative study. Next to each minimum sample size is a range of numbers.

As mentioned in Topic 2, you’ll have to contact more than the minimum sample size listed, factoring in unpredictable and unremovable nonresponse bias. The number range is generated assuming a 10% response rate (meaning 90% of people contacted don’t respond or participate in your study).

You can see similar numbers when you use sample size calculators online. But make sure to read the fine print before using them. Most online sample size calculators assume that you're using random (or even simple random) sampling. If you're not using random sampling, the sample size calculator is giving you a misleading sense of confidence.

If you use an online sample size calculator, make sure you know if it assumes that you used random sampling or not.

This table assumes your population is uniform or not very diverse. In practice, you want a larger sample size than listed to capture the diversity you know exists in your population. The table is also best if you use it for running quantitative research towards a strategic goal, problem, or prediction.

If you’re focused on something tactical (aka product-focused like with card sorting or benchmark testing research), your goal is to get at least 30 participants per segment.

You can read about the central limit theorem here for why 30 participants is an important number in quantitative research.

If you can, always try to get more than the minimum number listed above. Your resulting data will be a bit more stable, meaning the shape of the data will be a bit cleaner, and patterns will be easier to notice.

But what if you don’t meet the requirements for random sampling when conducting quantitative research? In that case, you’ll need some practical sample sizes for non-random sampling.

Sizes for Non-Random Sampling in Quant Research

If you’re using non-random sampling, your decision-making process is equally complex. You want to shoot for as large of a representative sample as possible while trying to lower nonresponse bias without exhausting your sampling frame. The table contains some base sample sizes you can consider to make things easier.

It’s hard to provide a sample size range when your population is diverse. Every situation is different, and many factors affect your non-random sample size. You can use the chart below to recognize if you’ll need a larger sample size than the numbers listed above.

If you find your study falls into the “More” boxes, it’s a strong signal you’ll need to take additional steps to ensure you’re collecting enough data for your stakeholders to trust and find value in your quantitative findings. Try to gather extra recruitment resources (for study incentives or marketing efforts) and make a conscious effort to monitor and boost response rates however you can.

Without understanding sampling and statistics, defending quantitative sample sizes can be hard. No online sample size calculator anywhere will contextualize sample sizes to your specific situation. If you can’t establish acceptable quantitative sample sizes for your workplace, you have ultimately three options:

• Study the quantitative research questions with a qualitative approach
• Contact a trained statistician for help
• Partner up with a data scientist colleague (or other quantitative-focused peers) to address sample size issues together

It's not the ideal answer but having a reliable and valid of conducting quantitative research is one of the hallmarks of a research-mature culture. It's not as simple as skimming a few charts in a book or online, finding a number, and telling yourself you're doing great quantitative research. You'll have to invest time (or form relationships with your quantitatively/statistically proficient partners) to grow your quantitative sampling capabilities.

The idea of representativeness has popped up in this chapter and in the last Handbook. Independent of how small or large your quantitative sample is, you need to make sure your sample is representative of the population or segment of interest. But how do you know if you have a representative sample or not?