The Quantifiable "Truth"

Unlike qualitative research, quantitative research works under the assumption that there’s one or a small range of true population values for what you're studying. For example, if you were studying how people felt about the latest redesign, there's an objective number or percentage of people that truly dislike the redesign. When taking a quantitative approach, your job is to get as close to that true population value as possible.

To figure out what that true value is, you use objectivism, positivism, and deductive reasoning. Let’s revisit the same research philosophy diagram from this Topic but fill it in for quantitative research.

Quantitative research is fueled by realism. This research philosophy believes that the real world, how people interact with each other, or the true nature of their experiences is all based on objective facts outside of what people can sense and interpret.

This is a strange idea so let’s approach it differently: imagine trees that are growing deep in a dark forest that you've never heard of. Whether you knew this (or set out to confirm it) or not, those trees would still exist. That's what realism suggests – the "truth" is out there, waiting to be discovered. And everyone studying something quantitatively, with an appropriate study design, should come to the same or similar findings.

Realism suggests the "truth" is out there, waiting to be discovered.

In some sense, your bias or perspective as a researcher can make it harder to collect objective data. You might subtly influence someone's response. How do you collect enough quantitative data without your unnecessary influence? That's when a measurement instrument can be useful.

‍

Quantitative Instruments

In practice, quantitative research can collect lots of data from lots of people. For any single participant, you might ask dozens of questions (like in a survey). Each response will be recorded as quantitative data. If you asked 100 respondents ten questions each, very quickly you get a treasure trove of data, waiting to be analyzed. But how do you collect all of that data?

While you could sit down and administer a survey to one thousand people, the smarter move is to create something to collect that data for you. This is a measurement instrument. Examples of instruments include survey tools, A/B testing platforms, unmoderated testing software, or tree testing functionality.

The diagram above shows how instruments work in quantitative research. You create or use an instrument, designed to collect specific quantitative data. Every participant or respondent (like in a survey) interacts with this instrument, providing their data. You aggregate and combine all of the data to form sample statistics. You then use your statistics to estimate or infer the population parameter (aka true population value(s)). But if designed poorly, your instrument will be a biased estimator of the true population value, either over- or underestimating it.

In an ideal world, your instrument perfectly measures the population parameter every time for every person who interacts with it. In practice, your instrument (if created and validated appropriately) is reasonably good at measuring and estimating the true population value.

There’s always some amount of “noise” in your instrument. But when you have hundreds – if not hundreds of thousands – of people being measured by the instrument, this “noise” gets spread out. More measurements in total or more data per participant make it easier to notice the “signal”, the true population value you’re studying.

‍

Variability: Everyone is Different

In quantitative research, you often design a study to study or measure changes in variables. A variable is any measurable quality that can change or vary across time or populations. It’s a broad definition because it can change with every quantitative study you run. For your quantitative studies, you could study variables like where someone uses the product, how often they use it, or how they feel about the product itself.

Examples of Variables in Quantitative Research

• Location (ex: Australia, Argentina, Antarctica, Amsterdam, etc.)
• Product usage (ex: time-spent-per-month or median session duration)
• Product sentiment (ex: positive or negative app ratings)

Variables can change, meaning there is variability. Variability is essential concept to understand because much of quantitative research is built around a simple, unchanging idea: everyone is different. Everyone, no matter the demographics, segment, job title or any characteristic, is different. These difference can be big or small in everyday life.

You can think about the variability of people in two, interrelated ways: variability across people and variability within a single person. Let’s break this down further by thinking about ice cream, apples, and toothaches.

Imagine you’re interested in knowing how enjoyable the local ice cream shop is. You put up a paper survey (aka your instrument) at the store and ask customers to fill out how they feel eating ice cream today. Quantitative research is best done with hundreds – if not hundreds of thousands – of people, but let’s focus on one customer in this example: an apple.

Imagine this apple buys ice cream every day and typically enjoys it. However, the day you put up your survey, the apple had a toothache. The apple’s tooth was irritated by the cold ice cream. The apple selects “very unenjoyable”, instead of “very enjoyable” as the apple might’ve typically selected. If you used this data, you would might incorrectly that apple customers find the ice cream shop unenjoyable.

Now zoom out and think about every customer. There’ll be some customers that are having better or worse days that’ll influence their attitudes on the survey. You have no idea what every customer’s real or true attitude is towards the ice cream shop or how much those attitudes can vary or change over time. You’re forced to use the data you collected to estimate everyone’s true attitude about this ice cream shop.

Variability is a real concern when designing quantitative studies. It’s hard to know how skewed or extreme any quantitative data you collect is when compared to the true population value. Large samples, replication studies, and repeated measurements are all quantitative strategies to address variability. Large sample sizes can manage this variability while replicated or repeated measurements can help capture and describe that variability. Check out this presentation for more.

‍

Accuracy & Precision

You might use the words “accuracy” and “precision” in your everyday life. Within the realm of quantitative research, accuracy and precision have very clear definitions. Accuracy is about how close your estimated value(s) is to the true population value. The closer the two values are, the more accurate your instrument is.

Sadly, you never really know or confirm the true population value (see “The Estimation Paradox” in Topic 1 in this Handbook), so it can be challenging to know how accurate your estimate is (strategies around this include using census data to check estimates or triangulating your estimate with other methods).

Precision is how clustered or grouped your measurements are. If you used an instrument multiple times, how close together or similar are the measurements you get each time? You want all of the values to be close together, giving you more confidence that your instrument is at least working consistently.

Accuracy and precision go hand in hand in your quantitative studies. To make these ideas easier to understand, check out these diagrams to cement these ideas. You want your estimates to be as accurate and as precise as possible. What gets in the way is error and bias.

‍

Error & Bias

Whenever you measure something, there'll be uncertainty with your measurement. These are subtle differences between your sample statistics and the true population value. You can split this uncertainty into two groups: random error and bias.

Random errors are uncertainties in your estimates from natural, random chance. There are slight variations each time you use the instrument. Think of random error as a ring of uncertainty around the true population parameter.

No matter how or what you're measuring, there'll always be some amount of unremovable random error. But if you use the instrument many times (or have a larger sample size), that random error gets spread out, becoming less of a concern for you. One example of random error is sampling error (you can read more about sampling error here).

Bias – or systematic error – is a consistent variation between your estimated and true population values. You can think of this as gunk or dirt on your instrument. The more dirt on your instrument, the more every measurement will be distorted and skewed. Unlike random error, bias gets worse with larger samples. You’re collecting more and more biased (”dirty”) data. With enough bias, you might never get close to the true population value you set out to study.

There are many ways that bias can creep into your study design. A simple search for "research biases" will give you more than enough reading for a lifetime. You can start your reading here with this article.

The goal is to lower bias and control error as much as possible. This means you have to make your instrument logically, which means understanding the concepts of validity and reliability.

‍

Validity & Reliability

While accuracy, precision error, and bias have to do with the quality of the instrument, validity and reliability have to do with how you made or used that instrument.

Let's start with reliability. Reliability is about one thing: consistency. If you measure the same thing the same way over and over, how many times do you get the same or similar result? If your results are consistent across the multiple trials and across time, then you can say that your instrument is at least reliable. If your instrument is reliable, you’ll get precise or clustered datapoints.

Valid instruments lead to both accurate and precise estimates for the true population value you’re estimating.

While reliability is important, it’s not enough in your quantitative research. Your real goal is always striving for validity. Validity asks, “Is your instrument capable of estimating the true population parameter at all?” If you want to understand how satisfied people are with your experience, asking them about where they want to vacation wouldn’t be a valid approach. Valid instruments lead to both accurate and precise estimates for the true population value you’re estimating.

Some of the common and helpful types of validity are listed below for quick reference.

When you're creating your measurement instrument, you want to make it as valid and reliable as you can. But note that something can't be valid and unreliable; something must first be reliable to be valid. That's why there are a ton of reliably inaccurate measurement instruments out there, but not a lot of valid ones. You can read more about validating a survey instrument here.

Estimating the true population value is a tricky business. Think about how good and reliable your approach is. Can you identify sources of biases and address them? Can you run a quantitative study more than once to see how reliable and precise your results are?

It's not as simple as using a quantitative research tool and naively accepting the results. It's a process of questioning and making iterative changes to how your team thinks about and practices quantitative research. For more on validity and reliability, check out this paper.

Making a quantitative instrument can be challenging. To speed things up, you want to have a clear idea of what exactly you're trying to measure. This is typically formalized into one or several research hypotheses.