Numbers, Units, Scale, and Context

Quantitative research uses numbers to describe people's expectations, attitudes, and behaviors towards a topic, product, or experience. To do describe complex relationships or phenomenon, you use numbers. But for many researchers and non-researchers alike, all numbers seem like perfect and powerful. Once you understand how to interpret a number, you can recognize the amount of information it represents and how valuable or useful that information is.

Every number you see has at least properties: the quantity, the unit, the scale, and the context. You need all four to really understand what a number is trying to describe or represent. Let’s start with the quantity and the unit as they usually exist as an observable pair. The quantity is the explicit number of “things” you have, while the unit describes what those “things” are.

A number without units is meaningless.

Without units, any number is essentially meaningless to your brain. Every quantity needs a unit. The unit is what make numbers real because it tells you what what the number is representing or quantifying. 5 participants isn’t the same as 5 visits to the hospital, even if the number is the same. Change the unit again to 5 km, 5 uncompleted tasks, or 5 days of holiday, the information you gain is tangible and understandable.

But a number and a unit alone aren’t enough. You need to expand your focus and understand what surrounds it: its scale and context.

What scale is a number being measured on? Is it measuring categories or not? Can your number be a fraction, decimal, or percentage? To make things simple, think of any number you see existing on either a discrete or continuous scale.

A discrete scale means only certain, whole numbers can exist on it. For example, the number of blueberries on a city bus would exist on a discrete scale. You can’t have 9-and-3/4th blueberries. Other examples include things like the number of people that like the color blue or the number of tickets sold to a concert.

But if you were describing how long each blueberry spent on the bus, you’d need a continuous scale. If average time-spent-on-bus was 10.1 minutes, it very easily could’ve been 10.12, 13.99, or 15 minutes exactly. The average time could be any number and it would still represent something real.

Discrete data typically involves using numbers to represent categories. And if a number doesn’t represent a category, then it’s likely continuous. What’s important to know is that these scales inform how you should analyze, interpret, and report them. Discrete data is treated as categorical data, while continuous data is treated as numerical data. Check out Topic 4 in this Handbook to learn more.

A number’s context tells you "how" and “why” it exists.

The last property covered here is the context the number exists in.  Why does this number matter? What’s the history of this number? How credible or reliable is this number? How did someone arrive at this number? What else does someone have to know to fully understand, appreciate, or challenge this number?

For example, “90 participants” has a quantity, a unit, and is an example of discrete data. But none of that information is valuable. What’s interesting about this particular group of 90 participants? When you add in context and read: “90 participants were able to complete the unmoderated usability task in under thirty seconds”, you can get appreciate and find value in this number.

If you also think about history, you might expand the context into something like: “90 participants were able to complete the unmoderated usability task in under 45 seconds, a 40% increase from the last test.”

Add in how the number was generated, “From a random sample of 100 participants using version 2 of the online design prototype” to the beginning of the phrase and you get a complete and robust understanding of what’s happening: the changes were from version 1 to version 2 show remarkable results.

Whenever you encounter a number, look to fully understand the context it exists in. It’ll give you important information to understand the idea the number is representing. And for a closer look at numbers as an idea, check out this book.

Numbers can seem more objective than qualitative data, but always ask yourself: “What quality or idea is this number representing?”

Keep in mind, your stakeholders might interpret numbers at face value. 90 participants can seen as just 90 participants. As you develop your own quantitative research skills, try to improve the quantitative literacy of your stakeholders and those around. Challenge incorrect or incomplete ideas about quantitative research, the use of numbers, or the idea that any number is a better, more objective representation of truth than no number at all.

In quantitative research, you use numbers to estimate relationships and values all around you. One common reason to conduct quantitative research is for estimating the true population value.

‍

The Estimation Paradox

Quantitative research can help you estimate different aspects of the relationship people have with a product. To simplify things, you can think of estimates falling to two buckets: estimates about the population and estimates about relationships between variables. Let’s focus on population estimates here as estimates about variables (such as through correlation or regression analysis) can require explaining many abstract ideas. You can read more here about estimates and relationships between variables here.

You care about population estimates when you consider how well your sample findings represent what’s true about everyone in your population-of-interest.

Whenever you set out to estimate something about the population based on a sample, you implicitly assume that there’s a true population value. The true population value is the value or answer the accurately describes how the entire population thinks, feels, or behaves.

You use your sample as a way to estimate something about the population. For example, if you randomly sampled 500 respondents and found that 80% disliked the latest smartwatch redesign. The true population value might be 83%, meaning your sample was a pretty good estimate.

In practice, there’s rarely ever one value but likely several values but the focus here is on a single true population value to make this idea easier to understand.

But there’s a problem here: in practice, you never know the true population value. There’ll always be a difference between your sample findings and what’s true about at the population level. This poses a real issue for you when designing and running quantitative studies: how do you know if your quantitative findings are accurate?

You can only estimate the true population value.

This is the estimation paradox: your sample can only help you estimate what’s true for a population, but you never know if that estimate is accurate. To illustrate this better, let’s step away from experience research and look at a simpler example: the speed of your car.

Let’s pretend your car doesn’t accurately display its current speed. If you’re driving 60 km/h, the car’s speedometer might read 50 km/h instead. If there was an objective third party outside of the car measuring your speed with a radar speed gun, you could see how inaccurate your speedometer is.

But in practice, you don’t have that radar speed gun. You don’t have an objective third-party or measurement to compare results with; you only have what the speedometer displays and your interpretation of what that speed means. If you did have an objective measurement instrument for knowing what the true answer is, you’d have no need to design and run a quantitative study. After all, you’d either know the answer before the study or choose instead to use this objective instrument.

Does that mean you shouldn’t run any quantitative research? Not exactly. Unless you take care to design a quantitative study in a valid and reliable manner, your sample might always over or underestimate the true population value. When done well, your quantitative findings can serve as the best available estimate for the true population value.

Replication (the act of repeating an entire study as designed to come to similar or the same conclusions) can help you generate more reliable estimates. However, replication is rarely done in experience research given product and stakeholder pressures. But replication also doesn’t guarantee that accuracy, only precision.

‍

Point vs. Interval Estimates

When you estimate, you can either arrive at a single value or a range of values. This is known as a point or interval estimate.

A point estimate is a single best guess for the true population value. It's a sample statistic or value (from a random & representative sample) that’s estimating the true population value(s).

On the other hand, an interval estimate is a range of best guesses. You take your point estimate and essentially add a "buffer" on either side. This buffer is known as the margin-of-error. This larger range stands a higher chance of actually containing the true population value(s) than a point estimate.

Interval estimates are more commonly known as confidence intervals (see this PDF for more help). Interval estimates are the most common way of estimating the true population value(s) across dozens of quantitative disciplines (such as data science or engineering).

Using numbers to estimate the true population value(s) is a common goal in quantitative research. But there are other goals or quantitative activities you can perform to estimate and understand different parts of an experience.

‍

Quantitative Activities

When people hear of quantitative research, they immediately jump to surveys and A/B tests. Those are examples of quantitative research methods. A quantitative activity is what you're setting out to do with those research methods. It's similar to the purpose or reason for the quantitative research in the first place.

Below are three of the most common activities when designing a quantitative study.

These activities can be used across a range of situations. It's up to you to figure out which activity is most relevant for you and your team. Review the table above or work with your stakeholders to understand what activity is relevant. Regardless of the reason for you to conduct quantitative research, you need to be aware of and manage its strengths and weaknesses.

‍

Quantitative Strengths & Weaknesses

Think of quantitative data as looking at a footprint in sand or mud. You have evidence that someone stepped here, but you have no idea why. Were they running? Were they late? Where are they going for a stroll? From the footprint alone, it'll be hard to know why. Other strengths and weaknesses of quantitative research are listed below.

In the last chapter, the idea of constructivism and interpretation were covered. Both are useful ideas to understand how qualitative research studies the "truth." How does quantitative research do the same?