To Infinity & Beyond

There are an infinite number of research questions you could study. In fact, the things people don’t know far outweigh everything that is known. Interestingly, when you generate answers for a research question, newer, better questions get discovered. Answers create more questions and often several interconnecting questions at a time.

Below is a list of research questions. At first glance, these all appear to be meaningful questions to study.

Examples of Research Questions

• “Is the product usable across cultures and geographies?”
• “How does the product compare to competitions in terms of everyday usage?”
• “Why do people dislike the product?”
• “What technology products do new parents shop for before their baby is born?”
• “How do recent graduates try to find a job?”
• “Where do podcast listeners regularly consume podcasts and why?”
• “Why are people in this country sharing more often than other countries?”
• "Which design variant formers better?"
• “How has sentiment towards search changed over the last three updates?”
• “Is there a relationship between user behavior and positive app sentiment?”
• “Why do people abandon their online shopping cart when they have more than five products?”
• “Why did the homepage see a spike in returning visitors the last month?”

While they might appear meaningful, without goals, context, and alignment, these questions are more “interesting” than important to study. Many of your stakeholders might ask you study "interesting" questions. In fact, "interesting" research questions are almost too easy to create.

For example, would it be interesting to learn that most people that use your product like blue dinosaurs? Absolutely.

Would it affect your stakeholder's plans for the product? Absolutely not.

You could spend an entire year studying "interesting" questions without influencing the product or service. And if you're reading this, you don't have a full year to study any question at all. Studying "interesting" questions is a surefire way to eliminate your research impact and waste your limited resources.

‍

The Cost of Fruitless Research

Whether you work at a big tech company or a startup down the street, you have only a finite number of resources for conducting research. Resources include things like your study budget, the number of customers who've opted into research, or the amount and diversity of research tools/ software you have available.

You use these resources to recruit participants, pay for tools, collect vendor reports, and do other activities. While big tech companies have enough money to replenish these resources, most researchers work with what little they have. But time is the one resource that can never be replenished or added.

Let's pretend that running an entire research study always takes three weeks. In the first week, you make a research plan and recruit participants. In the second week, you create and use an interview guide to collect data. In the final week, you analyze the data and share a small report. If it always takes three weeks to run one study, how many studies could you run in one year?

If it takes you three weeks to run one study, you can only run a maximum of 17 studies per year.

Seventeen studies. Sure, you could cut down the time to run each research study, but you probably can't go any faster than you are now. You'll also get burnt out, jumping between different questions, topics, and participants. Cutting down the time to run a single study also requires you to have lots of resources (namely budget and access to participants), which you probably don't have. But isn't it better to spend some time doing some research than no research at all?

In simple terms, if you spend one of your limited opportunities studying a fruitless question, your study will fail. A fruitless research question will have little to no impact on the product or service you're trying to improve. One example is the "interesting" questions from before. Another example involves questions that don't produce any actionable learning.

The best way to recognize fruitless questions is when you don’t know what happens after your study ends. You complete your analysis and present the findings to your team. And then one of your stakeholders asks a deceivingly innocent question: "So what?"

If you can’t answer the “So what?” question, you’re probably studying something fruitless.

So what if you completed a quantitative study? Why should your stakeholders care if your findings are qualitative or if you had a large sample size? Why should they pay attention to your design recommendation? How does answering your research questions help build a better product or service today and in the future? If you don't have a clear, tangible answer to the "So what?" question, then that's the first sign you're pursuing a fruitless question.

Fruitless questions also don't help your team make good decisions or avoid bad ones. And keep in mind good research kills bad ideas. Focusing on these questions can help you collect mounds of data, but nothing really of value, leaving current bad ideas to fester and mutate while reachable or recognizable good ideas never get discovered.

Studying something fruitless means losing time, money, energy, and participants. Your stakeholders won't learn anything of value. And if you've rushed a study and conducted biased, convenient research, then the things you learn end up being worse than doing nothing at all.

If you study enough fruitless questions, your stakeholders will question the value of research, choosing to devote fewer and fewer resources to it. Worst of all, your stakeholders or business might decide to skip any research altogether.

You have to get better at recognizing what not to study.

What can you do? To maximize the resources and time you have, you'll have to be really good at recognizing the difference between fruitless and fruitful research questions. The goal isn't necessarily knowing what to study but to be better at recognizing what not to study. Being able to differentiate between valuable and costly research questions is a critical skill you’ll need to develop to become an effective and impactful researcher. So what is a fruitful research question?

‍

Qualities of Fruitful Research Questions

Beyond an answer to the “So what?” question, a fruitful research question gives you and your stakeholders direction and focus when conducting research. A fruitful research question has a clear goal, alignment, and collaboration from stakeholders and gives you guidance when designing, recruiting, collecting, analyzing, and reporting findings. You want your research questions to be valuable, meaningful, and sustainable whenever you can.

A fruitful research question is valuable to your stakeholders, meaningful to your participants, and sustainable for you to study.

Every study you run has three groups of people involved: stakeholders, participants, and you. Stakeholders care about value, so any question studied has to be practical and important to them and the business. Your participants don’t want to answer 100 boring survey questions but give you meaningful feedback. Your questions must be relevant and engaging to them or you risk low study participation. Finally, any question studied has to be sustainable for you. You can’t always study small or hyper-specific research questions repeatedly without exhausting your attention and energy.

Below are examples of fruitful research questions across various products and domains.

Fruitful Question Examples

• What are the common phases new homeowners go through when buying large appliances?
• How has the streaming services’ “skip intro” function affected new customers’ behavior during their free month trial?
• What are the self-reported attitudes towards political in-feed advertising from emerging market users?
• What’s the relationship between app satisfaction & gaming frequency for young gamers between the ages of 10-15?
• How often are female users in Latin America using the 2x playback speed function? Why?

If you had to abstract the questions to find qualities of fruitful questions, you’d end up with the list below. Use the qualities as a "yes/no" checklist. If you answer "yes" a lot, it's a good sign you're on the right track. Do note that you’ll likely have several interconnected research questions to study at one time. Make sure the most important ones fit the qualities below.

Qualities of Fruitful Research Questions

• Valuable to your stakeholders
• Aligned and agreed upon with your stakeholders
• Meaningful to your participants
• Clear description or understanding of who the ideal participant is
• Sustainable for you to study (over time)
• Specific but not answered with a yes/no, a single statistic, or a short response
• Actionable because the post-study learning is designed to be used
• Addressable within your constraints
• Gives guidance and boundaries when collecting, analyzing, and reporting data
• Focused on a product-related problem or a prediction about the product

Let’s spend some time discussing the last quality listed above: studying problems and predictions.

‍

Studying Problems

The process of making & supporting any product will result in problems. Why is usage slowing in this market? Why didn't that feature work? Why are customer support colleagues helping so many people with an updated process? Problems are about existing products and how they're not working well today. Tactical research (an idea covered more in Topic 3 of this handbook) is done to understand these problems and validate their solutions.

As a general rule, you want to focus on problems that overlap with business goals and user needs because any solutions or opportunities out of your research will benefit both groups. But sometimes, your stakeholders might ask for research that’s slanted only towards business goals (such as ”we want research to help us increase the number of email sign-ups this quarter”).

Avoid or limit studying these types of problems. The value generated from studying these kinds of problems will be low, meaning you’ll spend more of your limited resources for very little gain. And remember, people don't care if the business reaches its goals.

Problems - like people, products, and stakeholders - evolve over time.

You should also be aware that product problems aren’t static. What confused and puzzled your stakeholders last year won’t be the same next year. Problems - like people, products, and stakeholders - evolve over time. Whenever you hear about a product experience problem, think about the problem across many dimensions. Some questions are listed below to help you expand your understanding of any problem (you can also ask these questions during stakeholder interviews and meetings).

Guide 04: Conducting Stakeholder Interviews

Ways to Think about Product Problems

• How many people have this problem?
• How long has this problem been happening?
• Is this problem part of a larger upward or downward trend or cycle?
• What would happen to this problem if the business did nothing for three weeks? Three months? Three years?
• What is the impact of this problem? (financially, to the brand, the legal implications, etc.)
• How would the business know the problem has been solved? 

Whenever you start to study a product experience problem, be ready for your stakeholders not to take any action. Being aware of a problem isn’t the same as alignment among stakeholders to solve that problem. Instead of being discouraged, consider your research a valuable piece of evidence to help prioritize what problems should get solved and when.

Whenever you can, have a conversation about what are the possible or plausible solutions the team can implement before running the study. It’ll help limit the scope of your research and increase the chances that impactful solutions get implemented. Check out the guide below to find ways to have more meaningful conversations before a study starts.

Guide 02: Aligning on a Study

You don’t only want to study problems. Whenever you can shift focus, look to the future and study predictions.

‍

Studying Predictions

Fruitful experience research can also be done on predictions. Everyone you work with is making predictions for the future. In a fast-changing world that's drowning in more and more technology, everyone works with and ingests lots of data. They use these data as signals to predict what'll happen in the future. However, by definition, a prediction could be wrong. But once you identify a prediction, you can break it apart to find fruitful questions.

Let's talk through an example you’re a researcher at you're a researcher at a company that makes educational software for math teachers. The software helps math teachers do live-stream classes, assign & grade homework and even proctor exams. Most of these math teachers work in a remote or distance-learning environment.

Right now, senior leadership at the company wants to add another market to expand its business: science teachers. Your stakeholders are planning on modifying current software elements to make it work for science teachers. It’s much cheaper and faster to change a few parts of the math teacher software for science teachers.

In this scenario, can you see the prediction? It's simple but might send the company down the wrong path. The prediction is that math teachers have similar needs as science teachers. But is that right?

What does a science teacher consider to be a successful class? How do they spend their instructional time compared to a math teacher? Do science teachers more or less support grading assignments?

Your stakeholders might be focused on a half-baked solution that’ll generate unnecessary, unpleasant, and entirely avoidable issues to be solved later. A short study comparing and contrasting science and math teacher needs might make all the difference between a positive user experience and an unprofitable product.

Regardless of where you work, what you work on, and who you study, there are fruitful questions in your work environment. So how do you actually find them?