There are lots of ways to collect information from respondents. Each methodology has its merits, but we wanted to know if one is better than the rest.
In order to answer this question, we analyzed data from 17,173 recruitment projects posted on Respondent, by researchers from companies like Microsoft, Intuit, Home Depot, and Atlassian.
Some interesting trends emerged.
Methodologies are going remote
As the trend line reveals, the share of remote studies is increasing, while the share of in-person studies is decreasing.
With the proliferation of great online tools making it easier and the cost of remote methodologies relatively low when compared with in-person methodologies, it’s no surprise that researchers are looking to conduct more of their research remotely.
As ‘Remote’ and ‘In-Person’ are broad categories, we dug into the data further.
2. Among remote qualitative methodologies, ‘one-on-one’ is dominant (n = 13,026 studies)
The reason ‘one-on-one’ dominates as the preferred methodology among remote studies is twofold:
Commercial Research Pressures: Or to put it more simply, deadlines and budgets. One-on-one research is typically faster to recruit for and doesn’t require special tools to conduct. Because of these reasons, it is also generally cheaper to complete than other methodologies.
Democratization of Research: Given the increase in non-traditional researchers like Product Designers and Front End Engineers conducting research, it makes sense that there is a preference for the methodology with the lowest level of complexity when compared to other methodologies.
When analyzing the benefits of the one-on-one methodology across the variables of speed, cost, and difficulty, it is the fastest, cheapest and easiest to conduct.
Furthermore, the correlation between popularity and relative strength across the variables of speed, cost, and difficulty continues across methodologies:
NB: These methodologies have been ranked by the author based on a decade of research experience and expertise.
Here we can see the second most popular methodology also scores second overall across the variables of speed, cost, and difficulty. The third most popular scores third, and so on, showing the least popular methodology, focus groups, to be the slowest, most difficult, and most expensive.
The relationship between popularity and the variables of speed, cost, and difficulty implies that these variables are researchers’ top considerations when choosing a methodology — the faster, cheaper, and easier a methodology is, the more likely it is to be chosen.
In order to test this theory further, let’s take a look at the popularity of in-person studies.
3. For in-person studies, one-on-one wins again (n = 3,132)
Given the prevalence of focus group facilities and the space that focus groups occupy in our collective understanding of market research, we did not expect the dominance of one-on-one interviews among in-person methodologies.
One of the possible reasons for the dominance of the one-on-one methodology is the massive growth of UX research*. As UX research is primarily concerned with online experiences which are largely an individual pursuit, it follows that researchers would prefer a methodology that best mirrors that lived experience. But does the one-on-one methodology also satisfy the greatest number of criteria across the variables of speed, cost, and difficulty?
It turns out, it does:
For the purpose of analysis, we removed ‘Other’ as it covers a wide range of methodologies. Again, these methodologies have been ranked by the author, based on experience and expertise.
In looking to the bottom of the table, the reason why ‘in respondent’s office’ performs the worst in the ranking is because of its extremes:
Difficulty: First you must recruit an employee, then they need to ask their boss’ permission to allow strangers into their workplace, then you must coordinate travel and schedules.
Cost: Travel to visit the respondent’s office plus offering a large enough incentive for people to agree to this add up.
Speed: Because of the onerous requirements, fewer respondents are willing to participate in this type of study, making it slower to recruit.
That the least popular in-person methodology also scores the lowest across the variables of speed, cost and difficulty supports the theory that these are the variables researchers consider most when choosing what type of study to conduct.
4. Does this mean fast, cheap and easy are best?
Fast, cheap, and easy have long been ultimate goals across industries and implementations. However, they do not account for the ‘value of insights,’ ie: the output.
When considering ‘what is best,’ it is prudent to consider the output alongside the input, as it is the reason we conduct the study in the first place.
Therefore, if we are to include ‘value of insights’ as a variable, one way to determine ‘what is best would be to compare the insights from a slow, expensive and difficult study — like an In Respondent’s Office study — to the insights from a fast, cheap and easy study — like a Remote One-on-One study. These two studies would need to share the same objective, the same participants, and the same researcher for an effective comparison.
In this scenario, you might expect (and would definitely hope) that the In Respondent’s Office study would deliver considerably better insights to justify its slow speed, high cost, and difficulty to complete. However, many researchers would conclude that the high cost, slowness, and difficulty would erode the additional benefit this methodology would deliver over its faster, cheaper, and easier counterpart.
5. So, what is best?
Despite the potential for slower, more expensive, and more difficult studies to yield better insights, it is hard to enumerate this benefit. Inputs, however, are easier to evaluate than outputs. In research, inputs can be enumerated in days, dollars, and years of experience required.
For this reason, researchers favor methodologies that perform better across known variables — speed, cost, and difficulty. By these variables, remote, one-on-one research is the best performing methodology by a wide margin.
As a community of researchers, product managers, and designers, we may yearn for complexity and depth, but we remain shrewd observers of tangible variables, apprehensive to weigh too heavily those as abstract as “insights.”
*To validate the claim that UX Research is growing, we can look to The Bureau of Labor Statistics. UX Researchers are included in the ‘Market Research Analysts’ category or the ‘Web Developers’ category, where there has been exceptional growth over the last decade.