There are lots of ways to collect information from respondents. Each methodology has its merits, but we wanted to know if one is better than the rest.

In order to answer this question, we analyzed data from 17,173 recruitment projects posted on Respondent, by researchers from companies like Microsoft, Intuit, Home Depot, and Atlassian.

Some interesting trends emerged.

Methodologies are going remote

As the trend line reveals, the share of remote studies is increasing, while the share of in-person studies is decreasing.
​
With the proliferation of great online tools making it easier and the cost of remote methodologies relatively low when compared with in-person methodologies, it’s no surprise that researchers are looking to conduct more of their research remotely.

As ‘Remote’ and ‘In-Person’ are broad categories, we dug into the data further.

​Among remote qualitative methodologies, ‘one-on-one’ is dominant (n = 13,026 studies)

The reason the One-on-One methodology dominates as the preferred methodology among remote studies is twofold:

When analyzing the benefits of the One-on-One methodology across the variables of speed, cost, and difficulty, it is the fastest, cheapest and easiest to conduct.

Furthermore, the correlation between popularity and relative strength across the variables of speed, cost, and difficulty continues across methodologies:

These methodologies have been ranked by the author based on a decade of research experience and expertise.

Here we can see the second most popular methodology, Unmoderated Studies also scores second overall across the variables of speed, cost, and difficulty.

The third and fourth most popular, Diary Study and Focus Group show the least popular methodologies, as these research studies tend to be the slowest, most difficult, and most expensive. They also require a lot of communication and planning.

The relationship between popularity and the variables of speed, cost, and difficulty implies that these variables are researchers’ top considerations when choosing a methodology — the faster, cheaper, and easier a methodology is, the more likely it is to be chosen.

In order to test this theory further, let’s take a look at the popularity of in-person studies.

Given the prevalence of focus group facilities and the space that focus groups occupy in our collective understanding of market research, we did not expect the dominance of one-on-one interviews among in-person methodologies.

One of the possible reasons for the dominance of the One-on-One methodology is the massive growth of UX research*.

UX research is primarily concerned with online experiences; a largely individual pursuit. It follows reason then, that researchers would prefer a methodology that best mirrors a lived, anecdotal experience.

But does the One-on-One methodology also satisfy the greatest number of criteria across the variables of speed, cost, and difficulty?

It turns out, it does:

For the purpose of analysis, we removed ‘Other’ as it covers a wide range of methodologies. Again, these methodologies have been ranked by the author, based on experience and expertise.

In looking to the bottom of the table, the reason why ‘in respondent’s office’ performs the worst in the ranking is because of its extremes:

That the least popular in-person methodology also scores the lowest across the variables of speed, cost and difficulty support the theory that these are the variables researchers consider most when choosing what type of study to conduct.

4. Does this mean fast, cheap and easy are best?

Fast, cheap, and easy have long been ultimate goals across industries and implementations. However, they do not account for the ‘value of insights,’ ie: the output.

When considering ‘what is best,’ it is prudent to consider the output alongside the input, as it is the reason we conduct the study in the first place.

Therefore, if we are to include ‘value of insights’ as a variable, one way to determine ‘what is best would be to compare the insights from a slow, expensive and difficult study — like an In-Person study that takes place at the Respondent’s Office — to the insights from a fast, cheap and easy study — like a Remote One-on-One study.

These two studies would need to share the same objective, the same participants, and the same researcher for an effective comparison.
​
In this scenario, you might expect (and would definitely hope) that the In-Person Office study would deliver considerably better insights to justify its slow speed, high cost, and difficulty to complete.

However, many researchers would conclude that the high cost, slowness, and difficulty would erode the additional benefit this methodology would deliver over its faster, cheaper, and easier counterpart.

5. So, what is the best research methodology?

Despite the potential for slower, more expensive, and more difficult studies to yield better insights, it is hard to enumerate this benefit. Inputs, however, are easier to evaluate than outputs. In research, inputs can be enumerated in days, dollars, and years of experience required.

For this reason, researchers favor methodologies that perform better across known variables — speed, cost, and difficulty. By these variables, remote, one-on-one research is the best performing methodology by a wide margin.

As a community of researchers, product managers, and designers, we may yearn for complexity and depth, but we remain shrewd observers of tangible variables, apprehensive to weigh too heavily those as abstract as “insights.”

​*To validate the claim that UX Research is growing, we can look to The Bureau of Labor Statistics. UX Researchers are included in the ‘Market Research Analysts’ category or the ‘Web Developers’ category, where there has been exceptional growth over the last decade.