I’ve heard a few times now – from some within (and some flirting on the edges of) our industry – that market researchers need to stop focusing on research methodology so much.

There are two main ‘reasons’ put forward here;

1. First, that methodological considerations are so elementary that we don’t need to talk about them any more [they’re not, and we do]
2. And second, that it’s simply passé to focus on methodology; if we do, the market research industry will be left behind [it’s not, and if that’s the reason the market research industry is going to be left behind – if, indeed, it isactually going to be left behind – I’ll eat my stripy hat].

The upshot of these discussions is the irritating and ill-defined – but always emphatic – argument that we, as an industry, need to innovate.

But sound methodologies and innovation are not mutually exclusive concepts.

Notwithstanding that research methodology, itself, can be innovative, we need not, should not, and must not sidestep careful methodological considerations for each and every market research project we undertake.

Careful methodological considerations – built into the research design, and used to frame the analysis – are a fundamental bedrock for useful innovation.

‘One big focus group’

I’ve heard this term thrown around a bit (Google it) to describe ‘naturally occurring’ conversations on the internet as a rich source of customer insight. It annoys me.

Listening to/gathering online content is absolutely nothing like a focus group.

Beyond the obvious (ie there’s not a great deal of focus in terms of sample, or being able to probe specific topics), and maybe surprisingly, it lacks some key aspects of spontaneity than can be generated in a focus group. Moreover, unlike a focus group, it doesn’t readily provide a good foundation for analysis.

Highly edited

In many cases that we, as market researchers, will be interested in, the content that ends up in the netographer’s dragnet – eg a blog post, a comment on a community thread, a tweet, etc – is the product of a process that involves considerable cognitive effort on the part of the creator; either word crafting a thought/response, choosing an image, shooting and editing a video etc.

And various factors will shape that effort, such as the intended or perceived audience, the perceived importance of that audience, the background and motivation for generating and posting the content in the first place etc.

Clearly, this is a highly controlled, highly edited process – a fact that seems at odds with the perception of unsolicited online content as somehow more authentic than the content a researcher can get via other (more direct) methods of inquiry.

But the key issue here is that there are very few clues to help the researcher understand the motivations of any particular individual driving their broadcast to the online world.

The mother of issues

As a qualitative researcher, motivation is one of the key factors I’m interested in understanding. It’s the very cornerstone of my analysis; the context. But I can’t readily get that online. Without directly asking, in a timely and appropriate fashion (a whole other blog post), I’m not privy to the backroom.

Of course, not understanding the motivations/context behind the content is fine if you’re simply gathering and presenting content. But it’s the mother of issues if you want to provide your client with any substance.

We’re getting good at capturing data and making it look pretty. But is our industry paying enough attention to its analysis?

Fluffy techniques

Useful qualitative research output has two defining features; depth and clarity. And perhaps surprisingly, this is where what might be perceived to be ‘fluffy’ research techniques come into their own. They’re an efficient way to get depth and clarity.

An example, asking the same question in two different ways, will help to explain what I mean.

Q: Does Brand X have an image problem?

There are 3 possible answers to this question; ‘Yes’, ‘No’ or ‘I don’t know’. Some clarity yes, but not much depth there. None of these answers are very helpful in terms of understanding anything about the possible image problem.

Let’s ask the same question, but in another way:

Q: If Brand X came to life as a person, what kind of person would they be? What kind of music would they listen to? How would they take their coffee? etc.

I can understand that to someone who hasn’t had much experience with qualitative research, the above line of questioning may seem somewhat fluffy.

Allow me to de-fluff it.

The output

As noted, the second question is just another version of the first. But the answers will be quite different.

Asking the question using this ‘fluffy’ approach delivers answers in 3D. And then following up, down and sideways on the answers with ‘Why? Why? Why?’ (not literally, in essence) helps to build a rich, relatively holistic picture of Brand X’s image. Critically, it also provides a context for interpretation.

The above technique is called personification. It’s just one example of a fluffy technique: there are many more in the qualitative toolbox. When used appropriately, fluffy techniques really deliver the goods. Research participants love them: they’re fun and engaging and something a bit different. Researchers love them because they enable deeper understanding of attitudes and perceptions, and thus greater insight.

Effectual fluff. Neat huh?

(This was a post I originally wrote for Marketing Mag)

Do you agree or disagree with the following statement…?

Data quality is critically compromised when double-barreled questions/statements are used in market research surveys and researchers who write them into their surveys should be shot.

;P

What has Botticelli’s The Birth of Venus got to do with market research? Well, note the crop. But as nice as it is, it’s only the bigger picture that tells us the actual story.

This is at the heart of the next issue for discussion; the ‘new’ (?!) practice of listening online.

Of course ‘listening’ itself, as a method of research inquiry, is pretty obvious and hardly new. What is new* for market research however, is;

• the online location per se,
• some of the online listening technologies, and
• in some cases, the actual content generated online.

But new or not, you still need to know exactly who it is that you’re listening to. And you also need to think about the context.

For example…

If you’re undertaking a market research listening exercise for a client, let’s assume, quite reasonably, that your focus will be on listening to their customers or potential customers.

Is the Internet a good place to listen?

Well, of course it is! But there are some very important questions to ask before you begin;

Of all your client’s customers/potential customers, how many have access to the Internet?

And of all your client’s customers/potential customers who have access to the Internet, how many are confident enough in a) their opinions and b) their writing (or video editing) skills, to express those opinions publicly online?

And of all your client’s customers/potential customers who have access to the Internet, and who are confident enough in a) their opinions and b) their writing (or video editing) skills to express their opinions publicly online, how many bother?

And of all your client’s customers/potential customers who have access to the Internet, who are confident enough about a) their opinions and b) their writing (or video editing) skills to express their opinions publicly online, and who bother, how many express those opinions in an articulate way (ie in a way that a marketer or market researcher might find of use)?

The skews are breaking my brain.

Defining your sample in terms of exactly who it is that you want to get feedback from is absolutely key in terms of determining where and how you should collect your data.

Notably, if your client’s target market comprises a wider group of people than the customers or potential customers who fit the very narrow profile described above, then – and critically – collecting useful data to generate useful output will mean going well beyond the insights you glean from your ‘new’ online listening endeavours.

*relatively speaking and/or as the hype would suggest

There’s been a lot of talk about engaging research participants in this ‘new’ research paradigm.

I’m focusing on qualitative market research here because firstly, that’s my thing…

: )

…and secondly, because I hear that ‘engagement’, within the context of market research online communities, is community-nirvana. The ‘best’ communities are engaged communities.

This strikes me as, paradoxically, both obvious and alarming.

There’s a very fine line between engaging research participants enough to… well…participate in our market research, and over engaging them.

Without due diligence, research effects (pick one of many) are likely to confound the research output in unintended, unexpected and underestimated ways.

I don’t, for one second, suggest that other research methodologies are free of research effects – they aren’t! – but surely this doesn’t automatically generate a license for ‘new’ market research to ignore them.

And while I don’t think you can necessarily control for engagement, some questions;

• To what degree should you try to ‘create’ it?
• How much is too much? When does it start to mess with what you’re looking for from your research?
• How do you disentangle it from your analysis?

I’ve just come back from a fantastic trip to Chicago where I attended the ESOMAR Online Research 2009 conference. You can find Jeffrey Henning’s brilliant recap here (he pretty much live-blogged it – very impressive!).

Anyway, not surprisingly, there was lots of talk of ‘new’.

And it would have been easy to come away with the message that the market research industry really needs to get with the ‘new’ programme or, quite simply, it will wither away and die.

Because the new world of research is here! New methodologies. New technologies. New ways of engaging with respondents (Gasp! Did I say that? I meant ‘participants’, ‘co-creators’ or ‘collaborators’).

All well and good, but what does this actually mean?

New kinds of output?

Let’s all take a moment to think. What, as market researchers, is our purpose?

Market research is about understanding the market. At a very basic level, the end goal is to deliver information that will help our clients make relatively informed/better decisions about how to sell their products or services.

And here’s my point; the output (ie what our clients are paying for) is only ever going to be as ‘new’ as the questions they/we ask*.

Over the next few blog posts I’m going to take a look at some of the elements of ‘new’ I outlined above. Examine them closely. Explore what they mean for researchers at a practical level. Separate, if you like, the hype and theory from the actual task of delivering useful output.

Should be interesting…

: P

*If you’re in the ‘listening’ camp, ie “Oh no, no! We don’t ask questions, we just listen to the conversation!”, I’ll argue that you’re still (implicitly) asking questions when you choose to/not to include any particular content in your analysis.

The term ‘traditional research’ is often used to describe offline methodologies; mostly, I’ve observed, in the context of selling what I assume must be ‘non-traditional’ (??) online methodologies.

In this context, the word ‘traditional’ conjures imagery of antiquated, moth-eaten and fusty research practices.

And clearly, many offline methodologies are anything but. Not to mention that many research solutions require a hybrid of both online and offline approaches.

But if we have to throw labels around, it’d probably be more accurate to substitute the word ‘traditional’ with ‘established’ (as Paul Vittles from TNS usefully suggested during question time at an AMSRS breakfast seminar I attended last week).

So does this make ‘non-traditional’ research ‘unestablished’ research?

: P

Wow. Doesn’t that put a whole new spin on it!?

Great post over at Ray Poynter’s (always) inspiring blog.

The way Ray described a short-term community, I think, delineates a clever, and potentially efficient, research methodology.

But I’m yet to be convinced of the long-term research community concept (although I’ve no doubt some readers are probably pulling their hair out over my apparent inability/reluctance to see the light/‘get it’).

: P

I’m assuming that in a ‘community for market research’ (vs a ‘community for customer relations/brand building’) context, a client will actually have some questions they want to ask and/or issues they’d like to explore.

If they don’t – if it’s a case of just wanting to pick up on issues entirely generated by the research community itself – then I’m guessing that they’d probably get better value by undertaking some basic social media monitoring … (god forbid).

Anyway, assuming we have a community that’s being used for market research, and there are some questions to throw into that research (in whatever shape or form), here are my questions;

1) At what point in time, along the short to long-term spectrum, does the nature of the output you get from a research community change? And probably more importantly, in what way does it change?

(Or is it different from day one because of how the participants are briefed about purpose of the community and their role in it?)

2) Would one analyse the data coming out of long-term vs short-term research communities differently? This, I suppose, brings us to the issue of the research community objectives. From the sales pitch, I sometimes get the impression that the sole objective of the exercise is to get the research community members to bond; never mind the insight, they’re bonding!

: P

Kidding.

But really, what are the issues, the benefits and drawbacks of community members ‘developing a sense of community’ (and from that, I would assume we mean ‘belonging’), particularly over a longer time frame?

Does it make participants more honest, or more willing to share? Maybe. But (it could easily be argued), maybe not. And what impact does, for example, group/clique think etc have on the output? How would you identify/measure the impact in such a wide (uncontrolled? quasi-controlled?) landscape?

(And to take it to the extreme, if one’s aiming for ‘uncontrolled’, then back to the point above; isn’t the ‘community’ just a very limited method for undertaking social media monitoring?)

3) If developing a sense of community/belonging is one of the key operating principles for long term research communities, then how do new recruits (or exiting members for that matter) impact the existing community dynamic and thus affect the output? How about changes in community moderators/managers?

4) Are there any studies looking at the differences in terms of valuable/usable output between short-term vs long-term research communities?

Are there any actual or even theoretical definitions of the ROI (tangible or even intangible) that one might expect from a short-term compared to a long-term research community available?

(I’ll probably come back to this with more questions when I’ve had time to think more).

Coincidentally, after my last post, this came up in my Google Reader today;

More on the Problems with Opt-in Internet Surveys

Here’s the first article;

Study Finds Trouble for Opt-in Internet Surveys

I had the great privilege of attending an AMSRS Professional Development session earlier this year to hear Jon Krosnick speak. He was brilliant.

And so it’s with great interest that I follow this very timely and fascinating debate (hosted on Gary Langer’s excellent blog).

Make sure you follow the links to get the full story/debate. It’s an important one.

P.S And here’s a link to the study itself.

So! Continuing with the #stuffiwonder theme…

The telephone vs online survey debate.

The one that goes;

“Really, given that everyone’s moving from landline to mobile/cell these days, telephone survey sample representivity is seriously compromised”.

More often than not (and, of course, depending on who’s doing the debating), it ends with a nod to online panel surveys. In this context, “…they’re probably just as good as – if not better – than telephone surveys”.

Right?

Well, I don’t know.

Panels are opt in. And yes, the same can (and should) be argued about telephone interviews. You most definitely need research participants to opt in beyond a “Bugger off, I’m eating my dinner” response.

But what differences might we see, in terms of motivation and the research output, between a sample comprising individuals who;

• Have been approached randomly (and I get that it’s not really random; the population will be limited to those with landlines), vs
• Sign up to be part of a/several market research panel/s and/to get paid for their opinions?

How do market research online community providers populate their clients’ communities?

(When I say “market research” communities, that’s exactly what I mean; a community used as a market research tool. I’m not talking about online communities that are used in a marketing/customer relations exercise.

I’m not quite sure that the difference is apparent to all, but they’re not the same; not by a long shot.

In one, you’re giving the community members love because you want to make them happy. In the other, the relationship is somewhat more pragmatic; you want to learn from them. Notably, if you’re giving them love to make them happy, you’re not necessarily going to learn much, because they’ll be all nice and lovely back).

Anyway, focusing specifically on market research communities; what checks are in place to ensure that the people who end up in the community represent the people the client actually wants to hear from (ie the population of interest)?

To borrow from the delightful John Lacey, I’m filing this one under #stuffiwonder.

Here’s another excerpt from my Marketing Magazine series on Qualitative Research…

Within a qualitative research context, there’s no right way to ask a question per se. There are actually many right ways to ask a question. And there are also many wrong ways to ask a question.

The wrong ways

You may have heard about some of the following heinous qualitative research crimes:

• Asking leading questions
• Asking closed ended questions
• Asking vague questions

Why are these ‘wrong’?

Because leading questions ‘lead’ people to a particular answer, closed ended questions can end the discussion prematurely, and vague questions elicit vague answers that have little grounding for interpretation.

Well, theoretically. But all is not what it seems. An experienced moderator might use any of these types of questions purposefully, and with excellent effect:

• A leading question often works well to test a hypothesis, or as stimulus in itself, to get the conversation going
• A closed ended, or vague question can provide a foundation to open the discussion in interesting and new ways

They’re all part of the qualitative researcher’s toolkit and used in a timely and purposeful way, can add tremendous depth to the discussion.

How exciting –  The Age of Conversation 3 will be out in April!

In the meantime, a list of the authors;

Here’s a post I wrote for Marketing Magazine about the ROI on qualitative research;

The value of market research – whether we’re talking about qualitative or quantitative research – is difficult, if not impossible, to quantify.

That’s because market research rarely, if ever, works alone in shaping strategy. It’s just one of many tools in a marketer’s tool bag.

In addition to this, market research is only ever:

• As good as the research brief and the questions it asks
• As good as the analysis and the debrief
• As useful as its end users make it; it’s what they do with the output that can determine success or otherwise

Given the variables listed above (so called because they vary), it’s pretty much impossible to put a figure on its value per se.

Relevance

Let’s look at it in another way.

If a particular product or service or piece of communication is relevant, it’s far more likely to end up in the shopping basket (so to speak). So the absolutely fundamental, most basic question for marketers should be:

“How can we make our products/services/communications more relevant to our customers/potential customers?”

And there are two ways marketers can go about answering this question:

1. They can ask their customers/potential customers
2. They can guess

Ask them

If marketers ask their customers/potential customers (and listen to them), they’ll be in an excellent position to create relevant products, services, communications etc.

The value of qualitative research here is obvious; it’s a very good way of asking, and listening, to your customers/potential customers to find out what’s relevant to them.

By being relevant, you’re optimising the chance of collecting the sale. Therein lies the return on your investment.

Guess work

If marketers decide not to ask and, in effect, guess what the market wants, they run the risk of getting it wrong.

Consider the time, resources and money wasted when bad guesswork delivers a dud. Go one step further; cost it out. And add the opportunity cost.

When you have that figure, my work here is done. Because that figure gives you a very good estimate of the ROI for good qualitative research.

Nothing to sneeze at, is it?