Can you conduct an old-fashioned focus group online and get reliable results?

Tom Brailsford, Manager of Advancing Research, has helped conduct and analyze Hallmark Cards' market research for more than 25 years. He says, "I've done about everything there is to do in consumer research. Questionnaire design, stats modeling, strategic simulation, you name it."

We contacted him to hear how Hallmark Cards is testing Internet- based research tactics, and what ideas other marketers could use.


-> Three reasons why Hallmark's testing online tactics

Reason #1: Traditional tactic response rates are going down. "Consumers are harder and harder to reach by phone, harder to reach at malls," Brailsford notes. "There is a large and growing proportion of consumers who do not want to be reached by phone for market research purposes.

Reason #2: Offline tactic costs are rising. The postage rate hike means mailed surveys are a big investment. Focus groups can cost more than $10,000 just to hear from a few dozen consumers in three cities.

Reason #3: Getting in-depth, ongoing feedback from a large number of consumers is almost impossible offline.

"The research I've read suggests that the value of the content you get from a focus group falls off significantly when the group exceeds three in size. In other words, if you have 10 people in a focus group, you aren't getting much additional information from seven of participant. And the average speaking time per respondent is about 10 minutes." Then they leave the room forever.

Brailsford adds, "Traditional marketing research is a point-in- time exercise. You ask a bunch of consumers a question, get back and answer, and you may never see or talk to those consumers again. It's like dipping a ladle into a river to take out a little dose."


-> Hallmark's Online Research Test 'The Idea Exchange'

In November 2000, Hallmark launched an online market research test project entitled 'The Idea Exchange.' Between 200-250 consumers in a particular demographic agreed to visit The Idea Exchange's site on a regular basis. (Since then the Company has added three more Idea Exchange sites for other niche demographics it is studying.)

The Idea Exchange functions as a private members-only community for these consumers as well as a research lab for Hallmark. Features include:

o Bulletin boards for discussions
o quick polling and online surveying
o member profiles
o visual stimuli (such as pictures of members and of suggested
Hallmark Card product creative)

What is most interesting about the features is that both consumers and researchers can use them. For example, consumers can put up their own surveys, start their own discussion threads on the bulletin boards, and post whatever content they would like in their personal profile.

This goes radically beyond most consumer packaged goods companies idea of interactivity online. Interactivity is no longer just consumers taking your survey, but consumers being able to launch their own.

Brailsford explains, "Not only do we have an opportunity to harvest consumer information, but also to watch them discuss issues that are important in their everyday lives. I refer to it as 'online anthropology.' It's an unprecedented view into their lives, the kinds of issues they talk about, and how they talk about them."


-> Selecting and recruiting consumer members

"We didn't make an attempt to create a nationally representative sample," Brailsford says, although he notes that may be possible in the future as the Net population mirrors the general population more and more closely.

Hallmark used a wide variety of tactics to recruit new members, including phone, email, and even viral recruiting by asking current members to nominate new ones.

The goal was to get 200-250 members per community who would take an active role in the online community. People who want to lurk quietly are screened out over time. Members must feel comfortable sharing opinions with others online.

Members also had to have plenty of online experience, including a high comfort level with bulletin boards. "We didn't want to train them on these issues. We didn't want to be dealing with people who didn't know how to use a mouse."


-> How Hallmark uses The Idea Exchange

Brailsford gives five main examples of how Hallmark uses The Idea Exchange:

1. Quick answers. Unlike offline surveys and focus groups (which can take more than two months to devise, conduct and analyze), online surveys are very speedy. You can get an answer back in less than a week.

2. Improving offline surveys. "Research dollars are scarce," Brailsford notes, "If you're going to spend a lot of money to do a huge mailed survey, nobody wants dumb or confusing questions on it, or questions that are going to give you information that isn't useful to you."

"We've had a couple of instances where we tried survey questions on the community first. We asked 'Hey what was confusing about this question?' or 'When you saw this question, what did you think about?' Then the final surveys were developed using this feedback."

3. Indirect questions. Brailsford notes that the problem with most consumer research is that research questions are usually "a head-on attack."

A hypothetical attack-style question would be, "Will the increase in postage affect the number of cards you send? Definitely, Maybe, Not at all … the usual range of responses." An indirect version of the same question might be a discussion on, "How do you stay in touch with people?"

Brailsford says, "Do they even mention postage? You can learn as much from what they don't say as from what they do say." You may even find what you think you want to know is not the best question at all.

In order to get the most from indirect questions, The Idea Exchange asks that members answer posted questions on their own before they look at other member's responses. This greatly reduces the "me too" factor of people simply agreeing with each other's posts without thinking independently and putting it into their own words.

4. Product and packaging development. Hallmark uses the Gallery section of each Idea Exchange community to get quick feedback on package and product design. Plus, Hallmark writers and copywriters often read through transcripts of board discussions looking for verbiage that their marketplace tends to use.

5. High level strategic planning. "Our CEO has actually posted a couple of questions on the Boards," says Brailsford, "the consumers love it that our CEO wants to know what they think. Oh wow, somebody's really listening!"

Brailsford adds that the research team was happily surprised to learn how useful board discussions were in terms of strategic issues. "We asked them to talk to us about issues we were wrestling with. We didn’t tell them it was a key strategy. Their discussion was very informative in some very high-level strategic dialogs."


-> Measuring and analyzing results data

"People always want to know how much revenue, how many new products?" says Brailsford. However he notes a perfect measurement is still impossible, "We're just starting out. It's very much positioned as an R&D effort. It's research; it's an experiment. We don't know exactly how to measure it."

Some measurements are fairly easy, such as consumer votes for new packaging or improving survey question wording. Some are much harder (i.e. how do you tabulate the results from the transcript of an extended discussion, much less it's impact on your company's bottom line?)

"A research analyst might come to us and say, 'I want to post a question about party goods'," explains Brailsford. "For two weeks we get them to discuss that, then we cut and paste the discussion into a Word document. How do you analyze that? It's a very time consuming process."

"If I sat down and read through a bunch of open-ended responses and you sat down, and each of us made a list of key learnings, what are the chances that our lists would match?" asks Brailsford. "There might be some overlap, but maybe not much."

One tactic is to analyze the discussion on the most basic level: "For example, go through all the text and pull out all the nouns and verbs. Make a list. Can you group them into anything that makes sense?"

Brailsford adds, "I think the science of text-analysis is in its infancy." He is very excited about the future.


-> Biggest mistake

"Early on for the first community, we thought we would need big incentives so we spent a lot of time developing a huge incentives program rewarding various points for various types of contributions. We set up this whole bureaucracy to capture points. It turned out to be massive overhead," admits Brailsford.

In reality consumers enjoyed interacting with each other in the communities so much that they did not need big incentives.

Now the community sites offer "minimal" incentives such as occasional low-cost prize drawings, and send a weekly email newsletter to remind people to return.

Brailsford says, "One of the things that startled us was how involved these members became with these communities. They really bonded together. It became like a soap opera, you had to sign on every day to see what was going on or you'd miss something."

"There are some incredible stories. Consumers established prayer boards, posted daily progress reports in their pregnancy, it's been absolutely amazing. If you ask them what do they get out of the site, it is 'I get ideas from other consumers.' They've heard how to potty train kids, they value that. Other incentives can be minimal."