July 13, 2000
No summary available.
Since BetaSphere started in 1996 they’ve helped 70 companies beta test more than 250 software programs, including many advertising and marketing-related programs and web-based applications.
They are still the leader in the field. Heck, they’re still about the only company in the field. As President Michael Shoppell explains “Traditionally this is done in-house. When companies do go outside, they tend to go to usability labs. But that’s a different focus, those are tiny tiny slivers of information. Or they use market research firms doing surveys. But again that gives you very limited information.” The whole field of marketing software beta testing is, Shoppell assures us, “like where CRM was 8-10 years ago, everybody had home brewed ways of doing things.”
We asked Shoppell to share his top five tips for companies wishing to beta test new software or Web based applications for marketers:
1. HOW TO FIND GREAT BETA TESTERS
You need to look for people who have real business problem that your product or service solves.
We’ve found that’s a problem for many start-up companies in particular. Many of their marketers have a very, very fuzzy understanding of who it is they’re going after and they often don’t have any prospects at hand. They say, “Ok we’re done with product. now we have to go find somebody to use it.”
So, we help them figure out what their market is and how to position their product to make people in it stand up raise their hands. In general marketers aim too wide. We’re very big proponents of Geoffrey Moore: the more specifically you can tailor your message to resonate with these guys, the better results you’ll get.
Lots of marketing products are group-ware oriented. This collaborative type of beta testing these require ratchets up difficulty of getting people to use it, especially if you want clients and their agencies to both beta test something together. Imagine how difficult it is to get both on board! You have to make it very compelling and very easy.
The target number of beta testers needed varies by what kind of information the customer’s looking for. For marketing applications, numbers tend to be in the 20-50 person range. You need to search within a wide cross section of user types, systems and business segments. Build a matrix at beginning of various types of testers you need and then work to fill in holes.
To get people to say ‘yes!’ focus on the benefits of the product. Later you can focus on the software. Tell them why it’s compelling for them to make this change.
2. TIPS TO MAXIMIZE USABLE RESULT
You have to remember when someone is beta testing something for you, they have specific needs they’re trying to meet as well.
When we talk to new potential evaluators, in many cases they have not had good experiences beta testing in the past. The primary complaint is that they don’t hear from company they’re giving feedback to. A company will put something on their web site, ‘here’s a freebie try it out.’ Well, people usually won’t try it. And usually those that do won’t hear back. It’s not too motivating to make an effort if you’re not getting response. Beta testing’s already on the list of 100 things to do and this pushes it to the bottom
So, you need to be really proactive. We provide our evaluators with an easy web-based feedback system. Our customers use it to receive information from evaluators; the evaluators can log bugs, request support, make suggestions, fill out surveys … and actually chat with other evaluators!
Chat use varies from target market to target market. In some cases it gets an incredible amount of usage. Technical people like sys ops and admin ops are less inclined to yack -- they just focus on tech aspect, what’s broken. But businesspeople like to compare notes with their peers, ‘here’s what I found.’
For web-based CRM applications, some companies will also personally discuss the programs with evaluators on the phone or occasionally go one site with a customer. It helps to get customers motivated to use the application.
To get great results, sometimes we put people in teams. We’ll break 50 people into teams of 5-10 to collaborate and compete. The team that finds most issues with product wins something. Our prizes have included Palm organizers, Motorola pagers, Sharper Image items, etc.
[editorial note: if anyone wants us to beta test something we’d prefer a bottle of champagne or a free T1 line for a month!]
3. WHEN TO USE AN OUTSIDE FIRM
The majority the folks we’re working with have their backs up against the wall. Their time and resources are stretched, their personal bandwidth is so limited in terms of managing the program and dealing with evaluator feedback that they could only get an eyedropper’s worth of information if they tried to do it themselves. We turn that eyedropper into a fire hose for them.
You really need to ask yourself ‘am I prepared from bandwidth perspective to follow up with my evaluators? Can I turn around very, very quickly and provide more feedback to them?’ The high-touch element is critical to your success. These are very busy people -- the easier you can make it for them the more likely you’ll get results.
Also, a limited in-house beta test with your own friends and family might be ok if they really represent your target market. But the majority of our customers find it a challenge to find enough people in their targeted market who have the time and interest to make investment in 4-6 weeks to provide meaningful feedback. We have a database of more than 30,000 evaluators we’ve already worked with that we can segment out for you.
4. HOW LONG WILL THE TEST TAKE?
Using our services, the recruiting process will take 2-3 weeks. This includes refining your value proposition, building a user profile, getting the word out to prospects and picking from the people that apply.
Then the dialog begins. An evaluation period of 4-6 weeks is the absolute minimum. For marketing products this tends to be 6-8 weeks because these often require a business process change and it takes longer to get going.
5. HOW MUCH DOES USING AN OUTSIDE FIRM COST?
Things tend to start in the 35-40k range and scale up from there. Costs depend on the scope of the project, the number of people involved, etc.
Although 25-30% of our business is with start-ups, we haven’t gone the equity route. People are generally funded and they are willing to pay for our services because this is their flagship release. So much is riding on it and they have to build it right; build a portfolio of testimonials and references; and target customers to sell it to at the end. So beyond improving your product, beta testing helps you cement relationships with your future market.