How to Run an Effective Beta Program

Author: Brian Lawley – President, 280 Group LLC

Introduction

As a Product Management professional who is responsible for the overall success of your product, it is important that you ensure that the Beta programs you run are comprehensive and effective (even if you are not the one who has ultimate responsibility to run the program). This article will include some of the tips and best practices that I have found to be helpful in making Beta program successful over the years. Oftentimes customer Beta programs are one of the most critical factors in the success or failure of a product. Yet many times they are not well-planned and are executed at the last minute with a lack of resources and focus.

An effective Beta program can provide you with valuable information, such as whether your product is truly ready to ship, what features should be in the next release and how satisfied customers will or won’t be with the product. If done right, it can also set you up for critical launch and marketing deliverables that will add credibility to your story as you bring the product to market.

Setting Appropriate Goals

Make sure you define your goals up front and early. What are you really trying to accomplish? Do you want a broad cross-section of customers to spend extensive amounts of time using the product to prove that it is ready to ship? Or is your QA very extensive, and you just need a few customers to validate readiness with some hands-on customer usage. Do you want to gather early feedback for the next version? Or perhaps you want to find a group of customers that will be willing to talk to the press or provide you with quotes or testimonials.

Make Your Goals Concrete

The earlier you set your goals and the more concrete they are, the better. For example, I like to state that a certain number of customers must have installed the product, used it for N days (or N number of times or hours) successfully and found no major crashing bugs. I also like to agree up front with the team that the Beta testers will be surveyed at the end of the program, and that a certain percentage of them have to indicate they believe the product is ready to ship. (i.e. 9 out of 10 Beta customers must agree that it is ready).

Other goals might include finding 3-5 customers that are willing to give you quotes and talk to reporters when they ask for references. Or you might want to set a goal that the number of bug reports must show a decreasing trend to an acceptable level before the product can be declared ready to ship. Obviously there are a number of metrics you can use, but without defining success up front it will be difficult to create your overall plan, and even more difficult to push back to the team when there is pressure to ship the product and you don’t have enough data to be sure if it is ready yet.

Recruiting Participants

One of the biggest mistakes that I see companies make with Beta programs is that they significantly underestimate how difficult it will be to get customers to sign up for their Beta programs. They are usually pretty enamoured with their own products, and they assume that customers will be more than willing to participate.

Your success at recruiting customers will depend on a number of factors, including what stage of the product lifecycle you are in (upgrade version, brand new product, etc.), how well known and prominent your company is, and how popular your products are (Apple could probably find iPod Beta testers pretty easily). Other factors will be how easy it is to find your target customers and how much effort and time you will be asking them to put in.

If you have an Enterprise software application from a completely unknown new startup company that will require the Beta customers to go through a lengthy installation (or worse yet, could potentially affect their other mission-critical systems), you may have a difficult time recruiting. On the other hand, if you have a consumer application that takes little time and provides immediate benefits, you may find it easier to recruit.

Depending on all of these factors you will need to gauge what kind of recruiting program you need to do and how strong the incentives need to be. You may need to make personal phone calls and visits in order to convince customers to participate. Or you may be able to use email or even a form on your web site.

Sources for recruiting include the following:

Current customers
Prospects that didn’t purchase
VCs/Investors
Your personal network
Sales force & leads
Advertisements (Craigslist, local newspaper)

Incentives

In terms of incentives, there are many things that you can offer participants. Certainly the “Help us improve the product” angle will work to some degree (more if you have a fanatical user base). You might also want to offer free or reduced pricing or upgrades. Or you may even want to have a contest to keep users motivated. For example, one of the recent Beta programs that the 280 Group ran was called the “Great Bug Hunt”. For each valid bug submission the Beta tester received an entry into a drawing for an iPod. Not only did this incent them to sign up, but it got them enthused to continue using the product throughout the course of the whole program.

Response Rates

At this point you’ve created a compelling recruiting program and need to contact customers to get them to participate. Just how many customers do you need to contact in order to get an adequate number that agree to be part of the program? More importantly, how many will actually end up testing the product and doing what they have promised?

The response rate you’ll get in terms of participation and actual usage for a Beta program will vary widely based on a number of factors, including:

Popularity of the product
Whether the product is completely new and unproven
Who your company is (well-known or unheard of)
How personal and compelling your recruiting approach is
How stringent you are at selecting customers that fit your profile
Whether or not your product will affect mission-critical systems at the customer
How much time and effort you are asking the customer to put in

To give you an idea of the range of actual participation you might expect I’ll use numbers from Beta programs that I have run over the years.

For an existing (vs. brand new) product that is very popular, low-risk to install and use and doesn’t take a lot of time and effort you might be able to get away with contacting only 25 people initially. Out of those 20 would likely fit your criteria (if you were targeted in your initial contacts), and 15 might sign up. Of these you can expect 8-10 to actually use the product enough to give you some valid Beta feedback. (roughly 30% of the initial number contacted)

On the flip side, for a brand new unknown product from an unknown startup that has a high risk and time commitment for installation, you might contact 100 customers initially, find 40 that are interested, have 20 sign up and in the end have 5-8 that are actually active. (roughly 5-8% of the initial number contacted).

Beta Agreements

Once you’ve got the companies agreeing to participate you’ll want to have them sign a Beta agreement. It doesn’t have to be a formal agreement (though in bigger companies you’ll be forced to use an actual contract), but it should clearly state what the commitments and expectations are. Make it as simple as possible and include details such as length of program, incentives and rewards for participation, responsibilities of participants, expected amount of usage and support that will be provided. Getting participants to actually sign an agreement makes their commitment much more real, and you will have a higher probability that they will actually use the product and provide feedback.

Kicking Off the Program

Once you’ve lined up all of your participants and gotten them to sign a Beta program agreement and NDA you are ready to kick off the program. The most important thing at this point is to do everything that you can to avoid a false start. If the participants have a bad first experience with the product your chances of getting them to continue putting in effort will be much lower.

To avoid this make sure you have agreed with your team about what the criteria should be for starting the program. For example, you may all agree that the program won’t begin until all fatal/crashing bugs have been fixed. Or you may have the criteria that N number of users must have been running the product internally with no major problems for at least a week.

The other approach you may want to use is to deploy with one or two of the participants that are more technical and/or that you know better than the others. You may even want to go their company if they are local and watch them get up and running so that you can figure out where any of the likely bumps in the road might be for other participants.

Make sure that all components of the product are solid before deploying anything. Check and double check the installers, include a FAQ document so that participants won’t have to contact you to get answers, and include documentation and help system content if at all possible (even if these aren’t complete, include them so that you can get early feedback).

Running the Program

Once the program has started it will be critical that you communicate regularly with both the participants and your internal team. For smaller programs you should call them at least once a week to check in and make sure they are using the product and having no issues. For larger programs send regular email communications. Make sure you communicate about the overall status of the program, bug fixes/new versions that they need to install, updated FAQs and details about contests or incentives. If the participants hear from you regularly they will be reassured and be much more likely to continue putting in time and effort to help you with the product.

Also make sure to communicate internally with your team. Provide a weekly status report including number of bugs reported, usage by participants, whether you are meeting the stated goals and what you need from the team to continue to be successful. It’s critical that you keep the team informed so that as the program gets ready to wrap up everyone knows the status and there aren’t any surprises or disconnects about the results.

Exit Surveys

When you are ready to end the Beta program you should have the participants complete a short exit survey. If there is an incentive or a contest, make it a requirement to complete the survey in order to qualify.

The exist survey can be performed via email or using one of the popular online survey tools such as Zoomerang or Survey Monkey. Ask the participants how much they used the product, their overall impressions, whether they believe it is ready to ship and what can be improved. Also ask them to rate the features of the product on a scale of one to five.

Final report

Take the results of the survey and deliver a final report to your team. This should include a short summary, bug trend information, whether or not you met the original agreed upon goals and a summary of customer opinions and feedback. Deliver this report to the team prior to making the decision to ship the software – it will prove to be a useful and fact-based tool to make an informed choice.

Best Practices

You’ve put in a lot of hard work to run an effective Beta program, so make sure you leverage it as much as possible. Write thank you letters to the participants and follow-through on the incentives. When doing so ask if they are willing to be involved in future Beta programs or customer councils. And also ask if it is okay for you to call them occasionally if there are questions about the product when the team needs a customer opinion. Having a half dozen friendly customers that are easily accessible can be very valuable when contentious product issues come up and you need some external validation.

One last thing. Make sure that you also write a summary of the Beta program after the product ships. Include what worked well and what didn’t, and give your team members the credit they deserve for pulling off the program. That way you’ll be much more likely to get the cooperation you need the next time around.

About the Author:

With over 18 years of experience in Product Marketing and Product Management, Brian Lawley is a seasoned Marketing Consultant who helps companies in Silicon Valley and beyond to create breakthrough products and generate momentum in the marketplace. Throughout the course of his career Mr. Lawley has helped companies to define, launch and market over 25 award-winning, highly successful products.

As a Marketing Consultant and founder and President of the 280 Group ( www.280group.com ), Mr. Lawley has served as acting Vice President and acting Director of Marketing for clients, as well as working on a project basis. Prior to consulting he held several management positions with well-known companies in Silicon Valley, including Apple Computer, Symantec, Claris (now named FileMaker), Whistle Communications (acquired by IBM) and Digidesign (acquired by Avid). He frequently speaks on the topics of Marketing and Product Management, is editor of the 280 Insider newsletter and publishes articles in a variety of publications related to these topics. Mr. Lawley is the President of the Silicon Valley Product Management Association (SVPMA), was nominated for the 2006 Product Management Excellence Award for Thought Leadership by the AIPMM and has served on the Board of Directors for the California Coast Music Camp.

Mr. Lawley earned an MBA with honours from San Jose State University and a Bachelors Degree in Management Science from the University of California, San Diego with a Minor in Music and Computer Technology.

You may contact Brian Lawley at www.280group.com

About the 280 Group…

The 280 Group LLC provides consulting, contracting, training and templates to help companies define, launch and market breakthrough new products. For more information or a free consultation call 408-832-1119 or visit their web site at www.280group.com.

To receive future white papers like this one or to subscribe to the free 280 Group newsletter visit www.280group.com.