The 5 Types of Beta Testing Programs and Why 4 of Them Suck
We all know the supposed purpose of Beta testing programs: to get customers to actually use about-to-be-released software, in order to find and fix problems that would not have been found by internal-only testing.
So why (why?!?!) do so many so-called Beta testing programs seem explicitly designed to PREVENT this type of feedback?
Please humor me as I classify and explain the different types of Beta programs --- the vast majority of which do NOTHING to improve product quality or identify customer issues.
1. The "Don't You Dare Complain Because This is Free" Beta
This type of Beta is otherwise known as the "Google Beta". An Example? Take Google Gmail, which put out its Beta release on April 1, 2004. Over 5 years later, in July 2009, Gmail at LAST left Beta. No doubt, the only reason why it is not still in Beta is that Google decided to start charging business customers for email.
Clearly, the purpose of this type of Beta is NOT to collect customer feedback. More likely, the "Beta" designation is a euphemism for "No Support. Take It Or Leave It. Don't Complain To Us If It Doesn't Work Because It's Free."
If there is no mechanism in place to actively contact Beta customers and solicit their feedback (as opposed to just relying on them to send messages to a forum or send YOU an email), and if there is no path to ever release this product, well it's a "Don't Complain Because It's Free" Beta.
2. The Fake Beta
This is where a company has a ridiculously short Beta period, and as a result recruits very few (if any) test customers and thus gets very little feedback. The whole goal here is to NOT uncover issues. Instead it is to push a piece of junk out of the door on a certain date, regardless of (lack of) quality, while still being able to honestly look customers in the eye and tell them your product was Beta tested. Because, yes, savvy customers ask that.
(Even savvier customers ask how big the Beta program is, how many customers participated, how long it lasted, and how many issues were uncovered and subsequently fixed.)
Result? There is no real effort to sincerely, honestly collect feedback. And any feedback uncovered will probably be ignored - because "it's too late to fix it now." Bugs will be filed, and possibly, MAYBE, they'll be addressed in the next production release.
If your Beta uncovers less than 10 issues or is less than 4 weeks long, congratulations, you had a Paper Beta.
WARNING: Running a Paper Beta will DOOM your efforts to recruit customers for any future Beta program - Paper or not. The few customers that do participate be incensed that they spent time installing, testing, and providing feedback on your software, only to have their issues summarily ignored and potentially never fixed.
3. The "Agile" Beta
Allergic to any form of process, some Agile proponents will claim their awesome methodology eliminates the need for expensive Beta programs. Why? Because, supposedly, with Agile the product is "Beta every day," or at least it is at the end of each sprint or iteration. And part and parcel of the Agile philosophy is getting direct customer feedback throughout the entire development process. Ergo, the claim that no Beta Program is needed.
All this is fine and good, but it still does not obviate the need for a Beta program. Because without getting actual product into the hands of actual customers, you won't uncover the problems that can't be uncovered in internal-only testing. It is ESSENTIAL that the customer actually attempt to use the software ON THEIR OWN, for things THEY actually want to do. If they don't and you just provide a demo or screenshots (as is often the case for Agile-style "in process" customer feedback), well TOO BAD it doesn't count.
How did you get into this sad situation? Most likely, it's because you're letting the Development organization push you around. Because, after all, if the Beta program is eliminated, it is easier to declare the release "done" and move onto the next. But without Beta testing, big issues are sometimes not caught in time and risky bug fixes get delayed to the very last minute - as in minutes before that "final build." This is extremely dangerous yet happens all too often.
4. The Haphazard Beta
The Haphazard Beta is one where you get the product to a particular state, claim it's Beta, and THEN try to figure out (last minute!) how to actually recruit some customers and get some usable feedback. There's no organization, and no directed attempt to collect and quantify feedback.
It's a mess. A big loosey-goosey, panicky mess.
And it's your fault, Product Manager. Your fault.
If you had been on the ball and actually done some recruiting ahead of time, along with a smidge of planning about how to collect and handle issues, well you might have actually had a REAL Beta. Alas, it's too late to fix it all now. So hold on tight, pray you don't lose your job, and promise to do better next time.
5. The Real Beta
This is the Beta program done right. Real customers and prospects actually use the software and identify issues (at the bug level, workflow level, and higher levels) that you could not have found on your own.
And THEN... here's the kicker.... you actually DO something about the issues they find. That is, YOU ACTUALLY MAKE CHANGES. You fix the bugs. You fix the work flows... You learn how to make your product better.
And it all goes smoothly.
You have recruited enough customers - in both number and variety - to accurately represent your target market. You have enough time to fix their issues. You care enough about putting out a quality product that the final product is something that you can be proud of - something that won't create a PR disaster requiring a zillion-dollar advertising Band-aid, like the well-known disaster below...