Home The Pros & Cons of A/B Testing for Startups

The Pros & Cons of A/B Testing for Startups

One of the topics we touch on from time to time here at ReadWriteStart is the importance of solid design aesthetics for Internet startups. One of the key elements in creating a user-friendly design is A/B testing – a process by which two or more variations of a design element are tested with different groups of users. A/B testing, however, doesn’t apply strictly to the visuals of a site; it can also be a useful tool for startups hoping to learn what their users like best.

The Good: Learn About Your Users’ Habits

For designers, A/B testing is a nice way to gauge user response to small tweaks, including fonts, spacing, artwork or other visual elements. Startups can take advantage of these methods to test non-visual elements of their sites, such as pricing plans, new features, and registration forms. The ability to see feedback based on two or more variations can help a young company make the best decisions surrounding its business going forward.

Just recently, Automattic – Matt Mullenweg’s company behind WordPress – tested a pair of pricing options for a new product called VaultPress. In the service’s sign-up form, potential users are encouraged to suggest a price they would be willing to pay as compared to a suggested figure that the site is “planning to charge.” The suggested figures say $20 for some users, and $10 for others. With this test, it is likely that the company will uncover the best price point at which to sell the service in order to attract the most users.

One of the best looking tools I’ve seen for easy A/B testing of a webpage is a service called Optimizely. Users can drag-and-drop or click-and-drag changes to a site’s design and copy. After dropping a bit of code into the homepage header, they can then view real-time analysis of site usage based on the altered elements.

It’s not clear how well the service handles the testing of feature functionality on the site, as it looks geared mostly toward the look and feel of a page, but it could still be a great tool for startups. Optimizely is funded by Y Combinator and just entered private beta last week. For other A/B testing resources, consult today’s Webdesigner Depot article on the subject.

The Bad: Relying Too Heavily On Perfection

Not everyone agrees, however, that A/B testing is a wise practice. Just this morning, Jeff Atwood, author of the blog Coding Horror, expressed his idea that this kind of testing is a lot like the movie Groundhog Day. In the movie, Bill Murray – stuck in an infinite loop reliving the same day over and over again – attempts to win the love of a woman, Rita, by learning as much as he can about her each day and pretending to share all of her interests.

Atwood says that this is exactly what startups and designers are doing by relying too heavily on A/B testing. While romantic relationships are certainly different than business relationships, the parallels are interesting. Even though Murray’s character says presumably “all the rights things,” Rita can still tell there is something odd about it. Startups should be wary of testing too many of their decisions and should instead focus on just a few key elements at a time.

Rand Fishkin at SEOmoz warned a few weeks ago against falling into the “trap of A/B testing minutiae.” He says many are “tantalized” by the idea that a small change can go a long way, when in reality this is far from the norm.

“In all of these, some simple change accounted for big increases in click-through or conversion rate, leading to widespread praise and sharing,” says Fishkin. “The problem is – they’re the exception, not the rule. In fact, that’s precisely why they’re newsworthy and get so many mentions.”

Fishkin also says for the small changes A/B testing often produces, the time and energy is too high. He suggests that startups focus more on testing larger changes – like a sight redesign – rather than smaller ones – like button colors.

Like many things in business, design and life, there is no one perfect answer for all occasions. A/B may be valuable to your startup, if used correctly. Or it may not be. There certainly seems to be a place for A/B testing within startups, but don’t let it become too much of a good thing.

Image from SEOmoz.

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.