This post appears courtesy of the Ferenstein Wire, a syndicated news service. Publishing partners may edit posts. For inquiries, please email author and publisher Gregory Ferenstein.
Government agencies and watchdog groups are going after tech companies that make promises they can’t deliver to consumers. The recent flood of lawsuits and exposés is a reaction to Silicon Valley’s bad habit of unscientific propaganda.
Just in the last few months, the press has lambasted unicorn medical company, Theranos, for inaccurate blood tests, and food startup Hampton Creek Foods for questionable claims about its egg substitute. The Federal Trade Commission extracted a $2 million settlement with Lumos Labs for exaggerated conclusions of the benefits of brain games. And users are suing Fitbit for inaccurate heart-rate monitoring.
Science Versus Technology
While this may seem like a string of bad luck, it’s just the tip of the iceberg. In my experience as a professional reviewer of consumer technology, this is how much of Silicon Valley operates with respect to scientific claims.
More often than not, tech companies will make big promises to consumers, using their early adopters as a research base to verify the accuracy of their products, and then release limited conclusions if the data turns out positive.
That is, in their pursuit of short-term profit, tech companies have an incentive to release a product, advertise big benefits, and then hope the experience of an ever-expanding user base retroactively validates their conclusions.
Case in point: Soon after a Stanford report contradicted the claims of Lumos Labs—that brain-training games dramatically improve cognition—I asked the company’s executives for a response. They told me that they were confident in their brain-training product and that their users were seeing big benefits. I asked them for the data and whether they would let independent researchers verify the claims.
I was told that Lumos Labs had data scientists working on proving the claims and that the company would probably release the findings shortly. I never received the data.
Waiting For Data
I’ve seen this over and over again.
When I tested the accuracy of health trackers, I personally experienced wide variation in heart-rate monitoring and calorie burning during exercise. One of my trackers showed that I was burning 1,000 more calories per day than the others I was wearing at the exact same time. If it was accurate, I should have looked a lot more like a shirtless Hugh Jackman than I actually do.
When I confronted a company executive at a conference, he told me that some people, like myself, who do high-intensity training, can throw off the device.
In other tests, optical heart-rate monitors varied widely—by 30 to 60 beats per minute during exercise (the difference between an all-out sprint and a mild jog). Prior to the test, I had asked one company about independent scientific research and got no response. After the bad review, I was contacted by the company. I told them how I did my review and never heard back from the engineers (again).
The Numbers Don’t Ad(d) Up
It’s not just health: Advertisers are some of the worst offenders. Research scientists have long shown that making causal inferences about the effects of advertising is extremely difficult. Because click-through rates are so small, the sample sizes necessary to understand what works can reach into the millions, if not higher.
When I inquired with one major social network about how they measured return on investment, I was told that if was looking for academic-quality research, I would come up “empty-handed”. I was then referred to data scientists on their team who were conveniently unavailable.
Again (and again and again), statistical experts would tell me that they could not understand how companies could make scientific claims and the companies, when faced with this criticism, would tell me that the research was ongoing.
The Revenge Of Real Science
It appears that these days of unscientific propaganda are coming to an end. Bad press is one thing. It’s going to be far harder for companies to brush off government agencies going after them when they can’t back up quantitative claims with solid research.
In some cases, this could make life more difficult for tech companies. Proving things quantitatively can be very difficult—whether it be a heart monitor, calorie counter, brain game, or advertising campaign.
In my opinion, the best strategy is to be honest and release what data you can to the public. If scientists have doubts, give researchers secure access to test the data and do independent verification. Otherwise, users become unwitting lab rats for the efficacy of products we can’t verify.
Hiding data and exaggerating claims was always unethical. It may soon be legally dubious as well. And ultimately, it may prove to be a losing strategy. For Silicon Valley startups looking to win over consumers and regulators, that’s what really matters.
For more stories like this, subscribe to the Ferenstein Wire newsletter here.
Photo courtesy of Theranos