When app quality analytics service Applause was launched at the end of January, it caused a bit of a firestorm when it used an algorithmic analysis of inherently subjective data to quantify iOS and Android app quality. And you know what it revealed? People think that iOS apps are better than Android.
That assertion alone was enough to send the local chapter of the iPhone or Android Fan Boy Clubs into a fit of glee or burbling rage, depending on the allegiance.
The company behind Applause, Boston-based uTest, took the reaction to heart. As a response, Applause is launching the Applause Index, a daily look at its rankings for 60 of the top apps that can be found on both iOS and Android.
When ReadWrite had Applause run the data on all iOS and Android apps in January, one argument (among many) was that the sample apps we had Applause specifically dig into were not representative of the entire body of work of each app store. So, Applause disassembled the sample for the Applause Index, making sure that there was basically 100% overlap between iOS and Android for the indices. Applause also broke apps down in to sub-categories so people could take a closer look at how top apps in categories like Productivity, Games, Entertainment, Travel, Sports, Content and so on compare.
No Winning When It Comes To Subjectivity
Numbers like these are usually controversial. Reports that Android will hit the arbitrary number of 1 million apps first in Google Play usually draw comments of “non-story” asking for data on the quality of those apps. When data is made available to do something that is akin to impossible – quantifying inherently subjective data – it is torn apart.
The fact of the matter is that as long as we are pitting iOS against Android in any type of race, war, comparison, etc. somebody is always going to be unhappy.
Outside of just handing out the algorithm, Applause has been very forthcoming with its methods. With the new daily Applause Index, the company is also being liberal with its data that show what is essentially a crowd-sourced average score of perception, fine tuned through an algorithmic approach and spread across more than a million apps.
Critics of Applause do have some valid concerns. Foremost, user ratings are not cross-checked between iOS and Android versions (it is not “double blind”). If we really want to get a good view of comparative app ratings, we would make iOS users rate Android apps and Android users rate iOS apps without telling them which is which. The problem with that approach is that it is extraordinarily cumbersome to do at scale and relies on users opting in to a double-blind study.
Another problem falls into the vague category of sociological phenomena. Mainly, there is a perception that iPhone and iPad users are more loyal to their platform and hence more likely to rate apps higher. That is an opaque assertion and very difficult to quantify with actual data, outside of widely distributed surveys, which are in and of themselves also inherently subjective.
From the Android perspective, many argue that Android app ratings would be inherently lower than iOS because of the fragmentation of the Android ecosystem. For example, if one user has an HTC Desire running Android 2.3 Gingerbread and another user has a Samsung Galaxy Note II running Android 4.1 Jelly Bean, are they really going to rate an app the same way? The Note II user is likely to rank an app higher because the phone is newer, has better internal components and a better version of the operating system. Applause cannot, as yet, differentiate those two reviews and still provide an accurate score. The fact of the matter is that some apps just do not work as well on some.
If Users & Devices Are Not The Same, Make Sure The Apps Are
The criteria for the Applause Index includes:uTest chose the apps for the Applause Index very carefully. It applied a criteria to the top 60 apps chose for the index to make sure they are as similar as possible.
- Number of app versions and app reviews
- Length of time app has been available
- Performance data across Applause Attributes
- Representative allocation across sub-indices
- 100% overlap between iOS and Android
Using those guidelines, uTest picked a group of apps that are a decent snapshot of each app category. Applause has crawled a total of 1,177,456 apps between iOS and Android, analyzing more than 60 million reviews and star ratings between them.
Here is a snapshot of today’s rankings between iOS and Android apps in the Content, News & Books category.
If nothing else, the Applause Index is an enlightening and entertaining way to judge app quality from a quantifiable perspective. Is Applause a perfect system? No. What it does do is provide a certain criteria of data that allows people – brands, analysts, app makers, journalists, etc. – a decent idea of what people think of a certain app.
