Firefly is one of the core features of the Amazon Fire phone, if you could call it that. On the Fire phone, it launches like an app, but in practice Firefly works more like Siri’s voice recognition tools or Google Now’s anticipatory data system—two other “apps” woven deeply into their respective mobile operating systems.
See also: Amazon Fire Phone’s First Mistake: Cooperating With AT&T
All three approach the same puzzle from angles reflective of their creators, and Firefly—likely built partially out of Amazon’s 2009 acquisition of the visual product-search startup SnapTell—reflects its stop-at-nothing mission to reduce the distance, time and effort between the digital world and the physical one.
I had plenty of time to play around with Firefly on Amazon’s Fire smartphone; here are the tricks and tools that stood out.
1. TV And Movie Scene Recognition
Combining its X-Ray technology with IMDB’s database and the Amazon video collection, Firefly can listen into what you’re watching—here, Game of Thrones—and identify the series, episode and even the actors in a particular scene. (This is sort of similar to a new feature in Facebook’s mobile app.) When we tried it out on the Fire phone, the feature was quick and accurate, though obviously limited to Amazon’s selection of TV and movies.
2. Books, DVDs, Art … And Most Everything Else
Naturally, Firefly lets you identify and interact with basically any object you can buy on Amazon. It’s useful for buying stuff or bookmarking it for later via the wish list, but in theory you can use Firefly to quickly pull up contextual info just for the sake of learning about things, too.
3. MyFitnessPal Nutrition Info
Sure, you can scan a Raisin Bran box and order twenty pounds of cereal to your house with next-day delivery. But Amazon is making an effort to provide other kinds of data you might want out of a smartphone phone-turned-barcode scanner. With MyFitnessPal integration, Firefly can view the calorie count and other nutritional info of food products to help track what you’ve been eating.
4. Phone Numbers, URLs And Email Addresses
Point Firefly at any text and it’ll spin up a quite quick OCR-on-steroids feature. It can extract the kind of written data we often need to digitize, like phone numbers, web and email addresses, porting them into the digital world in seconds to be clicked or called. (That’s the glowy bit in the photo below.)
5. Music And Song Lyrics
Similar to its video recognition tricks, Firefly knows the lyrics of songs in Amazon’s database and displays them in real time. With a song playing, you can trigger the Fire phone’s right panel (by tilting the phone that way) and follow along to a micro-karaoke machine. Selecting a lyric can skip forward or back.
6. Firefly History
Firefly keeps a running log of all of the stuff you’ve inquired about in the world around you. Obviously, it’s a handy way for Amazon to not-so-gently remind us to buy products we’ve lusted after, but it’s also a useful reference for tracking all kinds of media and data that you might want to recall later for reasons other than shopping ’til you drop.
7. The Firefly SDK
Firefly is the Fire phone’s new jam, but Amazon wants developers to build cool stuff for it too. For example, third party wine enthusiast app Vivino has already paired Firefly’s robust image recognition muscle with its own database. Interested developers can check out the goods here.
In Amazon’s words, Firefly lets you “instantly identify and take action on” just about any person, place or thing you might encounter out there in the real world. It’s certainly the shortest distance between seeing a thing and owning it, but Firefly does provide rich contextual data about those things too—it just so happens that Amazon stocks each and every one of them.
All photos by Taylor Hatmaker for ReadWrite
