A job posting from Apple reveals the company is using or will use Hadoop for its iAds system. The job listing is for a “Senior Software Engineer – Hadoop” with experience in MapReduce, Hive and either HBase or Cassandra. Oozie and Flume are also mentioned. The ad was first spotted by The Register, and has since been removed from Apple’s website. However, searching through Apple’s job listing reveals other places that Hadoop may be in use, including improving the iOS experience.
The job description:
Candidate will be responsible for the following: design and build scalable Hadoop based ETL infrastructure, build modular components in the large volume data movement and management, mentor junior engineers in Hadoop based technologies, work with architects in implementing the data pipeline.
Apple launched iAds last summer. It’s not clear whether Apple’s been using Hadoop to power iAds all along or if it’s transitioning to Hadoop in the near future, running analytics on advertising interactions is exactly the sort thing Hadoop excels at.
iAds doesn’t appear to be the only place Apple is using Hadoop. Its listing for a Senior BI Analyst notes that experience with Hadoop/Pig is a place. Hadoop/Pig is also listed as a “plus” in the description for an iOS Data Analysis Engineer, who will be doing some pretty interesting data mining. From that job description:
How are iOS devices used? How can they be improved? What kinds of problems do our users experience? These are the kinds of questions the iOS Data Analysis team answers. We are looking for a passionate developer to help us build up and scale out our data processing capabilities.
If you love to code and you find working with terabyte scale data sets fascinating, we want to talk to you. With your experience in processing large datasets you will work along side other engineers to develop data processing pipelines capable of scaling with the growth of iOS. You will be investigating and using cutting edge software technologies to build a system that can evolve with the changing needs of iOS engineering. Your work will have a direct impact on the direction of iOS.
Hadoop is a database technology created by Doug Cutting and inspired by Google’s MapReduce algorithms. It’s particularly valuable for processing big data in clusters. Apple joins the ranks of many leading technology companies in using Hadoop, including: Facebook, Twitter and Yahoo. We’ve covered Hadoop frequently here, including how Twitter is using it.
Apple’s job listings demonstrate once again both how prevalent Hadoop is and the demand for individuals skilled in working with data.
