Amazon Web Services just reported jaw-dropping growth in the number of objects stored in Amazon S3 year over year.
“As of the end of 2011, there are 762 billion (762,000,000,000) objects in Amazon S3. We process over 500,000 requests per second for these objects at peak times,” AWS Evangelist Jeff Bar wrote on the company’s blog tonight. The company reported 262 billion objects in storage in Q4 of 2010. “This represents year-over-year growth of 192%; S3 grew faster last year than it did in any year since it launched in 2006.” Independent analysts say this is indicative of the growth of the cloud in general and of Amazon’s striking dominance of the market.
“Stunning, isn’t it?” Randy Bias, co-founder of
said to me about the news by email. “From 150% to almost 200% growth. That’s crazy. 500,000 requests per second at peak. Blows my mind.”
Bias says these are the big take-aways.
“S3 growth is accelerating, not just increasing. If other AWS services are accelerating similarly then we will see a major shift this year in AWS usage and likely revenue reporting in SEC filings.
“This is the largest storage system in the world bar none; there isn’t anything like it anywhere else that I’m aware of unless it’s some secret government/NSA vault.
“Check my math, but at 1Kbyte average per object, that would be 780PB of disk storage:
– 762,000,000,000 * 1024 (traditional KB)
– 780288000000000 / 1000 (KB for disk) / 1000 (MB for disk) / 1000 (GB for disk) / 1000 (PB for disk) [ disk capacity is in even 1,000 increments, not multiples of 2 ]
– That’s 780PB, but unclear if that’s replicated or unreplicated; probably replicated, which means 260PB of data with 3x replication.
– Average of 1Kbyte is probably too low.
– At 100TB per storage system that is 7,222 storage *servers*, each with 36 spindles at 3TB each; that might not be their configuration, but even if it’s 2 or 3 times as dense, that is a *lot* of storage servers.
– At those numbers, it’s a 26M/month business and a 300M/year run rate, which means it’s still roughly 30% of AWS revenue with EC2 being most of the rest.
“I don’t understand how people can’t see this kind of thing and just have their jaw hit the floor. People are paying for this. At this rate they will have 2 TRILLION objects in another year and it will be a $600M/year business.”
What’s behind such numbers? Widespread technology change.
“What we are seeing is the geometric explosion of cloud growth from multiple points,” Constellation Research analyst Ray Wang told ReadWriteWeb.
“First, broad based adoption driven by consumerization of IT. Second, the shift from transaction to engagement – we have social, mobile, analytical, and other unstructured data. Third, true elasticity has come to fruition as the promise of the cloud gets delivered. People are taking to the cloud because the tools are easy to use and they don’t have time or money to provision expensive servers. Instead they are using elasticity, which was the original premise of AWS. We could see it happening last year but this leap in growth is tremendous.”
Dave Linthicum, CTO and Founder of Blue Mountain Labs, says Amazon’s dominance is clear. “The rapid growth of AWS S3 is pretty much in-line with what I’m seeing in enterprises adopting cloud computing. The reality is that they are the 800 pound gorilla, and continue to gain weight. Unless they do something stupid, they are the storage provider to beat.”
Ray Wang concurs. “There are only a few companies in the world who can compete with Amazon,” he told me by IM tonight.
“It has established itself as one of the leading contenders. The barriers of entry are high. Very few folks can afford to build the data centers, the software infrastructure, and momentum to be profitable. Amazon is in the same league as Google, Microsoft, IBM, etc. The only other folks that could do it if they woke up are the telco’s – but we’ve all been telling them that for years. They haven’t paid attention.”
Amazon’s Barr explains the growth thusly. “Although we definitely made it easier for you to delete objects using Multi-Object Deletion and Object Expiration, we also gave you plenty of ways to upload new objects using Multipart upload, AWS Direct Connect, and AWS Import/Export,” he wrote in his blog post. He concluded by noting that running a system so complex is hard work and pointed to open jobs at AWS.