higher on benchmarks from Compuware CloudSleuth can be found here. And CloudHarmony's post can be found here. I will wait for you to go take a look and return to give you my own thoughts.
I have been performing my own benchmark tests for a long time, as well as reporting on others. Back before the cloud was called that. Back before the Internet was available to commercial organizations. Back to when Ethernet wasn't the only choice for networks. So I write this with a heavy heart, as someone who supposedly should know what he is doing. The problem is that anyone can test anything and, especially these days, publish it too. Good benchmarking is hard, for precisely the reasons that CH states. You have to know what you are comparing, and understand your results.
What I liked about the CH post is that they are very transparent about their process. They try to show you what underlies their thinking, and then dispassionately go through the points. They ask questions such as:
- What is being measured?
- What is being claimed?
- Is it a fair fight?
- How accurate are the reports and citations?
- What are the bottom-line implications for actual users?
These are all questions that benchmark consumers need to ask, and understand the answers. Think of what Consumer Reports does: they are very transparent about what they test, how they test, and how they interpret the results.
The problem is that testing cars and clouds are two different kinds of tests. While a car is made up of complex physical systems - engine, transmission, passenger amenities and overall body construction - the cloud is made up of a variety of Web and Internet protocols and applications that aren't so neatly packaged or their inner workings as well understood. In a car, we know in general that increasing horsepower and curb weight will decrease gas mileage. We don't have these generally understood metrics in the cloud. (Which is why reporting on MPGs from hybrid cars is an issue.)
All VMs - and indeed all network connections - are not created equal. Indeed, I can have an overloaded hypervisor running some heavy VMs on one provider (hopefully that isn't the case, but still) and compare it to a lightly-loaded hypervisor running smaller instances. I can put a bunch of crappy routers between me and the cloud data center or I can get on a Metro Ethernet link and move my packets across the Internet faster than the speed of light. Or so it might seem. You get my drift.
On the CH post, they look at two situations: one in which their tests were perverted by Joyent and one in which they poke holes in what their testing competitor ClouthSleuth did with a series of latency tests. The latter tests was what I wrote about in October.
Perhaps CloudHarmony has a point: this wasn't an apples-to-apples comparison. Only average latency was measured, and from 30 nodes around the world, a small sample at best. CH has a collection of 1000 nodes around the world to conduct their own latency tests, and they collect standard deviation and other stats that show you the range of latencies involved. Good for them. Only, when I ran their speed test (as you see in the above screen shot), each of the dozens of cloud services was tested by three or four or maybe six different nodes. So that 30 node test doesn't seem so off-base at all.
And yes, there is a difference between IaaS and PaaS players, and Azure coming out on top isn't really a fair fight. But still, this gives you a rough idea of what that two-page test Web app will do in the different environments.
Here is what I take away from the results, after re-reading everything: the cloud is a complex place. What I liked about ClouthSleuth was the time-series information, and no one has spent a year looking at what latencies are, even under these limited circumstances. Which is why I originally posted the article here. Certainly, you are not going to deploy a two-page toy website on expensive cloud providers. But I liked being able to see the entire year's worth of information, even if it is obscured by a simple arithmetical mean. And kudos to ClouthHarmony for putting so much data out in the public view too. We need both approaches, and the more information that you, the actual implementer, can get your hands on, the better.
So, CloudHarmony should be praised for what they are doing. They have a more recent test that is worthy of further exploration: comparing video codecs from a series of suppliers. And yes, Zencoder paid for the tests, and yes, Zencoder came out on top. Hmm.
Just sayin'. I will let you be the judge. Would I have heard about this test if Zencoder wasn't first? Probably not. But I am glad I know more about CloudHarmony's testing series.
At the end of the day, it is up to those of us who report on this data to do a better job to explain the context. And I will try to do better in the future.