Lies, statistics and benchmarks
Posted by Fraser Campbell Thu, 08 Mar 2007 23:56:00 GMT
VMware published a A Performance Comparison of Hypervisors at the end of January. Since the paper only discusses Windows I didn’t bother commenting – I don’t give two hoots about Windows.
Still the story is becoming amusing enough to warrant some study. Simon Crosby, Xensource CTO, recently published his thoughts the study:
All of XenSource’s commercial products match or beat ESX performance for Windows in all but a couple of benchmarks. This for a new product, and using the HVM feature set that has never been tuned. For Linux, we absolutely thrash ESX, which should come as no surprise. We’ll publish all of our results… just as soon as we get permission from VMware, that is.
Simon’s complete commentary is available in the Xensource blog here.
I would really have to agree with Simon, there are various commercial implementations of Xen (even if they aren’t allowed to be called that) and all will undoubtedly perform better than the older open source codebase that VMware did their comparison against.
Since VMware is comparing apples to oranges it reminds me a bit of the benchmarks I did a year ago with SuSE on Xen versus SuSE on ESX. Let’s just say embarrassing is not the word, joke might be the word but it sounds rather unprofessional – paravirtualization does help whether it’s just a paravirtualized network driver as in VMware’s windows benchmarks, or if it’s a completely paravirtualized kernel as in my SuSE on Xen tests.
Let’s hope that this evil policy of banning free and open benchmarking is lifted so that reasonable public discourse can take place without threat of lawsuit.
