Storage performance difference between KVM and Xen

admin — Mon, 28 Jun 2010 21:04:31 +0000

Several blogs and manuals with examples on kvm or xen setups use NFS as storage backend. Mostly they state that for production use iSCSI is recommended. However there are examples where NFS is part of the architecture, eg. OpenNebula. I tried to find specific statistics on the performance differences between NFS, iSCSI and local storage. During this search I encountered some pointers that NFS and Xen is not a good combination, but never a straight comparison.

I decided to invest some time and setup a small test environment and run some bonnie++ statistics. This is not a scientific designed experiment, but a test to show the differences between the platforms. Two test platforms are setup, 1 with a Xen server (DL360G6) (xen1) and a 12 disk SATA storage server (storage1), and another with a KVM server (DL360G5) (kvm1) and a 2 disk SATA storage server (storage2) . Both servers are connected with a gigabit network. I’ve also run a test with a 100mb/s network between the kvm1 and storage2 server. For reference I’ve also done tests with the images on localdisk.

I realize that LVM and iSCSI storage is most efficient, but storage with image files is very convenient and in case of cloud setups sometimes the only option.

		Seq output						Seq input				Random
		Per Chr		Block		Rewrite		Per Chr		Block		Seeks
	Size	K/sec	%CP	K/sec	%CP	K/sec	%CP	K/sec	%CP	K/sec	%CP	/sec	%CP
Xen-guest-via-nfs-tapaio	1G	3570	5	2436	0	1366	0	26474	41	24831	0	6719.0	1
xen-guest-via-iscsi	1G	25242	40	12071	1	15175	0	32071	42	47742	0	7331.3	1
kvm-guest-nfs-1gb-net	1G	8140	16	17308	3	11864	2	40861	81	71711	3	2126.6	54
kvm-guest-nfs-qcow-100mb	1G	1922	3	9874	1	3994	0	10720	22	10441	0	595.4	33
kvm-guest-nfs-qcow-100mb-2nd	1G	9735	21	2039	0	3197	0	10729	22	10463	0	685.3	38
kvm-guest-nfs-qcow-100mb-3rd	1G	5327	10	7378	1	4421	0	10655	18	10512	0	706.3	39


xenserver-nfsmount	1G	41507	60	60921	7	29687	1	33427	48	64147	0	4674.4	11
kvmserver-nfs-1G	20G	31158	52	32044	17	10749	2	19152	28	18987	1	90.3	1
localdisk-on-nfs-server-cloudtest3	4G	41926	65	43805	7	18928	3	52943	72	56616	3	222.6	0

The conclusion of the tests is that local storage is fastest. NFS storage with Xen is not a good combination. Xen runs best with iSCSI backed storage. KVM with NFS runs significantly better. It is safe to say that if you want to use NFS use it with KVM, not with Xen. In any case iSCSI is always the best option for Xen. I have not yet tested KVM with iSCSI but I expect this to perform better than NFS.

OpenNebula EC2 error

admin — Sun, 20 Sep 2009 21:18:42 +0000

While experimenting with OpenNebula and trying to build a public cloud with the EC2 interface to OpenNebula I encountered the following problem in the code:
[rogierm@cloudtest3 one]$ econe-upload /home/rogierm/test.img /usr/lib/ruby/1.8/rdoc/ri/ri_options.rb:53: uninitialized constant RI::Paths (NameError) from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' from /usr/lib/ruby/1.8/rdoc/usage.rb:72 from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require' from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require' from /usr/local/one/bin/econe-upload:61

I fixed this problem by adding the following line (above the other require statements) in econe-upload, or any other command giving the same error:
require 'rdoc/ri/ri_paths'

Rogierm's Blog » cloud

Storage performance difference between KVM and Xen

OpenNebula EC2 error