Rogierm's Blog

Time synchronisation in KVM, Xen or VMWare guests is a difficult subject. The best solution depends on the type and version of hypervisor and the type and version of OS that runs in the guest. This way it gets quite complicated. Each hypervisor vendor has a document on timekeeping:

• Redhat Enterprise Linux (RHEL)
• Fedora
• Ubuntu
• VMWare

While it is useful to have all a solution for all hypervisors on all types of hardware with all sorts of guest OS’s, most virtualisation shops have quite a stable and homogeneous environment. We mostly run a recent (5.4 or higher) version of CentOS as guest OS on a KVM hypervisor running on CentOS 6 running on a recent Intel Xeon platform.

This means that if:

1. The hardware has a Time Stamp Couter (TSC) $ cat /proc/cpuinfo | grep constant_tsc
2. The Guest has the kvm-clock echo /sys/devices/system/clocksource/clocksource0/current_clocksource

If the above is true, it is not recommended to use NTP in the VM Guest. Using NTP on the VM Host Server, however, is still recommended.

Summarizing: If the hostserver has TSC, and the guest is using the kvm-clock, you should only run NTP on the hypervisor.

Since the Cisco VPN client does not work under OSX Lion anymore there was no easy way to connect with certificate authentication. It took some time but I managed to get it working under Lion with the build in VPN Client. Find the steps below to get the certificates imported and working with the VPN Client.

• Create key: openssl genrsa -des3 -out vpn-cert2.key 1024
• Create CSR (make sure that the CN is a simple name, no spaces or special characters): openssl req -new -key vpn-cert2.key -out vpn-cert2.csr
• Request certificate with your CA
• Create a p12 file from the key and the certificate: openssl pkcs12 -export -inkey vpn-cert2.key -in certnew-3.cer -out vpn.p12
• Import the p12 file (containing the key and certificate) in the system keychain (not the login keychain, that doesn’t work): sudo security import vpn.p12 -k /Library/Keychains/System.keychain
• If needed you can import the CA in your keychain and trust the imported certificate: sudo security add-trusted-cert -k /Library/Keychains/System.keychain root.ca.pem
  Note:Make sure that if you import your own CA, that you do it this way. Otherwise the VPN server certificate will not be verified correctly.

To use the certificate for VPN authentication do the following:

• Open System Preferences
• Go to Network
• Click + to add network interface, Select Interface: VPN, VPN Type: Cisco IPSec
• Click Create
• In the Server Address type the hostname of the firewall. This is really important. The firewall has a certificate configured on the FQDN. Make sure the server address is the name of the certificate in the firewall. This FQDN can be found in the trustpoint configuration (see below)
• Enter the username
• Click Authentication Settings
• Select Certificate and Click Select
• Select the correct certificate that you just imported
• Click OK
• Click Apply

When you are not able to select the certificate you created the problem is that the CN is not supported. Make sure the CN that you used to create the CSR does not contain spaces or special characters.

Firewall trustpoint config:

crypto ca trustpoint CA1
enrollment terminal
fqdn fw.xxxx.com
subject-name CN=fw.xxxx.com,OU=IT,O=XXX Limited,C=NL,St=NH,L=Amsterdam


Last week I was looking at testing SplashID Enterprise. While a first installation with the MySQL database running on a Mac Mini was working fine, the installation with the Splash Enterprise Admin client failed when the database was running on a default Linux (CentOS) installation. I tried contacting SplashData support, but they could not help me, so I tried to debug myself. I enabled the query log so I could see the queries executed by the Admin client. These queries showed that SplashID ran queries reffering to specific tables in upper case (eg. MYSQL.USER). I manually tried some of these queries and these queries failed with “unknown table” error.

Now I found the problem, I thought it was easy to fix it. I just had to make MySQL case-insensitive. This sounds easier than it actually was Lots of articles talked about the character set and the collation, but these only affect the data in the tables, not the actual table name itself. Some googling let met to the lower_case_table_names setting in MySQL. It appears that Windows, Unix and MacOSX all have different default settings, and therefor behave differently.

Setting the following line in the my.cnf in the [mysqld] section solved my case problem with SplashID. MySQL now changes all table names to lower case.

lower_case_table_names=1


Update: I have not tested SplashID Enterprise yet, so I don’t know if it is any good

I’ve never really liked facebook, but also never really felt annoyed by it. But lately there are so many stories on the internet and in the news about facebooks total disregard for privacy. It might be just me noticing this, but I think not Anyway, just for fun I decided to collect the stories that show what impact changes in facebook can have.

Your-Face-Will-Soon-Be-In-Facebook-Ads
facebook-apps-kunnen-meer-privedata-gaan-opvragen
rogue-facebook-apps-access-your-home-address-mobile-phone-number
Facebook-Opens-Up-Home-Addresses-and-Phone-Numbers
facebook-app-permissions.html
Private photo’s public?
Afgeschermde facebook foto’s zichtbaar
facebook exploit macht alben nicht freunden einsehbar

Of course, there are also positive sides on facebook and what its use and users can achieve with it:
The-Inside-Story-of-How-Facebook-Responded-to-Tuni

I know facebook is a free service (as in beer), but that should not mean that they can use your data whichever way they want…

Last month I ran some performance tests over a Cisco ASA 5550 using iperf. There were some performance issues when the ASA was hit with a lot of simultaneous requests. The ASA 5550 is a powerful device so I did not expect any performance problems with 2000 concurrent requests. Our stresstests reported connection problems when the number of concurrent requests increased above 2000 while traffic was way below the maximum supported throughput. To check the wirespeed performance of the ASA I decided to run an iperf test. This test showed expected bandwidth results, but a lower MTU (1408), while all intermediate components are configured at 1500.

Some investigation showed that this was caused by a default maximum MSS setting in the ASA. It appears that the ASA has a default max MSS of 1380. This is set by the command:

sysopt connection tcp-mss MSS_size_in_bytes

The default is 1380 to prevent fragmentation on possible IPSec connections in the path.

To get to MTU 1500 the ASA needs to support an MSS of 1460. This is configured with the following command:

sysopt connection tcp-mss 1460

More information can be found here:

• http://cisco.com/en/US/products/ps6120/products_configuration_example09186a008081e621.shtml
• http://fixunix.com/firewalls/347579-cisco-asa-5510-mss-issue.html

Several blogs and manuals with examples on kvm or xen setups use NFS as storage backend. Mostly they state that for production use iSCSI is recommended. However there are examples where NFS is part of the architecture, eg. OpenNebula. I tried to find specific statistics on the performance differences between NFS, iSCSI and local storage. During this search I encountered some pointers that NFS and Xen is not a good combination, but never a straight comparison.

I decided to invest some time and setup a small test environment and run some bonnie++ statistics. This is not a scientific designed experiment, but a test to show the differences between the platforms. Two test platforms are setup, 1 with a Xen server (DL360G6) (xen1) and a 12 disk SATA storage server (storage1), and another with a KVM server (DL360G5) (kvm1) and a 2 disk SATA storage server (storage2) . Both servers are connected with a gigabit network. I’ve also run a test with a 100mb/s network between the kvm1 and storage2 server. For reference I’ve also done tests with the images on localdisk.

I realize that LVM and iSCSI storage is most efficient, but storage with image files is very convenient and in case of cloud setups sometimes the only option.

The  conclusion of the tests is that local storage is fastest. NFS storage with Xen is not a good combination. Xen runs best with iSCSI backed storage. KVM with NFS runs significantly better. It is safe to say that if you want to use NFS use it with KVM, not with Xen. In any case iSCSI is always the best option for Xen. I have not yet tested KVM with iSCSI but I expect this to perform better than NFS.

1. Open Terminal
2. Enter the following command:$ drutil eject

Libvirt is a toolkit to interact with several virtualization platform from a single interface. Considering you can stop and start virtual machines through this API, security is quite important. Libvirt offers several options to give authenticated access from remote machines. By default most distributions disable remote network access for libvirtd. However, I would like to access libvirtd on some of my KVM servers from a single management host to gather some information. The documentation on how to set this up is not too good, so I decided to write up a short how-to.

Step 1: Enable network access for libvirtd
First enable network access for libvirtd on the KVM server(s). On CentOS/RHEL this is done by uncommenting or adding the following line in /etc/sysconfig/libvirtd:

LIBVIRTD_ARGS="--listen"


Step 2: Install a CA on the management server
Install the Perl certificate tools:

yum install openssl-perl

Create Certificate authority:

cd /etc/pki/tls/misc/
./CA.pl -newca

Example output:

./CA.pl -newca
CA certificate filename (or enter to create)

Write out database with 1 new entries
Data Base Updated


Step 3: Create CSR’s

openssl genrsa -des3 -out kvm-server1.tmp
openssl rsa -in kvm-server1.tmp -out kvm-server1.key
openssl genrsa -des3 -out mgmt-host.tmp
openssl rsa -in mgmt-host.tmp -out mgmt-host.key
openssl req -new -key kvm-server1.key -out kvm-server1.csr
openssl req -new -key mgmt-host.key -out mgmt-host.csr


Step 4: Sign the certificates

openssl ca -config /etc/pki/tls/openssl.cnf -policy policy_anything -out /root/mgmt-host.crt -infiles /root/mgmt-host.csr
openssl ca -config /etc/pki/tls/openssl.cnf -policy policy_anything -out /root/kvm-server1.crt -infiles /root/kvm-server1.csr

Example output:

Using configuration from /etc/pki/tls/openssl.cnf
Enter pass phrase for /etc/pki/CA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number:
d8:95:24:4b:4e:b1:13:9c
Validity
Not Before: Feb 25 23:31:40 2010 GMT
Not After : Feb 25 23:31:40 2011 GMT
Subject:
countryName = XX
stateOrProvinceName = XX
localityName = XX
organizationName = XX
organizationalUnitName = XX
commonName = mgmt-host.xxx.nl
emailAddress = xxxxx
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
6C:EA:8B:C1:D6:XX:B6:6B:5B:18:02
X509v3 Authority Key Identifier:
keyid:C9:36:4A:XXXX:6F:FD:2E:86

Step 5: Copy over the certificates to the correct location
On the management host (mgmt-host):

mkdir /etc/pki/libvirt
mkdir /etc/pki/libvirt/private
mkdir /etc/pki/libvirt-vnc

cp /root/mgmt-host.key /etc/pki/libvirt/private/clientkey.pem
cp /root/mgmt-host.key /etc/pki/libvirt-vnc/clientkey.pem
cp /root/mgmt-host.crt /etc/pki/libvirt/clientcert.pem
cp /root/mgmt-host.crt /etc/pki/libvirt-vnc/clientcert.pem


Transfer the key and certificate files to the KVM server (kvm-server1). Ideally, you create the key and CSR on the host itself, so you only have to transfer the certificate. Then, copy the certificates and CA to the correct location on the KVM (libvirtd) server:


mkdir /etc/pki/libvirt
mkdir /etc/pki/libvirt/private
mkdir /etc/pki/libvirt-vnc

cp kvm-server1.crt /etc/pki//libvirt/servercert.pem
cp kvm-server1.crt /etc/pki/libvirt-vnc/server-cert.pem


Make sure the CA generated on the management server is available on the KVM server in the following file:
/etc/pki/CA/cacert.pem

Step 6: Reload libvirtd

/etc/init.d/libvirtd reload


Step 7: Test
With these certificates setup, you should be able to access libvirtd on kvm-server1 from mgmt-host. Use the following command to test:

virsh -c qemu://kvm-server1.xxxx.nl/system
Welcome to virsh, the virtualization interactive terminal.

virsh #

Use the list command to see a list of running guests on the server. This only works if these guests have also been created via libvirtd. Manually started KVM guests will not show up in this list.

I’ve made some quick changes to ONEMC to show the VNC port in the interface. I’ve updated the template that onemc creates with a GRAPHICS section. This enables vnc on the quest.

As a workaround until ONE can use the VMID in the graphics section, I use a virsh command to get the vncport. To get this working the webserver user should be allowed to execute the virsh command via sudo. Add the following to sudoers:

apache ALL=(ALL) NOPASSWD: /usr/bin/virsh *

Also I encountered some problems with the model section in the KVM template so I commented that out as well.

Below the patch and a screenshot listing the vnc ports in ONEMC

onemc_funcs.patch

When setting up SSL offloading on a Foundry ServerIron 4G-SSL the default installation allows weak (eg. DES, 56bit) ciphers and SSLv2. This is not a recommended setup, especially if you have to comply to certain security certifications, like PCI. The Foundry documentation does not give a lot of information on the ciphers that are supported. Below the commands to disable SSLv2 and allow only strong ciphers on an ssl accelerated host.

To change the ssl profile of a virtual server, always follow the following steps:

1. Remove the ssl profile from the virtual server
2. Change the ssl profile settings
3. Enable the ssl profile on the virtual server


server virtual vservername
no port ssl ssl-terminate sslprofilename

server virtual vservername
port ssl ssl-terminate sslprofilename