Archive for the 'Security' Category

How to access libvirtd from remote server

Published on March 2, 2010 in Linux, Security and Virtualization. 0 Comments Tags: , , , , , .

Libvirt is a toolkit to interact with several virtualization platform from a single interface. Considering you can stop and start virtual machines through this API, security is quite important. Libvirt offers several options to give authenticated access from remote machines. By default most distributions disable remote network access for libvirtd. However, I would like to access libvirtd on some of my KVM servers from a single management host to gather some information. The documentation on how to set this up is not too good, so I decided to write up a short how-to.

Step 1: Enable network access for libvirtd
First enable network access for libvirtd on the KVM server(s). On CentOS/RHEL this is done by uncommenting or adding the following line in /etc/sysconfig/libvirtd:

LIBVIRTD_ARGS="--listen"

Step 2: Install a CA on the management server
Install the Perl certificate tools:

yum install openssl-perl

Create Certificate authority:

cd /etc/pki/tls/misc/
./CA.pl -newca

Example output:

./CA.pl -newca
CA certificate filename (or enter to create)

Making CA certificate ...
Generating a 1024 bit RSA private key
..........++++++
.............++++++
writing new private key to '../../CA/private/cakey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:XX
State or Province Name (full name) [Berkshire]:XX
Locality Name (eg, city) [Newbury]:XXXXX
Organization Name (eg, company) [My Company Ltd]:XXXXX
Organizational Unit Name (eg, section) []:XXXX
Common Name (eg, your name or your server's hostname) []:CA XXX XXX
Email Address []:XXX

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Using configuration from /etc/pki/tls/openssl.cnf
Enter pass phrase for ../../CA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number:
d8:95:24:xx:xx:xx:13:9b
Validity
Not Before: Feb 25 23:14:08 2010 GMT
Not After : Feb 24 23:14:08 2013 GMT
Subject:
countryName = XX
stateOrProvinceName = XX
organizationName = XXXX
organizationalUnitName = XXXX
commonName = CA XXX XXX
emailAddress = XXXXX
X509v3 extensions:
X509v3 Subject Key Identifier:
XXX
X509v3 Authority Key Identifier:
keyid:XXXX
DirName:/C=XX/ST=XX/O=XXX/OU=XXXX/CN=CA XXX XXX/emailAddress=XXX
serial:XXX

X509v3 Basic Constraints:
CA:TRUE
Certificate is to be certified until Feb 24 23:14:08 2013 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated

Step 3: Create CSR’s

openssl genrsa -des3 -out kvm-server1.tmp
openssl rsa -in kvm-server1.tmp -out kvm-server1.key
openssl genrsa -des3 -out mgmt-host.tmp
openssl rsa -in mgmt-host.tmp -out mgmt-host.key
openssl req -new -key kvm-server1.key -out kvm-server1.csr
openssl req -new -key mgmt-host.key -out mgmt-host.csr

Step 4: Sign the certificates

openssl ca -config /etc/pki/tls/openssl.cnf -policy policy_anything -out /root/mgmt-host.crt -infiles /root/mgmt-host.csr
openssl ca -config /etc/pki/tls/openssl.cnf -policy policy_anything -out /root/kvm-server1.crt -infiles /root/kvm-server1.csr

Example output:

Using configuration from /etc/pki/tls/openssl.cnf
Enter pass phrase for /etc/pki/CA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number:
d8:95:24:4b:4e:b1:13:9c
Validity
Not Before: Feb 25 23:31:40 2010 GMT
Not After : Feb 25 23:31:40 2011 GMT
Subject:
countryName = XX
stateOrProvinceName = XX
localityName = XX
organizationName = XX
organizationalUnitName = XX
commonName = mgmt-host.xxx.nl
emailAddress = xxxxx
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
6C:EA:8B:C1:D6:XX:B6:6B:5B:18:02
X509v3 Authority Key Identifier:
keyid:C9:36:4A:XXXX:6F:FD:2E:86

Certificate is to be certified until Feb 25 23:31:40 2011 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

Step 5: Copy over the certificates to the correct location
On the management host (mgmt-host):

mkdir /etc/pki/libvirt
mkdir /etc/pki/libvirt/private
mkdir /etc/pki/libvirt-vnc

cp /root/mgmt-host.key /etc/pki/libvirt/private/clientkey.pem
cp /root/mgmt-host.key /etc/pki/libvirt-vnc/clientkey.pem
cp /root/mgmt-host.crt /etc/pki/libvirt/clientcert.pem
cp /root/mgmt-host.crt /etc/pki/libvirt-vnc/clientcert.pem

Transfer the key and certificate files to the KVM server (kvm-server1). Ideally, you create the key and CSR on the host itself, so you only have to transfer the certificate. Then, copy the certificates and CA to the correct location on the KVM (libvirtd) server:


mkdir /etc/pki/libvirt
mkdir /etc/pki/libvirt/private
mkdir /etc/pki/libvirt-vnc

cp kvm-server1.key /etc/pki/libvirt/private/serverkey.pem
cp kvm-server1.key /etc/pki/libvirt-vnc/server-key.pem

cp kvm-server1.crt /etc/pki//libvirt/servercert.pem
cp kvm-server1.crt /etc/pki/libvirt-vnc/server-cert.pem

Make sure the CA generated on the management server is available on the KVM server in the following file:
/etc/pki/CA/cacert.pem

Step 6: Reload libvirtd

/etc/init.d/libvirtd reload

Step 7: Test
With these certificates setup, you should be able to access libvirtd on kvm-server1 from mgmt-host. Use the following command to test:

virsh -c qemu://kvm-server1.xxxx.nl/system
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands
'quit' to quit

virsh #

Use the list command to see a list of running guests on the server. This only works if these guests have also been created via libvirtd. Manually started KVM guests will not show up in this list.

Cisco ASA and tacacs enable fails

Published on January 14, 2010 in Network and Security. 0 Comments Tags: , , .

While migrating the authentication of our ASA firewalls to tacacs, we enabled ‘enable’ authentication to tacacs and tried to switch to enable mode on the console. This did not work, and caused the following message in the tacacs log file:

Wed Jan 13 17:07:42 2010 [25444]: enable query for 'username' 13 from 10.x.x.x rejected

To fix this problem the tacacs configuration for the user needs to include the enable password in the profile, as shown below:

user = username {
login = des "XXXXXXX"
member = admin
acl = mgmt_devices
service = shell {
priv-lvl = 15
}
enable = des "XXXXXXX"
}

We use the following configuration on the ASA to enable AAA to tacacs.

aaa-server tacacs protocol tacacs+
aaa-server tacacs (outside) host 1.1.1.1
key TACACSKEY
aaa-server tacacs (outside) host 2.2.2.2
key TACACSKEY
aaa authentication ssh console tacacs LOCAL
aaa authentication telnet console tacacs LOCAL
aaa authentication serial console tacacs LOCAL
aaa authentication enable console tacacs LOCAL
aaa authentication http console tacacs LOCAL
aaa authorization command tacacs LOCAL

Cisco switch and Tacacs

Published on January 14, 2010 in Network and Security. 0 Comments Tags: , , , , , .

Tacacs is a great way to centralize user authentication, authorization and accounting. While tacacs originally is a Cisco thing, there is an open source server version available, tac_plus (http://www.gazi.edu.tr/tacacs/index.php?page=download). Installing the tacacs server is quite straight forward. Configuring the switch is not difficult either, as long as you think about possible failures. You don’t want to be locked out of your switches when your tacacs server is not available. I use the following configuration that uses two tacacs servers and asks for the enable password when neither of the tacacs servers is available. To enter ‘enable’ mode, the configured enable password suffices. Use the following Cisco configuration for a save AAA authentication.

NOTE: Always be careful when changing authentication and authorization configuration, as this might lock you out of the device. The savest way is to do this on the console of the machine.


aaa new-model
aaa authentication login default group tacacs+ enable
aaa authentication enable default enable
aaa authorization exec default group tacacs+ if-authenticated
aaa authorization commands 15 default group tacacs+ if-authenticated
aaa authorization network default group tacacs+ if-authenticated
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
aaa accounting system default start-stop group tacacs+
aaa session-id common
tacacs-server host 1.1.1.1 single-connection
tacacs-server host 2.2.2.2 single-connection
tacacs-server key TACACSKEY
tacacs-server directed-request

To restrict access to specific devices, you can configure an ACL in the tacacs configuration on the server (tac_plus.conf). See the example below.


user = username {
login = des "XXXX"
member = admin
acl = mgmt_devices
service = shell {
priv-lvl = 15
}
}
group = admin {
default service = permit
service = exec {
priv-lvl = 15
}
}
# acl's

acl = mgmt_devices {
permit = 12.12.12.12
permit = 13.13.13.13
}

Ten-thousands credit cards blocked

Published on October 21, 2009 in Security. 0 Comments Tags: , , .

Ten-thousands Belgian credit cards were blocked because hackers copied cardnumbers. This was published by Febelfin (the Federation of the Belgian finance sector) on tuesday October 20th 2009. It was stated that the criminals copied the numbers after they gained access to computer files. The hackers did not succeed in retrieving the pincodes.

I find it very strange that hackers were able to retrieve creditcard numbers from computer files. Specifically since PCI puts great restrictions on storing this stuff:

  1. PCI 3.1 – Keep cardholder data storage to a minimum
  2. PCI 3.2 – Do not store sensitive authentication data after authorization
  3. PCI 3.4 – Render PAN (PAN == cc-number), at minimum unreadable anywhere it is stored by using one of the following: one-way hash, truncation, strong cryptography with associated key-management processes and procedures
  4. PCI 3.4.1 – If dis encryption is used rather then file or database encryption, logical access must be managed independently of native OS access control

There are other requirements but the above give a good idea of the caution you must take when deciding to store this data. The full list of requirements can be downloaded from https://www.pcisecuritystandards.org/security_standards/pci_dss_download.html

Personally, I think these news items should contain more information, or should be evaluated somewhere so people can learn from the mistakes made by others. By being open on the causes of these problems people can easily prevent the holes that others left open. I know, this will probably not happen any time soon… :)

Link to original story: http://www.elsevier.nl/web/Nieuws/Internet-Gadgets/248802/Duizenden-creditcards-geblokkeerd-na-aanval-hackers.htm#

iSCSI security, how to secure your LUN?

Published on August 13, 2009 in Network, Security and Storage. 0 Comments Tags: , , , .

A SAN is often implemented as a dedicated network that is considered to be a secure network. However, the nature of a SAN is that it is a shared network. This involves some serious security risks, that should be evaluated when using an iSCSI based SAN. Some vendors consider an iSCSI network save when it is implemented as a dedicated switches network (Dell EqualLogic. Securing storage area networks with iSCSI. EqualLogic Inc., 2008.). They consider it virtually impossible to snoop or inject packets in a switched network. We all know this is not the case. If this is true, why do we use firewalls, ids and tons of other security measures? Even if iSCSI runs on an isolated network, and only the management interface of the storage devices are connected to a shared/general-purpose network, security is just as good as the hosts that are connected to the dedicated network. A single compromised host connected to the dedicated iSCSI network can attack the storage devices to get access to LUNs for other hosts.

When implementing an iSCSI network you should be aware of the security risks that this imposes on the environment. To estimate the risk, awareness of the methods that can be used to secure iSCSI is paramount. The iSCSI protocol allows for the following security measures to prevent unintended or unauthorized access to storage resources:

  • Authorization
  • Authentication
  • Encryption

Because iSCSI setups are generally shared environments access to the storage elements (LUNs) by unauthorized initiators should be blocked. Authorization is implemented by means of the iQN. The iQN is the initiator node name (iSCSI Qualified Name), this can be seen as a mac-address. During an audit, storage systems must demonstrate controls to ensure that a server under one regime cannot access the storage assets of a server under another.
Typically, iSCSI storage arrays explicitly map initiators to specific target LUNs; an initiator authenticates not to the storage array, but to the specific storage asset it intends to use.

As an added security method, the iSCSI protocol allows initiators and targets to use CHAP to authenticate each other. This prevents simple access by spoofing the iQN. And last, because iSCSI runs on IP, IPSec can be used to secure and encrypt the data flowing between the client (initiator) and the storage server (target).

Now that we know there are multiple ways to secure access to the storage resouces, you might conclude that iSCSI must be safe and secure to use. Unfortunately this is not evident. There are several flaws in the iSCSI security design:

  • iQN’s are trusted, but are easy to spoof, sniff and guessed
  • iSCSI authorization is the only required security method, and this uses only the iQN
  • Authentication is disabled by default
  • Authentication is (mostly) only implemented as CHAP
  • IPSec is difficult to implement

Because iQN’s are manually configured in the iSCSI driver on the client, it is easy to change them. To get access to a LUN that is only protected by a iQN restriction, you can sniff the communication to get the iQN, or guess the iQN as it is often a default string (eg.: iqn.1991-05.com.microsoft.hostname), configure the iscsi driver to use this name and get access to the LUN.

The CHAP protocol is basically the only authentication mechanism that is supported by iSCSI vendors. The protocol allows for other mechanisms like Kerberos. The CHAP protocol is not a protocol know for its strong security on shared networks. The CHAP protocol is vulnerable to dictionary attacks, spoofing, or reflection attacks. Because the security issues with CHAP are well known, the RFC even mentions ways to deal with the limitations of CHAP (http://tools.ietf.org/html/rfc3720#section-8.2.1).

While IPSec could stop or reduce most of the security issues outlined above, it is hard to implement and manage. Therefor not many administrators will feel the need to use it. It should not only be possible to make a secure network, it should also be made easy.

To reduce the risk, and make your iSCSI network as safe as possible, you should do the following:

  • Enable mutual (incoming/outgoing) authentication
  • Follow advice to secure CHAP
  • Enable CRC checksums
  • Do not only rely on iQN for authorization
  • Enable IPSec (if performance allows it)

Also vendors/distributors should enable authentication by default, and add other authentication mechanisms to the iSCSI target and initiator software.

References:
http://www.blackhat.com/presentations/bh-usa-05/bh-us-05-Dwivedi-update.pdf
http://en.wikipedia.org/wiki/ISCSI#Authentication
http://weird-hobbes.nl/reports/iSCSI%20security/

disable password aging

Published on August 4, 2009 in Linux and Security. 0 Comments Tags: , , , , .

In general it is a good idea to configure password aging as part of your password/security policy. In some cases however, this might cause unexpected problems. I’ve seen cases where an expired password prevented a machine from booting. In this specific case this was caused by a service that ran as the user with the expired password. In general you should not run services as a normal user account, but sometimes you just have to deal with things you can’t change. Generally the documentation states that to disable password aging you have to edit the /etc/shadow file, and remove the part where the password age is stored. This is quite error prone. If you do it this way, be sure to use vipw to prevent errors in this critical file. To disable password aging I recommend just using the command to enable it as well:

# chage -m 0 -M 99999 -E -1 username

Check the before and after:

# chage -l username
Minimum: 7
Maximum: 90
Warning: 7
Inactive: -1
Last Change: Jun 26, 2009
Password Expires: Sep 24, 2009
Password Inactive: Never
Account Expires: Never

After disabling password aging:

# chage -l username
Minimum: 0
Maximum: 99999
Warning: 7
Inactive: -1
Last Change: Jun 26, 2009
Password Expires: Never
Password Inactive: Never
Account Expires: Never

As a note, please only disable password aging when there is no other way to fix the problem.

Creditcard data stolen at Network Solutions

Published on July 28, 2009 in Security. 0 Comments Tags: , , , .

Yesterday I read an article in the Washington Post about a big security breach at Network Solutions where >500.000 credit and debit cards are stolen. Network Solutions acknowledge this security incident on their site. They claim the cause of this incident was malicious code that was uploaded to a platform supporting their merchants sites. How this was possible and how this could lead to the possible theft of half a million credit card numbers is unclear.

I’ve been involved in several PCI projects to help our customers become PCI compliant as required by the credit card issuing companies. While the need for most of the procedures and measures required by PCI is clear, some seem useless, costly and/or superfluous to implement. But after incidents like these, it is a lot easier to explain customers the point of the measures PCI requires of companies handling credit card data. Basically everything is aimed at protecting the creditcard data, and making sure in case of a security incident all needed audit trails are available to investigate the cause and source of the attack. At least that is how I look at it.

For instance, PCI requires you to audit the integrity of the files present on the system. This should not only include OS files, but also the (web) application code. It is premature to speculate if this ‘malicious code’ could have been detected by running a properly configured host-based IDS on the platform, such as Tripwire or Samhain.

Secondly, PCI requires you to establish roles that have access to a production platform to upload code. Staff members should be part of a role that authorizes them for the access they need to do their job. Access should be restricted to allow only this traffic. Not only should access be locked, but audit trails of all activity should be available upon request. These audit trails not only include who logged in to the systems, but should also include network IDS logs (eg. SNORT), commands that are executed (eg. sudo), output of these commands (eg. rootsh) and the reports of host-based IDS’s. Together these tools should give an auditor a good insight in the activity on a (compromised) server. Also SElinux could be a big help in restricting access. Strangely PCI does not require or advise the use of SElinux, while it does require the use of application level firewalls (eg. mod_security in Apache). But this is a different discussion :)

To secure the creditcard data itself, PCI requires that all creditcards should be stored in encrypted from. Manual access to keys to decrypt this data should not be possible. Based on this you can infer that the creditcard data is compromised by some sort of ‘man-in-the-middle’ attack. The malicious code could have intercepted the data after it was decrypted in the webserver, leaving the SSL tunnel, and before it was encrypted and stored in the database. But this is just speculating of course…

I’m looking forward to more details on this incident. I hope this is made publicly available so we can learn from the mistakes that were made.