Citrix XenServer 6.5 Performance Comparison

Noticed this on the Citrix blog today. A nice little performance comparison between ESXi 5.5 and XenServer 6.5. Citrix claims to show a 11-16% performance advantage over VMware.  I’m holding off on upgrading to 6.5 till summer, but can’t wait to run some of my own benchmarks against XenServer 6.2.

Citrix XenServer 6.5 Is Here!

Xenserver 6.5 promises to provide more performance, more GPU access, and bring back Workload Balancing (WLB).  I won’t be applying 6.5 for a few months, so please let me know of any experiences you have in the comments.

Juicy details here!

http://support.citrix.com/servlet/KbServlet/download/38334-102-714582/XenServer-6.5.0-releasenotes.pdf

Changes Coming In Citrix XenServer

I was just getting caught up on some Citrix information and ran across this blog post.  I’m happy to see they are addressing the concern of XenServer being dropped as a product once it was open sourced.  And I’m glad to hear they will be moving it forward with some big changes I didn’t even know needed to happen.  Like moving dom0 to 64-bit.  When I read that, I had to jump on one of my 6.2 servers and do a quick `uname -a` to see if it was true!   And sure enough, it’s kernel is compiled for i686.

Here are the official the preview release notes Citrix gave out:
http://www.citrix.com/content/dam/citrix/en_us/documents/downloads/xenserver/xs-tech-preview-release-notes.pdf

Citrix XenServer dom0 host “random” reboot

The other day I started to get paged around 12:10AM because one of our Xenserver hosts decided it was time to reboot. We have high availability (HA) turned on, so all the VMs running on this host were rebooted on other hosts per our HA config. This was good, that means paging will stop once all the servers are running again.

But what was the cause of this reboot? Of course I go straight to the logs and this is what I found in /var/log/kern.log

Nov 15 00:05:59 xenserverhostname kernel: [2461330.653319] nfs: server 10.0.0.11 not responding, timed out

Looks like issues with NFS timing out to my storage backend, which is a NetApp FAS22xx. We’ve never see performance issue with our NetApp ever, but it looks like we have something going on now. I noticed that we had a lot of volumes scheduled to run deduplication jobs starting off at midnight. I spread those out a bit so they weren’t all trying to run at the same time. I also noticed that our HA Xenserver Heartbeat was getting dedup’d as well. I turned that off because the heartbeat only takes up a few MB’s.

I also noticed that this timeout has been logged happen before, but not enough to cause a host to reboot. I believe HA/Xen will reboot the host once it goes over a timeout threshold, and that is why the server rebooted. I think we are dealing with a couple issues and I hate to use the term “perfect storm”, but it seems fitting. I think because there were a lot of NetApp jobs kicking off at midnight, jobs with lots of I/O getting kicked off on VMs at midnight, and issues with XenServer handling timeouts were at play. I think spreading out jobs on the NetApp, on the VMs, and applying patches will help, but only time will tell if it does.

http://support.citrix.com/article/CTX135623

Configure SNMP on Citrix XenServer 6.x

I love Cacti and use it all the time to monitor my system performance.  Here are the few steps needed to turn SNMP on with in your Citrix XenServer hosts.  This document assumes you already have Performance Monitoring Enhancements Pack for XenServer, SNMP, and Cacti already installed and functioning on your network.

  1. Edit: /etc/sysconfig/iptables
    Add the following lines AFTER the line “-A RH-Firewall-1-INPUT -p udp –dport 5353 -d 224.0.0.251 -j ACCEPT”
    -A RH-Firewall-1-INPUT -p udp --dport 161 -j ACCEPT
    -A RH-Firewall-1-INPUT -p udp --dport 162 -j ACCEPT
  2. Execute: service iptables restart
  3. Edit: /etc/snmp/snmpd.conf
    Replace the community with your current SNMP community if you have one.
    # sec.name source community
    com2sec notConfigUser default public
  4. Execute: chkconfig snmpd on
  5. Execute: service snmpd restart
  6. Test from another host: snmpwalk -v 2c -c public xenserver.someplace.com
    Note, replace public with your SNMP community!

Small Changes to Increase Security on Ubuntu Servers

Here is somethings I’ve done to help increase security on my Ubuntu boxes.  The goal when securing a linux system you need to prevent, detect, and react.  These small changes will help in that goal.

Be careful with these changes, as you can lock yourself out of the server.
Also, Ubuntu can use admin(<10.x) or adm(>12.x) is the admin group!!

  1. Increase SSH security by reducing grace time, not allowing root to login (Ubuntu has no root user, but incase you are compromised and a root account is added), and only allow groups you want to login the box.  I run a shell for friends, so in order to allow them to login, I create a “ssh” group and put them into that group.
    Open /etc/ssh/sshd_config
    LoginGraceTime 20
    PermitRootLogin no
    AllowGroups adm ssh
  2. “su” program available to non-admin users
    sudo chown root:adm /bin/su
    sudo chmod 4750 /bin/su
  3. Install more apparmor profiles, read up on apparmor and make sure to think about it when troubleshooting issues.  Sometimes when you don’t use default file paths, apparmor will not allow an application to read/write to locations not whitelisted.
    sudo apt-get install apparmor-profiles
  4. Install denyhosts, this will block bots trying to brutforce you.
    sudo apt-get install denyhosts
  5. Here is an example of my changes to denyhosts
    Edit /etc/denyhosts.conf (diff -U3 denyhosts.conf.orig denyhosts.conf)
    --- denyhosts.conf.orig 2009-07-21 09:54:25.000000000 -0500
    +++ denyhosts.conf      2009-07-21 10:00:59.000000000 -0500
    @@ -57,13 +57,15 @@
    #            'y' = years
    #
    # never purge:
    -PURGE_DENY =
    +#PURGE_DENY =
    #
    # purge entries older than 1 week
    #PURGE_DENY = 1w
    #
    # purge entries older than 5 days
    #PURGE_DENY = 5d
    +# purge entries older than 4 weeks
    +PURGE_DENY = 4w
    #######################################################################
    #######################################################################
    @@ -90,9 +92,9 @@
    # eg.   sshd: 127.0.0.1  # will block sshd logins from 127.0.0.1
    #
    # To block all services for the offending host:
    -#BLOCK_SERVICE = ALL
    +BLOCK_SERVICE = ALL
    # To block only sshd:
    -BLOCK_SERVICE  = sshd
    +#BLOCK_SERVICE  = sshd
    # To only record the offending host and nothing else (if using
    # an auxilary file to list the hosts).  Refer to:
    # http://denyhosts.sourceforge.net/faq.html#aux
    @@ -218,7 +220,7 @@
    # Multiple email addresses can be delimited by a comma, eg:
    # ADMIN_EMAIL = foo@bar.com, bar@foo.com, etc@foobar.com
    #
    -ADMIN_EMAIL = root@localhost
    +#ADMIN_EMAIL = root@localhost
    #
    #######################################################################@@ -285,7 +287,7 @@
    #
    #SYSLOG_REPORT=NO
    #
    -#SYSLOG_REPORT=YES
    +SYSLOG_REPORT=YES
    #
    ######################################################################
  6. In order to whitelist a host from getting into denyhosts, list the ips in this file: /var/lib/denyhosts/allowed-hosts
  7. Make sure changes have been applied:
    sudo /etc/init.d/denyhosts restart
  8. Install performance monitor SAR
    sudo apt-get install sysstat
    Edit /etc/default/sysstat
    Set: ENABLE="true"
    sudo /etc/init.d/sysstat start
  9. Install logwatch and monitor the emails it sends you (root).  This will give you a good overview of your system if you don’t have a syslog server.
    sudo apt-get install logwatch
  10. Install Root Kit Hunter, this a cron job that will check your system for root kits.  It keeps track of your binaries and in case their MD5 changes.
    sudo apt-get install rkhunter
  11. Edit this file /etc/rkhunter.conf and add these changes to the very bottom, these may not work for you but they have been some false positives I needed to whitelist.
    MAIL-ON-WARNING=root@localhost
    ENABLE_TESTS="all"
    DISABLE_TESTS="suspscan hidden_procs deleted_files packet_cap_apps apps"ALLOWHIDDENDIR=/etc/.java
    ALLOWHIDDENDIR=/dev/.static
    ALLOWHIDDENDIR=/dev/.udev
    ALLOWHIDDENDIR=/dev/.initramfs
    ALLOWHIDDENFILE=/dev/.blkid.tab
    ALLOWHIDDENFILE=/dev/.blkid.tab.old
    SCRIPTWHITELIST=/usr/local/bin/lwp-request
  12. After installing rkhunter, you will get emails indicating if there is anything odd happening on your box.  Most of the time it’s from updates, so if you run apt-get upgrade or apt-get dist-upgrade, you need to run this command to update rkhunter:
    sudo rkhunter --propupd
  13. Shared Memory, edit /etc/fstab and add:
    tmpfs           /dev/shm        tmpfs   defaults,noexec,nosuid  0  0
  14. sudo mount -o remount /dev/shm