Noticed this on the Citrix blog today. A nice little performance comparison between ESXi 5.5 and XenServer 6.5. Citrix claims to show a 11-16% performance advantage over VMware. I’m holding off on upgrading to 6.5 till summer, but can’t wait to run some of my own benchmarks against XenServer 6.2.
Tag: performance
Citrix XenServer 6.5 Is Here!
Xenserver 6.5 promises to provide more performance, more GPU access, and bring back Workload Balancing (WLB). I won’t be applying 6.5 for a few months, so please let me know of any experiences you have in the comments.
Juicy details here!
Changes Coming In Citrix XenServer
I was just getting caught up on some Citrix information and ran across this blog post. I’m happy to see they are addressing the concern of XenServer being dropped as a product once it was open sourced. And I’m glad to hear they will be moving it forward with some big changes I didn’t even know needed to happen. Like moving dom0 to 64-bit. When I read that, I had to jump on one of my 6.2 servers and do a quick `uname -a` to see if it was true! And sure enough, it’s kernel is compiled for i686.
Here are the official the preview release notes Citrix gave out:
Secrets of the Citrix Support Ninjas
A small but helpful list of Citrix / Xen tips. Check it out, as it may help you some day.
Citrix XenServer dom0 host “random” reboot
The other day I started to get paged around 12:10AM because one of our Xenserver hosts decided it was time to reboot. We have high availability (HA) turned on, so all the VMs running on this host were rebooted on other hosts per our HA config. This was good, that means paging will stop once all the servers are running again.
But what was the cause of this reboot? Of course I go straight to the logs and this is what I found in /var/log/kern.log
Nov 15 00:05:59 xenserverhostname kernel: [2461330.653319] nfs: server 10.0.0.11 not responding, timed out
Looks like issues with NFS timing out to my storage backend, which is a NetApp FAS22xx. We’ve never see performance issue with our NetApp ever, but it looks like we have something going on now. I noticed that we had a lot of volumes scheduled to run deduplication jobs starting off at midnight. I spread those out a bit so they weren’t all trying to run at the same time. I also noticed that our HA Xenserver Heartbeat was getting dedup’d as well. I turned that off because the heartbeat only takes up a few MB’s.
I also noticed that this timeout has been logged happen before, but not enough to cause a host to reboot. I believe HA/Xen will reboot the host once it goes over a timeout threshold, and that is why the server rebooted. I think we are dealing with a couple issues and I hate to use the term “perfect storm”, but it seems fitting. I think because there were a lot of NetApp jobs kicking off at midnight, jobs with lots of I/O getting kicked off on VMs at midnight, and issues with XenServer handling timeouts were at play. I think spreading out jobs on the NetApp, on the VMs, and applying patches will help, but only time will tell if it does.
http://support.citrix.com/article/CTX135623
Configure SNMP on Citrix XenServer 6.x
I love Cacti and use it all the time to monitor my system performance. Here are the few steps needed to turn SNMP on with in your Citrix XenServer hosts. This document assumes you already have Performance Monitoring Enhancements Pack for XenServer, SNMP, and Cacti already installed and functioning on your network.
- Edit: /etc/sysconfig/iptables
Add the following lines AFTER the line “-A RH-Firewall-1-INPUT -p udp –dport 5353 -d 224.0.0.251 -j ACCEPT”
-A RH-Firewall-1-INPUT -p udp --dport 161 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 162 -j ACCEPT - Execute: service iptables restart
- Edit: /etc/snmp/snmpd.conf
Replace the community with your current SNMP community if you have one.
# sec.name source community
com2sec notConfigUser default public - Execute: chkconfig snmpd on
- Execute: service snmpd restart
- Test from another host: snmpwalk -v 2c -c public xenserver.someplace.com
Note, replace public with your SNMP community!
Small Changes to Increase Security on Ubuntu Servers
Here is somethings I’ve done to help increase security on my Ubuntu boxes. The goal when securing a linux system you need to prevent, detect, and react. These small changes will help in that goal.
Be careful with these changes, as you can lock yourself out of the server.
Also, Ubuntu can use admin(<10.x) or adm(>12.x) is the admin group!!
- Increase SSH security by reducing grace time, not allowing root to login (Ubuntu has no root user, but incase you are compromised and a root account is added), and only allow groups you want to login the box. I run a shell for friends, so in order to allow them to login, I create a “ssh” group and put them into that group.
Open /etc/ssh/sshd_config
LoginGraceTime 20
PermitRootLogin no
AllowGroups adm ssh - “su” program available to non-admin users
sudo chown root:adm /bin/su
sudo chmod 4750 /bin/su - Install more apparmor profiles, read up on apparmor and make sure to think about it when troubleshooting issues. Sometimes when you don’t use default file paths, apparmor will not allow an application to read/write to locations not whitelisted.
sudo apt-get install apparmor-profiles - Install denyhosts, this will block bots trying to brutforce you.
sudo apt-get install denyhosts - Here is an example of my changes to denyhosts
Edit /etc/denyhosts.conf (diff -U3 denyhosts.conf.orig denyhosts.conf)
--- denyhosts.conf.orig 2009-07-21 09:54:25.000000000 -0500
+++ denyhosts.conf 2009-07-21 10:00:59.000000000 -0500
@@ -57,13 +57,15 @@
# 'y' = years
#
# never purge:
-PURGE_DENY =
+#PURGE_DENY =
#
# purge entries older than 1 week
#PURGE_DENY = 1w
#
# purge entries older than 5 days
#PURGE_DENY = 5d
+# purge entries older than 4 weeks
+PURGE_DENY = 4w
#######################################################################
@@ -90,9 +92,9 @@
# eg. sshd: 127.0.0.1 # will block sshd logins from 127.0.0.1
#
# To block all services for the offending host:
-#BLOCK_SERVICE = ALL
+BLOCK_SERVICE = ALL
# To block only sshd:
-BLOCK_SERVICE = sshd
+#BLOCK_SERVICE = sshd
# To only record the offending host and nothing else (if using
# an auxilary file to list the hosts). Refer to:
# http://denyhosts.sourceforge.net/faq.html#aux
@@ -218,7 +220,7 @@
# Multiple email addresses can be delimited by a comma, eg:
# ADMIN_EMAIL = foo@bar.com, bar@foo.com, etc@foobar.com
#
-ADMIN_EMAIL = root@localhost
+#ADMIN_EMAIL = root@localhost
#
#######################################################################@@ -285,7 +287,7 @@
#
#SYSLOG_REPORT=NO
#
-#SYSLOG_REPORT=YES
+SYSLOG_REPORT=YES
#
###################################################################### - In order to whitelist a host from getting into denyhosts, list the ips in this file: /var/lib/denyhosts/allowed-hosts
- Make sure changes have been applied:
sudo /etc/init.d/denyhosts restart - Install performance monitor SAR
sudo apt-get install sysstat
Edit /etc/default/sysstat
Set: ENABLE="true"
sudo /etc/init.d/sysstat start - Install logwatch and monitor the emails it sends you (root). This will give you a good overview of your system if you don’t have a syslog server.
sudo apt-get install logwatch - Install Root Kit Hunter, this a cron job that will check your system for root kits. It keeps track of your binaries and in case their MD5 changes.
sudo apt-get install rkhunter - Edit this file /etc/rkhunter.conf and add these changes to the very bottom, these may not work for you but they have been some false positives I needed to whitelist.
MAIL-ON-WARNING=root@localhost
ENABLE_TESTS="all"
DISABLE_TESTS="suspscan hidden_procs deleted_files packet_cap_apps apps"ALLOWHIDDENDIR=/etc/.java
ALLOWHIDDENDIR=/dev/.static
ALLOWHIDDENDIR=/dev/.udev
ALLOWHIDDENDIR=/dev/.initramfs
ALLOWHIDDENFILE=/dev/.blkid.tab
ALLOWHIDDENFILE=/dev/.blkid.tab.old
SCRIPTWHITELIST=/usr/local/bin/lwp-request - After installing rkhunter, you will get emails indicating if there is anything odd happening on your box. Most of the time it’s from updates, so if you run apt-get upgrade or apt-get dist-upgrade, you need to run this command to update rkhunter:
sudo rkhunter --propupd - Shared Memory, edit /etc/fstab and add:
tmpfs/dev/shm tmpfs defaults,noexec,nosuid 0 0 sudo mount -o remount /dev/shm