Citrix XenServer Error: VDI is not Available

Error: Starting VM ” – The VDI is not available

So you’re now trying to boot a VM in XenServer but you are getting the error “VDI is not Available”. This means that VM crashed, Xen Host crashed, or something just bad happen. Either way you need your server back.

  1. Find the UUID of the VDI in question.
    xe vdi-list
  2. Note exactly what UUID maps to which drive is on your server.  This is going to remove the VDI from the VM so we can reattach it correctly.  So drive order does matter, you don’t want to switch an OS VDI with a data VDI.
    xe vdi-forget uuid=<VDI UUID we found in step 1>
  3. Open XenCenter and navigate to the SR with your VDI.  Hit rescan
  4. Now goto your VM with issues and attach the VDI via the storage tab
  5. Boot your VM

Install Dell Openmanage on Citrix XenServer for Nagios checks

Like any good sysadmin, you want to know if anything is happening to your Dell hardware at any given moment.  Here is what I did to get Dell Openmanage installed in Citrix XenServer 5.6, 6.0, and 6.1.    Once openmanage is installed and working, you can then have Nagios ssh into the Xenserver host and run a check (this maybe covered in another post).

  1. I now send you on a quest.  Head to the dell website and start searching for the software.  Get something named “Dell OpenManage Server Administrator Managed Node (Distribution Specific)” or also called “OpenManage Supplemental Pack” or “OpenManage Server Administrator Managed Node” or “OM-SrvAdmin-Dell-Web-LX-7.1.0-5304.XenServer60_A00.iso” or this link?
  2. Transfer the iso to your xenserver host via scp.
  3. mount –o loop <openmanange-supplemental-pack-filename>.iso /mnt
  4. cd /mnt
  5. ./install.sh
  6. /etc/init.d/dataeng start
  7. Logout and back in and this command should work:
    omreport storage pdisk controller=0
  8. /usr/sbin/useradd nagios
  9. passwd nagios
  10. cd /home/nagios
  11. mkdir .ssh
  12. Now we need to generate or install a ssh key for Nagios to login without a password.  Here is how you would generate one:
    ssh-keygen -t dsa -b 1024 -f .ssh/id_dsa
    cat .ssh/id_dsa.pub >> .ssh/authorized_keys
  13. chown -R nagios:nagios .ssh
  14. chmod 750 .ssh
  15. chmod 640 .ssh/*
  16. mkdir bin
  17. chown -R nagios:nagios bin
  18. chmod 750 bin
  19. Get the nagios check script, this will be excuted by Nagios when it logins via ssh
    wget http://folk.uio.no/trondham/software/files/check_openmanage-3.7.3.tar.gz
  20. tar -xzvf check_openmanage-3.7.3.tar.gz
  21. cp check_openmanage-3.7.3/check_openmanage bin/
  22. If you are running Xenserver 6 or higher, you will need to run this command
    chmod o+rx /
  23. Log into your Nagios server
  24. Copy ssh id_dsa/.pub to nagios server, in nagios’s ~/.ssh
  25. Test logging in without a password
  26. Setup nagios checks (I plan posting this someone day)

Helpful links:

Small Changes to Increase Security on Ubuntu Servers

Here is somethings I’ve done to help increase security on my Ubuntu boxes.  The goal when securing a linux system you need to prevent, detect, and react.  These small changes will help in that goal.

Be careful with these changes, as you can lock yourself out of the server.
Also, Ubuntu can use admin(<10.x) or adm(>12.x) is the admin group!!

  1. Increase SSH security by reducing grace time, not allowing root to login (Ubuntu has no root user, but incase you are compromised and a root account is added), and only allow groups you want to login the box.  I run a shell for friends, so in order to allow them to login, I create a “ssh” group and put them into that group.
    Open /etc/ssh/sshd_config
    LoginGraceTime 20
    PermitRootLogin no
    AllowGroups adm ssh
  2. “su” program available to non-admin users
    sudo chown root:adm /bin/su
    sudo chmod 4750 /bin/su
  3. Install more apparmor profiles, read up on apparmor and make sure to think about it when troubleshooting issues.  Sometimes when you don’t use default file paths, apparmor will not allow an application to read/write to locations not whitelisted.
    sudo apt-get install apparmor-profiles
  4. Install denyhosts, this will block bots trying to brutforce you.
    sudo apt-get install denyhosts
  5. Here is an example of my changes to denyhosts
    Edit /etc/denyhosts.conf (diff -U3 denyhosts.conf.orig denyhosts.conf)
    --- denyhosts.conf.orig 2009-07-21 09:54:25.000000000 -0500
    +++ denyhosts.conf      2009-07-21 10:00:59.000000000 -0500
    @@ -57,13 +57,15 @@
    #            'y' = years
    #
    # never purge:
    -PURGE_DENY =
    +#PURGE_DENY =
    #
    # purge entries older than 1 week
    #PURGE_DENY = 1w
    #
    # purge entries older than 5 days
    #PURGE_DENY = 5d
    +# purge entries older than 4 weeks
    +PURGE_DENY = 4w
    #######################################################################
    #######################################################################
    @@ -90,9 +92,9 @@
    # eg.   sshd: 127.0.0.1  # will block sshd logins from 127.0.0.1
    #
    # To block all services for the offending host:
    -#BLOCK_SERVICE = ALL
    +BLOCK_SERVICE = ALL
    # To block only sshd:
    -BLOCK_SERVICE  = sshd
    +#BLOCK_SERVICE  = sshd
    # To only record the offending host and nothing else (if using
    # an auxilary file to list the hosts).  Refer to:
    # http://denyhosts.sourceforge.net/faq.html#aux
    @@ -218,7 +220,7 @@
    # Multiple email addresses can be delimited by a comma, eg:
    # ADMIN_EMAIL = foo@bar.com, bar@foo.com, etc@foobar.com
    #
    -ADMIN_EMAIL = root@localhost
    +#ADMIN_EMAIL = root@localhost
    #
    #######################################################################@@ -285,7 +287,7 @@
    #
    #SYSLOG_REPORT=NO
    #
    -#SYSLOG_REPORT=YES
    +SYSLOG_REPORT=YES
    #
    ######################################################################
  6. In order to whitelist a host from getting into denyhosts, list the ips in this file: /var/lib/denyhosts/allowed-hosts
  7. Make sure changes have been applied:
    sudo /etc/init.d/denyhosts restart
  8. Install performance monitor SAR
    sudo apt-get install sysstat
    Edit /etc/default/sysstat
    Set: ENABLE="true"
    sudo /etc/init.d/sysstat start
  9. Install logwatch and monitor the emails it sends you (root).  This will give you a good overview of your system if you don’t have a syslog server.
    sudo apt-get install logwatch
  10. Install Root Kit Hunter, this a cron job that will check your system for root kits.  It keeps track of your binaries and in case their MD5 changes.
    sudo apt-get install rkhunter
  11. Edit this file /etc/rkhunter.conf and add these changes to the very bottom, these may not work for you but they have been some false positives I needed to whitelist.
    MAIL-ON-WARNING=root@localhost
    ENABLE_TESTS="all"
    DISABLE_TESTS="suspscan hidden_procs deleted_files packet_cap_apps apps"ALLOWHIDDENDIR=/etc/.java
    ALLOWHIDDENDIR=/dev/.static
    ALLOWHIDDENDIR=/dev/.udev
    ALLOWHIDDENDIR=/dev/.initramfs
    ALLOWHIDDENFILE=/dev/.blkid.tab
    ALLOWHIDDENFILE=/dev/.blkid.tab.old
    SCRIPTWHITELIST=/usr/local/bin/lwp-request
  12. After installing rkhunter, you will get emails indicating if there is anything odd happening on your box.  Most of the time it’s from updates, so if you run apt-get upgrade or apt-get dist-upgrade, you need to run this command to update rkhunter:
    sudo rkhunter --propupd
  13. Shared Memory, edit /etc/fstab and add:
    tmpfs           /dev/shm        tmpfs   defaults,noexec,nosuid  0  0
  14. sudo mount -o remount /dev/shm

Citrix XenServer How to Force Shutdown Virtual Machines

Sometimes you just can’t get a VM to shutdown, it maybe an issue with XenTools or sun spots. Here is a list of commands that will help you get that damn thing shutdown.

  1. Disable High Availability (HA) so you don’t run into issues
  2. Log into the Xenserver host that is running your VM with issues via ssh or console via XenCenter
  3. Run the following command to list VMs and their UUIDs
    xe vm-list
  4. First you can try just the normal shutdown command with force
    xe vm-shutdown uuid=<UUID from step 3> force=true
  5. If that just hangs, use CONTROL+C to kill it off and try to reset the power state.  The force is required on this command
    xe vm-reset-powerstate  uuid=<UUID from step 3> force=true
  6. If the VM is still not shutdown, we may need to destroy the domain
  7. Run this command to get the domain id of the VM.  It is the number in the first row of output
    list_domains
  8. Now run this command using the domain ID from the output of step 7
    Before XenServer 7.x:
    /opt/xensource/debug/xenops destroy_domain -domid <DOMID from step 7>
    XenServer 7.x and greater:
    xl destroy <DOMID from step 7>