Citrix XenServer 6.1 Internal error: File “xapi_xenops.ml”, line 1788, characters 3-9: Assertion failed

Today I rebooted a xen host, everything came back up as I would expect, and after trying to live migrate VMs back onto the host I got this error message.

Internal error: File "xapi_xenops.ml", line 1788, characters 3-9: Assertion failed

I thought it was something wrong with xencenter, but then tried via command line and got this error:

Error code: SR_BACKEND_FAILURE_46
Error parameters: , The VDI is not available [opterr=VDI SOME-UUID already attached RW]

 

The message “The VDI is not available” I’ve seen many times before and I knew what to do.

 

 

Citrix XenServer PCI Passthrough

Sometimes you need to use a PCI device on a virtual machine. We have a vendor who has supplied a USB dongle for software licensing, which is really annoying. Their software doesn’t merit it’s own server, so we want to put it into our Xenserver pool.

    1. Shutdown the VM you want to attach your device too.
    2. SSH into the host that has your PCI device.  Use the following command to locate the PCI ID for your device.   Should look something like this “00:1d.7″
      lspci -v
    3. Now locate the UUID for the VM you want to attach the PCI device to with the following command
      xe vm-list
    4. Now use the following command to set the pass through
      xe vm-param-set other-config:pci=0/"PCI ID" uuid="UUID of VM"
    5. Now boot the VM and check if your device is now attached.  If not, make sure you are using the correct PCI ID.  USB devices can get tricky, as there maybe many USB devices listed.

Citrix XenServer 6.1 – XenServer Tools out of date (version 6.0 installed)

Recently we upgraded our XenServer resource pool to 6.1 because the new feature allowing you to do live migration of VDI’s sounds unreal.  I still get amazed that this stuff works, well most of the time.  Now we are having issues with XenTools reporting to be out of date in XenCenter, I see the message:

XenServer Tools out of date (version 6.0 installed)

Great, I’ll just pop the XenTools CD in, uninstall the old version, and install the new version.  Nope, that does not work.  Instead it makes the issues worst, as you try and reboot the system but XenServer never figures out the server is shutdown and you end up having to do vm-reset-powerstate or even destroy the domain.  Either way this sucks.

Citrix just updated it’s XenServer blog yesterday acknowledging the issues with XenTools is real and they will try and fix their processes.  And they are right, this sort of thing really does hurt your confidence in the system.  And we all know virtualization is built on confidence.

Now here is the real place to get all the info on what needs to happen.  It’s a two part update, first you need to apply hot fix XS61E009 and then XS61E010.  Then you have the fun of updating XenTools.  But lots of work non the less when you have 10s, 100s, 1000s of VMs.

http://support.citrix.com/article/CTX135099
http://support.citrix.com/article/CTX136252
http://support.citrix.com/article/CTX136253

*UPDATE*

If you run in to blue screen on Windows while installing the new XenTools from the XS61E010… I know more issues??  If you can get into safe mode to remove xentools from the machine and reinstall, that may work.  Other wise you can try looking yup your VMs UUID and changing the device ID.  This may work if your VM does not get the correct device ID set during the new xentools install. Use this command:

xe vm-param-set uuid=<vm_uuid> platform:device_id=0001

or

xe vm-param-set uuid=<vm_uuid> platform:device_id=0002