Switching a XenServer VM from HVM to PVM

 Linux, xenserver  Comments Off on Switching a XenServer VM from HVM to PVM
Feb 032018

(Disclaimer: I am by no means an expert with XenServer. So please don’t take anything you read here for granted. It’s my own experience and what I found in documentation and online.)

Switching a XenServer Linux VM from hardware assisted virtualization to paravirtualization nowadays is quite simple, since most Linux distributions already come with a Xen aware kernel (Ubuntu 16.04 defintely does, but you should check). So, most of what is described here is no longer necessary.

The very first thing to do, is this: Take a snapshot of your working HVM. So, if anything goes wrong, you can easily revert to the snapshot. If the VM is too large (e.g. you just virtualized a large file server), you can get away with temporarily detaching the data disks from the VM, take the snapshot and re-attach the data disks again.

Now, if you haven’t already done that, install the XenTools as described in point 30 and following of the article linked above:

  1. Attach the “guesttools.iso” image to the virtual DVD drive or your VM
  2. Mount it from the console (or via ssh)

    mount /dev/disk/by-label/XenServer\\x20Tools /mnt/cdrom
  3. Goto the directory named “Linux” and run the install.sh script
  4. Reboot to make sure the VM still works

Once you have verified that, get the boot parameters from the first menu entry in /boot/grub/grub.cfg

You need:

  • The kernel (or make sure that /vmlinz points to the right kernel)
  • The ramdisk (or make sure that /initrd.img points to the right ramdisk)
  • The boot parameters for the kernel, in particular the UUID of the root partition

You should verify that the UUID is correct otherwise you will end up with an unbootable system!

Shut down your VM.

Once you have all that, use either the script listed on the linked article or download my slightly modified version.

Log in to the console of the physical host on which the VM resides (I suggest using ssh) and execute the script.

It will ask you for the name of the VM to paravirtualize. If you are not sure, enter L to get a list.

After entering the name, you will be asked several questions. For most of them it should be safe to simply press enter and go with the default, but for one it isn’t:

Specify Kernel arguments (root=UUID=... ro quiet):

You must specify the kernel parameters, in particular the UUID of the root partition here. If you don’t your VM will not boot.

Once the script exits, everything is done. XenCenter should now show the VM as being in Virtualization mode “Paravirtualization (PV)”.

Boot the VM and enjoy.

What? It doesn’t boot? You get a grub error? OK, you did make a snapshot as I told you above, didn’t you? If yes, simply revert to that snapshot and try again, just in case you made a mistake. You did not make a snapshot? You’re an idiot. Yes, I mean that, I was an idiot too and I regretted it, why do you think I wrote the previous article Switching a XenServer VM from PVM back to HVM? Try the steps I list there and you might get lucky.

 Posted by on 2018-02-03 at 13:21

Clone a Linux system without imaging it

 Linux  Comments Off on Clone a Linux system without imaging it
Jan 252018

If I want to clone a Linux system (or any the boot partition/drive of any operating system), I usually use Clonezilla and make a image of the boot disk or boot partition. Unfortunately those image files can become quite large and it is a pain in the lower back to restore them on a smaller hard disk than the original one.

This becomes an issue if you want to move such a system to a virtual machine where you want to keep the size of the boot vdisk to a minimum. So, what other options are there?

In Linux almost everything is a file and there isn’t much “magic” involved in the boot process, so why not take an existing Linux VM and simply synchronize the files only? This won’t be perfect, of course since the disk device names will change so you will most likely end up with an unbootable target system. But we all know how to fix that, don’t we? Simply fix the grub configuration and edit fstab and we are done.

The command to synchronize the files over the network is rsync:

rsync -av --one-file-system --numeric-ids -X -H --acls --delete --sparse --exclude=proc --exclude=dev --exclude=var/log / IpOfTheTargetSystem:/

Where IpOfTheTargetSystem is the IP address of the target system. (And don’t forget the trailing colon and forward slash).

This must be executed as root on the source system. Make sure that ssh login as root works on the target system first, otherwise you will wonder what the problem is later.

Note that this will overwrite the /etc and /boot directories, which has the advantage of cloning your configuration but the disadvantage mentioned above: You will most likely end up with an unbootable system. Also, notice the –delete switch? It will, without asking you, delete all files on the target system that do not exist on the source system.

Also, don’t just execute any command you find on the Internet! Who knows what nefarious purpose I have by posting it here?

(Actually this is mostly for me so I can look up all the switches I had to find out for it to do what I want.)

 Posted by on 2018-01-25 at 10:09

Installing Webmin on Ubuntu 16.04 LTS (Xenial Xerus)

 Linux  Comments Off on Installing Webmin on Ubuntu 16.04 LTS (Xenial Xerus)
Jan 242018

The instructions how to install Webmin on Debian (and thereby also Ubuntu) seem a bit outdated because edits to the file /etc/apt/sources.list should be replaced by adding a file to the directory /etc/apt/sources.list.d/.

So, instead of adding

deb https://download.webmin.com/download/repository sarge contrib

to the file


create a new file


with that content and possibly a comment why you added it.

The rest seems to be up to date:

“You should also fetch and install my GPG key with which the repository is signed, with the commands:”

cd /root
wget http://www.webmin.com/jcameron-key.asc
apt-key add jcameron-key.asc

“You will now be able to install with the commands:”

apt-get update
apt-get install apt-transport-https
apt-get install webmin

“All dependencies should be resolved automatically.”

The reason why I installed Webmin was that updating a server from Ubuntu 14.04 to 16.04 broke the Webmin installation. I kept getting the error “module proc does not exist”. Google did not turn up anything useful for this so I decided to simply uninstall Webmin:

apt remove webmin
apt autoremove

And then I reinstalled it with the procedure described above. The error went away. I also got a new UI which will take a while to get used to.

 Posted by on 2018-01-24 at 11:06

Switching a XenServer VM from PVM back to HVM

 Linux, xenserver  Comments Off on Switching a XenServer VM from PVM back to HVM
Jan 232018

(Disclaimer: I am by no means an expert with XenServer. So please don’t take anything you read here for granted. It’s my own experience and what I found in documentation and online.)

If switching a XenServer Linux VM to paravirtualization fails, you usually end up with a non booting VM which is quite annoying. Switching it back to hardware assisted virtualization isn’t difficult, if you know what to do:

  1. Open a console on the XenServer host (local or via ssh)
  2. Get the UUID of the VM you want to change:
    xe vm-list name-label="NameOfTheVM"
  3. change two parameters of the VM
    1. Set HVM-boot-policy to “BIOS order”
      xe vm-param-set uuid=UuidOfTheVM HVM-boot-policy="BIOS order"
    2. Set PV-bootloader to “”
      xe vm-param-set uuid=UuidOfTheVM PV-bootloader=""

If everything works, the virtualization mode of the VM in XenCenter should be switched back to “Hardware-assisted Virtualization (HVM)” and the VM will boot again.

Source: This Citrix Forum post

Another article about this

 Posted by on 2018-01-23 at 18:27

How to shrink a Windows VM in XenServer

 xenserver  Comments Off on How to shrink a Windows VM in XenServer
Jan 082018

Apparently shrinking a Windows VM (actually any kind of VM) cannot be done with XenServer and XenCenter. You need to create an image of the original volume to a smaller virtual disk in order to do that.

On top of that, the tool that used to work fine for this, Citrix XenConvert, has been deprecated by Citrix and is no longer available for download.

And just to make matters worse, this answer on ServerFault links to an article that is no longer available (or did, until I edited it). Fortunately it’s still in the WaybackMachine

Now, where do we get XenConvert? It’s available for download here in various versions. I took version 2.5. and followed the instructions in the above mentioned article. You might want to virus check the installer before running it.

  1. Create a new virtual disk with the desired size.
  2. Attach it to your VM.
  3. Start the VM.
  4. Optionally, shrink the source drive (partition) from within the virtual machine (Windows 7+ comes with the tools for that).
  5. Format the newly created virtual disk.
  6. Install XenConvert
  7. Start XenConvert, select From: Volume and To: Volume (That’s not the default.)
  8. Select source and destination volumes.
  9. Run the conversion (which takes forever).
  10. Activate the new partition in the Disk Manager (if you forget this, you won’t be able to boot from it).
  11. Shutdown the VM.
  12. Detach the original virtual disk.
  13. Boot the VM (from the new virtual disk).

If everything works, you can now delete the original virtual disk.

 Posted by on 2018-01-08 at 18:28

Deleting a XenServer Storage Repository

 xenserver  Comments Off on Deleting a XenServer Storage Repository
Jan 042018

(Disclaimer: I am by no means an expert with XenServer. So please don’t take anything you read here for granted. It’s my own experience and what I found in documentation and online.)

In my previous post, I described how to add a Storage Repository to a XenServer using the xe command line tool.

Now, since I have installed the device driver for the RAID controller, created a RAID 5 and added it as Storage Repository (in the same way as described in the linked article), I want to get rid of the SATA drive which I had added as a temporary measure. Guess what, there seem to be no way of doing that using the XenCenter tool (even though the second link below mentions a “Detach” option I could not find it, see the disclaimer above). So, again, it’s the xe command line to the rescue.

xe help

writes a list of supported commands to the console which only contains one command with an sr- prefix: sr-list. Only if you tell it to give you a complete list, you will see what we need here, the sr-destroy command.

xe help --all
[... very long list of commands ...]
xe help sr-destroy
command name            : sr-destroy
        reqd params     : uuid
        optional params :
        description     : Destroy the SR.

But where do we get that uuid? Simple, by asking for it:

xe sr-list
uuid ( RO)                : 3f1d80c6-929a-f547-9ab1-63a2ca638cbf
          name-label ( RW): XenServer Tools
    name-description ( RW): XenServer Tools ISOs
                host ( RO): xenserver1
                type ( RO): iso
        content-type ( RO): iso
uuid ( RO)                : <UUID of SR>
          name-label ( RW): temporary-sata-disk
    name-description ( RW):
                host ( RO): xenserver1
                type ( RO): lvm
        content-type ( RO): user

As you can see, there is one Storage Repository with the label temporary-sata-disk. That’s the one I want to remove (the actual list is longer). So

xe sr-destroy uuid=<UUID of SR>

should delete that Storage Repository, right?

No, of course not, you get the error message

The SR is still connected to a host via a PBD. It cannot be destroyed or forgotten.
sr: <UUID of SR> (temporary-sata-disk)

There is also an sr-forget command but it displays the same error message.

So simple guessing doesn’t get us any further. Maybe we should read the docs? Nah, we don’t do that yet, we google and find this and also this.

One of the answers in the second link gives an extended version of the sr-list command:

xe sr-list uuid=<UUID of SR> params all
uuid ( RO)                    : <UUID of SR>
              name-label ( RW): temporary-sata-disk
      allowed-operations (SRO): unplug; plug; PBD.create; update; PBD.destroy; VDI.resize; VDI.clone; scan; VDI.snapshot; VDI.mirror; VDI.create; VDI.destroy
                    VDIs (SRO):
                    PBDs (SRO): <UUID of PBD>

This shows a lot of stuff, in particular it shows a list of allowed operations and a list of VDIs and PDBs currently connected to the Storage Repository. In the case above, we can see, that there are no VDIs connected to it and one PDB. So we follow the instructions in the first link:

xe sr-list name-label=temporary-sata-disk
uuid ( RO)                : <UUID of SR>
          name-label ( RW): temporary-sata-disk
    name-description ( RW): 
                host ( RO): xenserver1
                type ( RO): lvm
        content-type ( RO): user

which gets us the UUID of the SR (OK, we already had that one)

xe pbd-list sr-uuid=<UUID of SR>
uuid ( RO)                  : <UUID of PBD>
             host-uuid ( RO): 78059c4a-73f2-4975-936a-537529772d67
               sr-uuid ( RO): <UUID of SR>
         device-config (MRO): device: /dev/sdb
    currently-attached ( RO): true

which gets us the UUID of the PDB (we already had that one too)

xe pbd-unplug uuid=<UUID of PBD>

followed by

xe sr-forget uuid=<UUID of SR>

After that last command, the Storage Repository vanishes from XenCenter. Also

xe sr-list

no longer lists it. So I guess, it’s safe now to

  • Shut down the computer
  • Remove the hard disk that was used for that Storage Repository
  • Turn on the computer again

One last question: What is the difference between the xe commands sr-forget and sr-destroy? My google fu nearly left me here, until I found a link to XenServer Administrator’s Guide – Layer 8 Consulting (PDF!). There, on page 45 it says
Destroying or forgetting a SR
You can destroy an SR, which actually deletes the contents of the SR from the physical media. Alternatively you can forget an SR, which allows you to re-attach the SR, for example, to another XenServer host, without removing any of the SR contents.”
So, there you go.

 Posted by on 2018-01-04 at 17:20

Installing XenServer updates via XenCenter fails

 xenserver  Comments Off on Installing XenServer updates via XenCenter fails
Jan 042018

(Disclaimer: I am by no means an expert with XenServer. So please don’t take anything you read here for granted. It’s my own experience and what I found in documentation and online.)

There is at least one reason why installing updates for XenServer via XenCenter may fail (with unhelpful error messages of course): You haven’t got created any Storage Repository yet.

One might think that this is pretty unusual but your’s truly has actually managed to run into this problem. Why? Because I didn’t install the drivers necessary for accessing the hardware RAID controller (in my case the LSI megaraid package). Usually you do that during the XenServer setup but I was at that time not aware that I would actually need a driver. And thus apart from the boot disk there was nowhere to put a Storage Repository. And I didn’t want to put that on the boot ssd. So I ended up with a XenServer installation that didn’t have any Storage Repositories.

So, what can you do?

You can attach a single SATA hard disk to the system and configure it as a storage.

One might think that is easy to do using XenCenter since there is a menu entry Storage -> New SR, but no, there you can only add NVS, iSCSI, Hardware HBA or Software FCoE. Local drives aren’t even mentioned.

So you have to resort to the xe command as described here:

  1. Shut down and turn off the computer
  2. Attach the hard disk and boot
  3. On the console use fdisk to get information about the new hard disk
    fdisk -l

    Search for the new hard drive in the list. It will be the last on the list and it is indicated as /dev/sdc. c is the position it is in. Generally, it starts at a, and the list continues. You can verify the device path using the SCSI ID in /dev/disk/by-id directory by listing out the contents.
    Note: I wouldn’t rely on the drive being the last one in the list. Definitely verify the drive parameters that fdisk outputs.

  4. Run the following command from the command line interface (in the console or on the a computer with XenCenter installed):
    xe sr-create name-label=<Name of Storage> shared=false device-config:device=<Path of the Storage device> type=lvm content-type=user
    • Name of Storage is the name of the Storage Repository you require
    • Path of the Storage device is the path as noted in the preceding tasks, /dev/sdc)
      Now, the installed hard drive is visible in the XenServer Console.

    Note: This works fine if you run it on the server console. If you run it from a remote computer with XenCenter, you need additional parameters to actually connect to the server. In that case insert the following between xe and sr-create:

    -s <server> -u <username> -pw <password>
    • server is the server’s name or IP address
    • username is the user to logon (e.g. root)
    • password is the password of the given user

And yes, it worked. After adding this Storage Repository I could install the updates.

 Posted by on 2018-01-04 at 15:16

new Seagate Barracuda drives are very slow

 Linux  Comments Off on new Seagate Barracuda drives are very slow
Dec 292017

Seagate has changed their Barracuda desktop 3,5″ 2 and 4 TB drives from using 3 platters to using only 2 while doubling the cache RAM from 128 to 256 MBs.

We have been buying this type of drives for various usages, one of them is the 4 TB drive for the backup of our main file server (using Dirvish). Before the change, an ext3 format under Ubuntu Linux took a few minutes. After the change, it now takes literally(!) hours. The inode tables counter grows very quickly until it has reached about 1000, after which it starts crawling upwards veeeeeerrrrryyyy sloooooowwwwwlyyyyyy. I guess about 1000 is when the cache has been filled and any further writing goes directly to the actual drive.

Once the formatting is done, the initial backup takes several days rather than the usual 8 to 12 hours it took before the change.

Once the initial backup is done, the incremental backups only take the usual 30 minutes unless there have been very many changes, then again you can see that the drive is very slow when writing large amounts of data.

It does not matter whether the drive is connected via USB 3 (which is the way we use it for the backup) or SATA, the slowdown is reproducible every time.

Another use for these drives was the 2 TB variant to store videos in our measurement vehicles. There are 2 of these drives in one PC which are used to write MJPEG streams of 2 HD cameras each. These cameras take a picture every metre, which comes out at about 22 pictures per second when driving at 80 km/h. Before the change that was no problem at all. We could even drive at 100 km/h (27 pictures per second). After the change, the videos suddenly had lots of lost frames because the drive is just too slow to write them.

For now the 1 TB variant seems to be as fast as ever.

So now we are looking for alternative drives to use. Until a few years ago we only bought Hitachi but switched to Seagate when Western Digital acquired HGST and raised the prices. I guess we’ll have to look into WD / HGST again as well as checking out other Seagate product lines.

 Posted by on 2017-12-29 at 14:25

Find all subdirectories containing xml files on Linux command line

 Linux  Comments Off on Find all subdirectories containing xml files on Linux command line
Nov 152017

So I don’t forget:

The following Linux command shows a list of all subdirectories that contain a least one *.xml file:

find . -type f -name '*.xml' | sed -r 's|/[^/]+$||' | sort | uniq

This finds all xml files where the extension is all lower case. But since many Windows programs upper case it (and we are talking about a Samba server here), we’d better also search for *.XML:

find . -type f -iname '*.xml' | sed -r 's|/[^/]+$||' | sort | uniq

(The only difference is the “i” in -iname instead of -name.)

This is an adaptation of this answer on unix.stackexchange.com.

 Posted by on 2017-11-15 at 17:58

Rebooting a Linux server unattended

 Linux  Comments Off on Rebooting a Linux server unattended
Sep 042015

Just so I can look it up when I need it again:

/sbin/shutdown -r now

is the command that when run as root will reboot a Linux server.

It can simply be added as a scheduled command in WebMin:

 Posted by on 2015-09-04 at 09:52