[Systems] moving VMs on housetree

[Bernie Innocenti]

El mié, 09-12-2009 a las 19:01 -0500, Bernie Innocenti escribió:
> This is a quick memo of what I've been doing to move the VMs from
> treehouse to housetree. Since I was working in a hurry, I did not bother
> documenting all the trivial details such as installing ssh keys and
> rebooting. This isn't a step-by-step procedure to be repeated blindly.
> 
> == Host preparation ==
> 
> First of all, we synced the base OS to housetree with a clever
> rsync hack:
> 
>  housetree:~# rsync -PHAXphax --numeric-ids --delete \
>         --exclude /etc/hostname \
>         --exclude /etc/resolv.conf \
>         --exclude /etc/hosts \
>         --exclude /etc/network/interfaces \
>         --exclude /var/lib/dhcp3/ \
>         --exclude /etc/mdadm/mdadm.conf \
>         --exclude /etc/zzz_profile.conf \
>         --exclude '/etc/ssh/ssh_host_*' \
>         --exclude '/root/.ssh/id_*' \
>         / housetree:/
> 
> Then regenerate initrd to pick up the correct mdadm.conf:
> 
>  housetree:~# mkinitramfs -o /boot/initrd.img-2.6.31-16-server 2.6.31-16-server
> 
> CAVEAT: on reboot, we found that udev had messed up the network
> interfaces. We had to rm them from /etc/udev/rules.d/. To disable them
> for good:
> 
>  housetree:~# cd /lib/udev/rules.d/
>  housetree:~# mv 75-persistent-net-generator.rules 75-persistent-net-generator.rules.nothanks
> 
> 
> == Backups ==
> 
> I generated an ssh key for root on housetree and added it to our backup server.
> 
> NOTE: we must split our backups in separate user accounts for extra
> security
> 
> NOTE^2: we must use different time slots for each backup.
> 
> 
> == Partitioning ==
> 
> I created LVM partitions and formatted them. aslo-proxy-root is
> formatted in news mode so it won't run out of inodes:
> 
>  housetree:~# lvcreate -L 20G -n aslo-proxy-root housetree 
>  housetree:~# lvcreate -L 10G -n aslo-db-root housetree 
>  housetree:~# lvcreate -L 20G -n aslo-web-root housetree 
>  housetree:~# mkfs.ext4 -L aslo-db-root  -O flex_bg,extent,uninit_bg,sparse_super /dev/housetree/aslo-db-root 
>  housetree:~# mkfs.ext4 -L aslo-web-root  -O flex_bg,extent,uninit_bg,sparse_super /dev/housetree/aslo-web-root 
>  housetree:~# mkfs.ext4 -T news -L aslo-proxy-root -O flex_bg,extent,uninit_bg,sparse_super /dev/housetree/aslo-proxy-root
>  housetree:~# tune2fs -c -1 -i 0 -o journal_data_writeback,user_xattr,acl /dev/housetree/aslo-db-root 
>  housetree:~# tune2fs -c -1 -i 0 -o journal_data_writeback,user_xattr,acl /dev/housetree/aslo-web-root 
>  housetree:~# tune2fs -c -1 -i 0 -o journal_data_writeback,user_xattr,acl /dev/housetree/aslo-proxy-root 
> 
> Partition /srv holds the VMs that don't have a dedicated LV:
> 
>  housetree:~# lvcreate -L 100G -n housetree-srv housetree
>  housetree:~# mkfs.ext4 -L housetree-srv -O flex_bg,extent,uninit_bg,sparse_super /dev/housetree/housetree-srv 
>  housetree:~# tune2fs -c -1 -i 0 -o journal_data_writeback,user_xattr,acl /dev/housetree/housetree-srv
>  (edit /etc/fstab)
>  housetree:~# mount /srv
> 
> == Initial syncing ==
> 
> We've copied over the images from treehouse:
> 
>  treehouse:~# rsync -PHAXphax --numeric-ids --delete /srv/ housetree:/srv/


Here's the rest of the story:

== Moving over LVM logical volumes ===

* Log into the console to shutdown the VM

  treehouse:~# virsh console aslo-web-root

* Mount filesystems on both sides

 treehouse:~# mkdir /mnt/aslo-web-root
 treehouse:~# mount /dev/treehouse/aslo-web-root /mnt/aslo-web-root/
 treehouse:~# ssh housetree mkdir /mnt/aslo-web-root
 treehouse:~# ssh housetree mount /dev/housetree/aslo-web-root /mnt/aslo-web-root

* Copy over everything

 treehouse:~# rsync -HAXhax --numeric-ids  /mnt/aslo-web-root/ housetree:/mnt/aslo-web-root/

* Cleanup mounts

 treehouse:~# ssh housetree umount /mnt/aslo-web-root
 treehouse:~# umount /mnt/aslo-web-root

* Start VM on the other host

 treehouse:~# virsh start --console aslo-web

* Adjust network settings etc


== Extract remaining filesystems from VM disk images ==

* Mount the root partition contained within the hard disk file.

 treehouse:/# mkdir /mnt/aslo-proxy-root 
 treehouse:/# mount -o loop,offset=$((63*512)) /srv/vm/aslo-proxy.qcow2 /mnt/aslo-proxy-root

NOTE: I used the "u" option of fdisk to find that the start sector was 63.

* Create a regular LV to host the new filesystem:

 treehouse:/# lvcreate -L 20G -n aslo-proxy-root treehouse
 treehouse:/# mkfs.ext4 -T news -L aslo-proxy-root -O flex_bg,extent,uninit_bg,sparse_super /dev/treehouse/aslo-proxy-root
 treehouse:/# tune2fs -c -1 -i 0 -o journal_data_writeback,user_xattr,acl /dev/treehouse/aslo-proxy-root
 treehouse:/# mkdir /mnt2
 treehouse:/# mkdir /mnt2/aslo-proxy-root
 treehouse:/# mount /dev/treehouse/aslo-proxy-root /mnt2/aslo-proxy-root
 
 treehouse:/# rsync -HAXhax --delete --numeric-ids /mnt/aslo-proxy-root/ /mnt2/aslo-proxy-root/

* Edit /etc/fstab on the new VM to adjust the root partition:

 LABEL=aslo-proxy-root   /    ext4    errors=remount-ro       0       1

I generally disable swap on VMs. I'm not really sure about its
usefulness, and it may be quite harmful in some cases.

* Copy the kernel modules in VM (needed later):

 cp -a /lib/modules/2.6.31-16-server/ /mnt2/aslo-proxy-root/lib/modules/


* Cleanup mounts (unless you like to experience data corruption):

 treehouse:/# umount /mnt2/aslo-proxy-root
 treehouse:/# umount /mnt/aslo-proxy-root

* Edit the xml description of the VM to fix a few things:

 treehouse:/# virsh edit aslo-proxy

* Replace the disk file with this:

    <disk type='block' device='disk'>
      <source dev='/dev/treehouse/aslo-proxy-root'/>
      <target dev='vda' bus='virtio'/>
    </disk>

* Since we're using a bare partition which no longer contains an MBR
with GRUB and all that, we boot off an external kernel:

    <kernel>/srv/vm/kernel/ubuntu/vmlinuz-2.6.31-16-server</kernel>
    <initrd>/srv/vm/kernel/ubuntu/initrd.img-2.6.31-16-server</initrd>
    <cmdline>console=ttyS0,115200n8 vga=normal root=LABEL=aslo-proxy-root ro</cmdline>

NOTE TO SELF: perhaps qemu is fucking up something when loading the
kernel in memory in this fashion? Could this explain the occasional
lock-ups we've seen?

* Sit back and enjoy your shiny new VM:

 virsh start --console aslo-proxy

DISCLAIMER: the above procedures are highly mystical and should only be
performed by experienced personnel. DO NOT ATTEMPT IT AT HOME.

-- 
   // Bernie Innocenti - http://codewiz.org/
 \X/  Sugar Labs       - http://sugarlabs.org/