Virtualization #

Highly-customized QEMU VM with Buildroot #

Boot minimal Alpine VM #

Download Alpine’s netboot images, which provides multiple flavors of vmlinuz and initramfs.

qemu-system-x86_64 -enable-kvm \
    -m 1G -kernel vmlinuz-virt \
    -initrd initramfs-virt \
    -append "console=ttyS0 ip=dhcp alpine_repo=http://dl-cdn.alpinelinux.org/alpine/edge/main/" \
    -nographic -nic user,model=e1000

Install minimal Alpine VM #

Download Alpine’s virt image.

qemu-img create -f qcow2 alpine.qcow2 8G
qemu-system-x86_64 -enable-kvm \
    -m 1G -nographic -nic user,model=e1000
    -boot d -cdrom alpine-virt-3.10.2-x86_64.iso -hda alpine.qcow2

# Run alpine-setup in the VM and reboot
# Boot with SSH port forwarding
qemu-system-x86_64 -enable-kvm \
    -m 4G -smp 4 \
    -hda alpine.qcow2
    -device e1000,netdev=net0 \
    -nographic -netdev user,id=net0,hostfwd=tcp::5557-:22

Vagrant’s Quick Ubuntu #

Install and start libvirtd:

sudo apt install libvirtd
sudo systemctl start libvirtd

Create a file named Vagrantfile:

Vagrant.configure("2") do |config|
	# check here: https://app.vagrantup.com/generic/boxes/ubuntu2004
	config.vm.box = "generic/ubuntu1804"
	config.vm.provider "libvirt" do |v|
		v.memory = 16384  # 16G
		v.cpus = 4
		v.cpu_mode = "host-model"
	end
end

Create the VM:

vagrant up
vagrant ssh

Ubuntu Cloud-init #

Mark Manis’s guide: Setting up a QEMU VM with a custom kernel and Ubuntu

Libguestfs #

Libguestfs has many useful utilities to deal with VM image:

guestmount: mount the root file-system of a guest image read-only:

sudo guestmount -a guest.img -i --ro /mnt

`virt-builder` with custom mirror #

virt-builder is useful for quickly building a VM image of arbitrary Linux distribution.

# If you are dropped into initramfs shell, press F10 to enter grub menu
# and replace /dev/vda1 with /dev/sda1.  Then, modify /etc/default/grub,
# and run grub-mkconfig -o /boot/grub.d/grub.cfg
sudo apt install libguestfs-tools

virt-builder debian-10 \
        --no-check-signature --format qcow2 --size 30G \
        --root-password password:rootpass \
        --output debian-10.qcow2

virt-builder fedora-26 \
	--source http://libguestfs.thycat.com/index \
	--no-check-signature --format qcow2 --size 30G \
	--output /vm/images/fedora-26.qcow2


# Boot it
MY_IMG=debian-10.qcow2
qemu-system-x86_64 \
    -enable-kvm -cpu host -smp 4 -m 1G \
    -device virtio-scsi-pci,id=scsi0 -device scsi-hd,drive=hd0 \
    -drive file=$MY_IMG,if=none,format=qcow2,id=hd0 \
    -net user,hostfwd=tcp::8080-:22 \
    -net nic,model=virtio \
    -nographic

Quickly set up Debian:

echo << EOF > /tmp/run-install-debian.sh
#!/bin/sh
ssh-keygen -A
sed -i 's/deb\..*org/free\.nchc\.org\.tw/g' /etc/apt/sources.list
apt update
apt install -y build-essential sudo
useradd -G sudo -m yunchih
echo 'yunchih ALL=(ALL) NOPASSWD: ALL' >> /tmp/yunchih
chown root:root /tmp/yunchih
mkdir -p /etc/sudoers.d/
mv /tmp/yunchih /etc/sudoers.d/
EOF

virt-builder debian-10 \
        --no-check-signature --format qcow2 --size 30G \
        --root-password password:rootpassword \
        --run /tmp/run-install-debian.sh \
        --ssh-inject yunchih:file:yunchih-ssh-key \
        --output debian-10.qcow2
rm /tmp/run-install-debian.sh

List all available distributions: virt-builder --list

Run commands on VM: `guestfish` #

guestfish -a /vm/images/fedora-26.qcow2
# in the fish shell:
<fs> run    # start the VM
<fs> ! id   # prefix by !: arbitrary commands sent to shell
<fs> ! ls
<fs> list-filesystem

Other patterns: here

Debootstrap #

debootstrap is useful for quickly testing various Debian versions, the following commands create a minimal Ubuntu 16.04 raw image:

dd if=/dev/zero of=ubuntu.img bs=1M count=2000 
mkfs.ext4 ubuntu.img
mount -o loop ubuntu.img /mnt
debootstrap --arch=amd64 xenial /mnt
umount /mnt

For aarch64:

sudo debootstrap --arch=arm64 \
	--variant=minbase --foreign \
	testing /mnt \
	http://ftp.tw.debian.org/debian/

To mount again to add package:

sudo losetup -Pf debian.img
sudo mount /dev/loop0p1 /mnt
sudo debootstrap --arch=arm64 --include=vim --variant=minbase --foreign testing /mnt http://ftp.debian.org/debian/

Boot with QEMU for x86:

dd if=/dev/zero of=debian.img bs=2M count=2000
sudo mkfs.ext2 debian.img
sudo mount debian.img mnt
sudo debootstrap --arch=amd64 stretch mnt http://ftp.tw.debian.org/debian/

# chroot
sudo mount --rbind /dev mnt/dev
sudo mount --make-rslave mnt/dev
sudo mount --rbind /sys mnt/sys
sudo mount --make-rslave mnt/sys
sudo mount -t proc /proc mnt/proc
sudo chroot mnt

# in the chroot
passwd # change root password
apt-get install wget sudo openssh-server # install package is fast
useradd -m -G sudo yunchih

# boot my Linux 4.19 image on Debian stretch, whose default kernel is 4.9
qemu-system-x86_64 -M pc -smp 8 -m 400 -enable-kvm \
    -kernel /tmp/dac_imgs/linux-4.19.91/arch/x86_64/boot/bzImage \
    -drive file=/tmp/debian-img/debian.img,if=virtio,format=raw \
    -append "rootwait rw root=/dev/vda console=tty1 console=ttyS0" \
    -device virtio-net-pci,netdev=n0 \
    -netdev user,id=n0,hostfwd=tcp::5555-:22 \
    -nographic

Rescue a broken image #

A newly-installed kernel might be unbootable. We need to enter the image to delete it

sudo guestmount -a /tmp/mnt/.yunchih/images/for-zns.qcow2   -m /dev/sda1 /mnt
sudo mount -o bind /dev /mnt/dev
sudo mount -o bind /dev/pts /mnt/pts
sudo mount -o bind /proc /mnt/proc
sudo mount -o bind /run /mnt/run
sudo mount -o bind /sys /mnt/sys
sudo chroot /mnt /bin/bash

# inside the chroot
apt-get remove linux-image ...
grub-mkconfig -o /boot/grub/grub.cfg
exit

umount /mnt/run /mnt/sys /mnt/dev/pts
umount /mnt/dev
umount -lf /mnt

Libvirt/QEMU #

Boot an image with KVM enabled (do you have /dev/kvm permission?):

# or /usr/bin/qemu-system-x86_64
/usr/libexec/qemu-kvm -smp 4 \
	-m 4096 -kernel vmlinuz-3.13.0-133-generic \
	-initrd initrd.img-3.13.0-133-generic \
	-drive format=qcow2,file=ubuntu-14.04-server-cloudimg-amd64-disk1.img \
	-append "console=ttyS0 root=/dev/sda" \
	-nographic -display none -serial mon:stdio

View non-graphical console of a Libvirt guest #

Enable the serial terminal in guest:

systemctl enable serial-getty@ttyS0.service
systemctl start serial-getty@ttyS0.service

Add to /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0"
GRUB_TERMINAL="serial console"

Make sure the serial ID matches (edit with virsh edit):

<serial type='pty'>
  <target port='0'/>
</serial>
<console type='pty'>
  <target type='serial' port='0'/>
</console>

`virt-install` script for PXE installation #

Assign more than 2G memory to the VM because dracut (initramfs), one of the most critical part of OS bootstrapping, will mount RAM-based filesystem which requires that much space.
br0 is the public interface bridge, br1 is the private interface bridge.
boot_order=1 tells QEMU to boot on the desired interface, which contains a PXE service in the VLAN.

sudo virt-install \
        --connect=qemu:///system \
        --name [name of your VM] \
        --vcpus=2 \
        --memory=4096 \
        --network bridge=br0,model=virtio,mac=52:83:3d:f1:9b:d9 \
        --network bridge=br1,model=virtio,mac=52:54:00:f2:8c:ae,boot_order=1 \
        --disk path=/imgs/[disk].qcow2 \
        --pxe \
        --os-type=linux ＼
        --os-variant=rhel7 ＼
        --graphics=vnc,password=meowmeow,listen=7788 \
        --noautoconsole

Query the os-variant parameter with: osinfo-query os.

Broken network interface #

Check if the bridge is intact:

sudo brctl show
sudo domiflist [guest]

If their output disagrees, add manually:

sudo brctl addif br1 vnet10

Somtimes, restarting the libvirtd.service service also fixes the problem (this will not kill the QEMU processes it manages, thus is almost harmless to your service; it will restart the dnsmasq instance it manages for VM NAT).

Resize small console window #

When you enter a serial console via virsh console, the console size doesn’t expand to real size of your terminal. The resize command, a part of the xterm package, solve this problem:

eval `/usr/bin/resize`

If installing xterm is undesirable, manually adjust the console size:

stty rows $HEIGHT cols $WIDTH
stty rows 56 cols 132

Docker #

# With newer docker version
docker run -v /var/db:/var/db:Z rhel7 /bin/sh
# Manual
chcon -Rt svirt_sandbox_file_t /path/to/volume

Keep a running Docker #

This runs the interactive mode in background:

docker run --name ubuntu-1 -dit ubuntu

Run a shell:

docker exec -it ubuntu-1 /bin/bash

Or attach:

docker attach ubuntu-1

And exit with ^P^Q.