# Virtual Machine PBot can interact with a virtual machine to safely execute arbitrary user-submitted system commands and code. ## Installation methods ### libvirt and QEMU This document will guide you through installing and configuring a Linux virtual machine on a Linux host by using the widely available [libvirt](https://libvirt.org) project tools, such as `virt-install`, `virsh`, and `virt-viewer`. Additionally, if you'd prefer not to use libvirt, this guide will also demonstrate equivalent Linux system commands and QEMU commands. ### Vagrant A much simpler and easier method to install the virtual machine is to use the Vagrant virtual machine management and provisioning framework. To install the PBot virtual machine with just a couple of simple easy commands, read the [PBot Vagrant guide](../applets/pbot-vm/vagrant/README.md) instead. ## Host vs guest * `host`: your physical Linux system hosting the virtual machine * `guest`: the Linux system installed inside the virtual machine The commands below will be prefixed with `host$` or `guest$` to reflect where the command should be executed. ## Environment variables Many commands can be configured with environment variables. If a variable is not defined, a sensible default value will be used. Environment variable | Default value | Description -------------------- | ------------- | ----------- PBOTVM_DOMAIN | `pbot-vm` | The libvirt domain identifier PBOTVM_ADDR | `127.0.0.1` | `vm-server` address for incoming `vm-client` commands PBOTVM_PORT | `9000` | `vm-server` port for incoming `vm-client` commands PBOTVM_SERIAL | `5555` | TCP port for serial communication PBOTVM_HEALTH | `5556` | TCP port for serial health-check PBOTVM_CID | `7` | Context ID for VM socket (if using VSOCK) PBOTVM_VPORT | `5555` | VM socket service port (if using VSOCK) PBOTVM_TIMEOUT | `10` | Duration before command times out (in seconds) PBOTVM_NOREVERT | not set | If set then the VM will not revert to previous snapshot PBOTVM_VAGRANT | not set | If set then commands suitable for Vagrant VMs will be used # Initial virtual machine set-up These steps need to be done only once during the first-time set-up. ## Prerequisites For full hardware-supported virtualization at near native system speeds, we need to ensure your system has enabled CPU Virtualization Technology and that KVM is set up and loaded. ### CPU Virtualization Technology Ensure CPU Virtualization Technology is enabled in your motherboard BIOS. host$ egrep '(vmx|svm)' /proc/cpuinfo If you see your CPUs listed with `vmx` or `svm` flags, you're good to go. Otherwise, consult your motherboard manual to see how to enable VT. ### KVM Ensure KVM is set up and loaded. host$ kvm-ok INFO: /dev/kvm exists KVM acceleration can be used If you see the above, everything's set up. Otherwise, consult your operating system manual or KVM manual to install and load KVM. If you do not have the `kvm-ok` command, you can `ls /dev/kvm` to ensure the KVM device exists. ### libvirt and QEMU If using libvirt, ensure it is installed and ready. host$ virsh version --daemon Compiled against library: libvirt 7.6.0 Using library: libvirt 7.6.0 Using API: QEMU 7.6.0 Running hypervisor: QEMU 6.0.0 Running against daemon: 7.6.0 Just QEMU (assuming x86_64): host$ qemu-system-x86_64 --version QEMU emulator version 6.0.0 Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers If there's anything missing, please consult your operating system manual to install the libvirt and/or QEMU packages. On Ubuntu: `sudo apt install qemu-kvm libvirt-daemon-system` On OpenSUSE Tumbleweed: `sudo zypper in libvirt virt-install virt-viewer` ### Make a pbot-vm user or directory You can either make a new user account or make a new directory in your current user account. In either case, name it `pbot-vm` so we'll have a home for the virtual machine. ### Add libvirt group to your user Add your user (or the `pbot-vm` user) to the `libvirt` group. host$ sudo adduser $USER libvirt or host$ sudo usermod -aG libvirt $USER Log out and then log back in for the new group to take effect. Or use the `newgrp` command. ### Download Linux ISO Download a preferred Linux ISO. For this guide, we'll provide instructions for Fedora and OpenSUSE Tumbleweed. Why? I was initially using Fedora Rawhide for my PBot VM because I wanted convenient and reliable access to the latest bleeding-edge versions of software. I've since switched to OpenSUSE Tumbleweed for easy access to packages that are even more bleeding-edge than Fedora Rawhide. If you are more comfortable in another Linux distribution then feel free to choose that instead. Make sure you choose the minimal install option without a graphical desktop. The ISOs used in this guide are (you may instead prefer to navigate to the websites to download a more current image): https://download.fedoraproject.org/pub/fedora/linux/releases/35/Server/x86_64/iso/Fedora-Server-netinst-x86_64-35-1.2.iso or https://download.opensuse.org/tumbleweed/iso/openSUSE-Tumbleweed-NET-x86_64-Snapshot20240321-Media.iso I recommend using OpenSUSE Tumbleweed since that's what I've tested on most recently. ## Create a new virtual machine To create a new virtual machine, this guide offers two options. The first is libvirt's `virt-install` command. It greatly simplifies configuration by automatically creating networking bridges and setting up virtio devices. The second option is manually using Linux system commands to configure network bridges and execute QEMU with the correct options. ### libvirt To create a new virtual machine we'll use the `virt-install` command. This command takes care of setting up virtual networking bridges and virtual hardware for us. If you prefer to manually set things up and use QEMU directly, skip past the `virt-install` section. * First, ensure you are the `pbot-vm` user or that you have changed your current working directory to `pbot-vm`. The Linux ISO downloaded earlier should be present in this location. Execute the following command: Fedora (using Spice graphical display): host$ virt-install --name=pbot-vm --disk=size=12,path=vm.qcow2 --cpu=host --os-variant=fedora34 --graphics=spice --video=virtio --location=Fedora-Server-netinst-x86_64-35-1.2.iso OpenSUSE Tumbleweed (using PTY serial console): host$ virt-install --name=pbot-vm --disk=size=12,path=vm.qcow2 --cpu=host --os-variant=opensusetumbleweed --graphics=none --console=pty,target.type=virtio --serial=pty --extra-args=console=ttyS0,115200n8 --video=virtio --location=openSUSE-Tumbleweed-NET-x86_64-Snapshot20240321-Media.iso You may use `virt-install --os-variant list` to list the available `--os-variant` options present on your machine. Note that `disk=size=12` will create a 12 GB sparse file. Sparse means the file won't actually take up 12 GB. It will start at 0 bytes and grow as needed. You can use the `du` command to verify this. After a minimal Fedora install, the size will be approximately 1.7 GB. It will grow to about 2.5 GB with all PBot features installed. For further information about `virt-install`, read its manual page. While the above command should give sufficient performance and compatability, there are a great many options worth investigating if you want to fine-tune your virtual machine. To list virtual machines and their state use `virsh list --all`. If you need to ungracefully shutdown the virtual machine use `virsh destroy pbot-vm`. If you need to delete the virtual machine and its storage volume use: `virsh undefine pbot-vm --storage vda --snapshots-metadata`. ### QEMU If you prefer not to use libvirt, we may need to manually create the network bridge. Use the `ip link` command to list network interfaces: host$ sudo ip link 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 74:86:7a:4e:a1:95 brd ff:ff:ff:ff:ff:ff altname enp1s0 3: virbr0: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:83:3f:59 brd ff:ff:ff:ff:ff:ff inet 192.168.123.1/24 brd 192.168.123.255 scope global virbr0 valid_lft forever preferred_lft forever Create a new bridged named `pbot-br0`: host$ ip link add name pbot-br0 type bridge host$ ip link set pbot-br0 up Add your network interface to the bridge: host$ ip link set eth0 master pbot-br0 Give the bridge an IP address (use an appropriate address for your network): host$ ip addr add dev pbot-br0 192.168.50.2/24 We will use the `qemu-bridge-helper` program from the `qemu-common` package to create the TAP interface for us when we start the virtual machine and to remove the interface when the virtual machine is shut-down. To set the program up, we need to create its access control list file: host$ sudo mkdir /etc/qemu host$ sudo chmod 755 /etc/qemu host$ sudo echo allow pbot-br0 >> /etc/qemu/bridge.conf host$ sudo chmod 640 /etc/qemu/bridge.conf To allow unprivileged users to create VMs using the network bridge, we must set the SUID bit on the `qemu-bridge-helper` program: host$ chmod u+s /usr/lib/qemu/qemu-bridge-helper With the bridge configured, we move on to creating a sparse disk image for the virtual machine: host$ qemu-img create -f qcow2 pbot-vm.qcow2 12G Then we can start QEMU (assuming x86_64) and tell it to boot the installer ISO: Fedora: host$ qemu-system-x86_64 -enable-kvm -cpu host -mem 1024 -hda pbot-vm.qcow2 -cdrom Fedora-Server-netinst-x86_64-35-1.2.iso -boot d -nic bridge,br=pbot-br0 -usb -device usb-tablet OpenSUSE Tumbleweed: host$ qemu-system-x86_64 -enable-kvm -cpu host -mem 1024 -hda pbot-vm.qcow2 -cdrom openSUSE-Tumbleweed-NET-x86_64-Snapshot20240321-Media.iso -boot d -nic bridge,br=pbot-br0 -usb -device usb-tablet This command is the bare minimum for performant virtualization with networking. See the QEMU documentation for interesting options to tweak your virtual machine. ### Install Linux in the virtual machine After executing the `virt-install` or `qemu` command above, you should now see Linux booting up and launching an installer. For this guide, we'll walk through the Fedora 35 and the OpenSUSE Tumbleweed installers. You can adapt these steps for your own distribution of choice. Fedora: * Click `Partition disks`. Don't change anything. Click `Done`. * Click `Root account`. Click `Enable root account`. Set a password. Click `Done`. * Click `User creation`. Create a new user. Skip Fullname and set Username to `vm`. Untick `Add to wheel` or `Set as administrator`. Untick `Require password`. Click `Done`. * Wait until `Software selection` is done processing and is no longer greyed out. Click it. Change install from `Server` to `Minimal`. Click `Done`. * Click `Begin installation`. Installation will download about 328 RPMs consisting of about 425 MB. The `vm.qcow2` file should be about 2 GB after installation completes. You can close the Spice window. To reattach use `virt-viewer pbot-vm`. Tumbleweed: * Follow on-screen instructions and TAB to `Next` until you reach the `System Role` screen. * Ensure you select the `Server` role to install a small set of packages suitable for servers with a text mode interface. * On `Suggested Partitioning` TAB to `Guided Setup`. * Select `Next` until you reach `Filesystem Options`. * TAB to `Btrfs` and press SPACE and then arrow-keys to change this to `Ext4` to improve random-access IO performance and reduce writes. Then TAB to `Next` and continue. * Continue following on-screen instructions until you reach the `Local User` screen. * Enter `vm` for `Username` and set a password. Then TAB to `Next` and continue. * At the `Installation Settings` tab to `Change` and select `Security`. Untick `Enable Firewall` to make things easier. Then TAB to `Next` and continue. * Verify installation settings and then TAB to `Install` and begin the installation. Installation will download about 800 packages consisting of about 1.7 GiB. The `vm.qcow2` file should be about 2.4 GB after installation completes. The VM will automatically reboot into a shell after installation. You can press `^]` to detach from the VM's serial PTY console. To reattach use `virsh console pbot-vm`. ### Set up serial ports While the installation is in progress, switch to a terminal on your host system. #### libvirt Go into the `applets/pbot-vm/host/devices` directory and run the `add-serials` script to add the `serial-2.xml` and `serial-3.xml` files to the configuration for the `pbot-vm` libvirt machine. host$ ./add-serials This will enable the `/dev/ttyS2` and `/dev/ttyS3` serial ports in the guest and connect them to the following TCP addresses on the host: `127.0.0.1:5555` and `127.0.0.1:5556`, respectively. `ttyS2/5555` is the data channel used to send commands or code to the virtual machine and to read back output. `ttyS3/5556` responds with the output of the `vmstat` command to let us know about the virtual machine's health. You may use the `PBOTVM_DOMAIN`, `PBOTVM_SERIAL` and `PBOTVM_HEALTH` environment variables to override the default values. To use ports `7777` and `7778` instead: host$ PBOTVM_SERIAL=7777 PBOTVM_HEALTH=7778 ./add-serials If you later want to change the serial ports or the TCP ports, execute the command `virsh edit pbot-vm` on the host. This will open the `pbot-vm` XML configuration in your default system editor. Find the `` tags and edit their attributes. #### QEMU Add the following options to your `qemu` command-line arguments: -chardev socket,id=charserial1,host=127.0.0.1,port=5555,server=on,wait=off -device {"driver":"isa-serial","chardev":"charserial1","id":"serial1","index":2} -chardev socket,id=charserial2,host=127.0.0.1,port=5556,server=on,wait=off -device {"driver":"isa-serial","chardev":"charserial2","id":"serial2","index":3} If necessary, replace `5555` and `5556` with your preferred `PBOTVM_SERIAL` and `PBOTVM_HEALTH` values. ### Set up virtio-vsock VM sockets (AF_VSOCK) are a Linux-specific feature (at the time of this writing). They are the preferred way for PBot to communicate with the PBot VM Guest server. Serial communication has several limitations. See https://vmsplice.net/~stefan/stefanha-kvm-forum-2015.pdf for an excellent overview. To use VM sockets with QEMU and virtio-vsock, you need: * a Linux hypervisor with kernel 4.8+ * a Linux virtual machine on that hypervisor with kernel 4.8+ * QEMU 2.8+ on the hypervisor, running the virtual machine * [socat](http://www.dest-unreach.org/socat/) version 1.7.4+ If you do not meet these requirements, the PBot VM will fallback to using serial communication. You may explicitly disable VM sockets by setting `PBOTVM_CID=0`. You can skip reading the rest of this section. If you do want to use VM sockets, read on. First, ensure the `vhost_vsock` Linux kernel module is loaded on the host: host$ lsmod | grep vsock vhost_vsock 24576 1 vsock 45056 2 vmw_vsock_virtio_transport_common,vhost_vsock vhost 53248 2 vhost_vsock,vhost_net If the module is not loaded, load it with: host$ sudo modprobe vhost_vsock Once the module is loaded, you should have the following character devices: host$ ls -l /dev/vhost-vsock crw------- 1 root root 10, 53 May 4 11:55 /dev/vhost-vsock host$ ls -l /dev/vsock crw-rw-rw- 1 root root 10, 54 May 4 11:55 /dev/vsock A VM socket address is comprised of a context ID (CID) and a port; just like an IP address and TCP/UDP port. The CID is represented using an unsigned 32-bit integer. It identifies a given machine as either a hypervisor or a virtual machine. Several addresses are reserved, including 0, 1, and the maximum value for a 32-bit integer: 0xffffffff. The hypervisor is always assigned a CID of 2, and VMs can be assigned any CID between 3 and 0xffffffff - 1. We must attach a `vhost-vsock-pci` device to the guest to enable VM sockets communication. Each VM on a hypervisor must have a unique context ID (CID). Each service within the VM must have a unique port. The PBot VM Guest defaults to `7` for the CID and `5555` for the port. #### libvirt While still in the `applets/pbot-vm/host/devices` directory, run the `add-vsock` script: host$ ./add-vsock or to configure a different CID: host$ PBOTVM_CID=42 ./add-vsock In the VM guest (once it reboots), there should be a `/dev/vsock` device: guest$ ls -l /dev/vsock crw-rw-rw- 1 root root 10, 55 May 4 13:21 /dev/vsock #### QEMU Add the following option to your `qemu` command-line arguments. -device vhost-vsock-pci,guest-cid=$PBOTVM_CID or -device {"driver":"vhost-vsock-pci","id":"vsock0","guest-cid":$PBOTVM_CID,"vhostfd":"28","bus":"pci.7","addr":"0x0"} See full QEMU command-line arguments [here.](#qemu-command-from-libvirt) In the VM guest (once it reboots), there should be a `/dev/vsock` device: guest$ ls -l /dev/vsock crw-rw-rw- 1 root root 10, 55 May 4 13:21 /dev/vsock ### Reboot virtual machine * First ensure you set-up serial/vsock as described above! We are rebooting to ensure the new devices are loaded. Fedora: Once the Fedora installation completes inside the virtual machine, click the `Reboot` button in the installer window. Login as `root` when the virtual machine boots back up. Tumbleweed: The Tumbleweed installer will automatically reboot to a shell after the installation. Login as `root` and run `shutdown now -h`. Then run `virsh start pbot-vm`. (Using `shutdown now -r` to reboot will not initialize the new serial/vsock devices.) Login as `root` when the virtual machine boots back up. ### Install PBot VM Guest Next we install the PBot VM Guest server script that fosters communication between the virtual machine guest and the physical host system. We'll do this inside the virtual machine guest system, logged on as `root` while in the `/tmp` directory. guest$ cd /tmp The `rsync` command isn't installed with a Fedora minimal install, but `scp` is available. Replace `192.168.100.42` below with your own local IP address; `user` with the user account that has the PBot directory; and `pbot` with the path to the directory. guest$ scp -r user@192.168.100.42:~/pbot/applets/pbot-vm/guest . Once that's done, run the following command: guest$ ./guest/bin/setup-guest This will install `guest-server` to `/usr/local/bin/`, set up some environment variables and harden the guest system. Additionally, it'll autodetect your chosen OS/distribution and attempt to run any provisioning scripts from the [`./guest/provision`](../applets/pbot-vm/guest/provision) directory. After running the `setup-guest` script, we need to make the environment changes take effect: guest$ source /root/.bashrc ### Install software If you received the message `!! No provisioning script available for $OS. Install packages manually. !!`, you must manually install any software and programming languages you want to make available in the virtual machine. To do so, follow one of the [PBot VM provisioning scripts](../applets/pbot-vm/guest/provision/) as a guide. ### Start PBot VM Guest We're ready to start the PBot VM Guest server. On the guest, as `root`, execute the command: guest$ guest-server This starts up a server to listen for incoming commands or code and to handle them. We'll leave this running. ### Test PBot VM Guest Let's make sure everything's working up to this point. On the host, there should be two open TCP ports on `PBOTVM_SERIAL` and `PBOTVM_HEALTH` (default values `5555` and `5556`). On the host, execute the command: host$ nc -zv 127.0.0.1 5555-5556 If it says anything other than `Connection succeeded` then make sure you have completed the steps under [Set up serial ports](#set-up-serial-ports) and that your network configuration is allowing access. Let's make sure the PBot VM Guest server is listening for and can execute commands. The `vm-exec` command allows you to send commands from the shell. Change your current working directory to `applets/pbot-vm/host/bin` and run the `vm-exec` command: host$ cd applets/pbot-vm/host/bin host$ ./vm-exec -lang=sh echo hello world This should output some logging noise followed by "hello world". You can test other language modules by changing the `-lang=` option. I recommend testing and verifying that all of your desired language modules are configured before going on to the next step. If you have multiple PBot VM Guests, or if you used a different TCP port, you can specify the `PBOTVM_SERIAL` environment variable when executing the `vm-exec` command: host$ PBOTVM_SERIAL=7777 ./vm-exec -lang=sh echo test ### Save initial state Switch back to an available terminal on the physical host machine. Enter the following command to save a snapshot of the virtual machine waiting for incoming commands. * Before doing this step, ensure all commands are cached by executing them at least once. For example, the `gcc` and `gdb` commands take a long time to load into memory. The initial execution may take a several long seconds to complete. Once completed, the command will be cached. Future invocations will execute significantly quicker. host$ virsh snapshot-create-as pbot-vm 1 If the virtual machine times-out, PBot will revert to this saved snapshot. ## Install host packages Ensure the following packages are installed on the host machine: * fortune -- used to generate STDIN input when no `-stdin` flag is provided to the bot's `cc` command * astyle -- used to pretty-format C-family languages ## Initial virtual machine set-up complete This concludes the initial one-time set-up. You can close the `virt-viewer` window or press `^]` to detach from the VM's serial PTY console. The virtual machine will continue running in the background until it is manually shutdown (via `shutdown now -h` inside the VM or via `virsh shutdown pbot-vm` on the host). # Start PBot VM Host To start the PBot VM Host server, change your current working directory to `applets/pbot-vm/host/bin` and execute the `vm-server` script: host$ cd applets/pbot-vm/host/bin host$ ./vm-server This will start a TCP server on port `9000`. It will listen for incoming commands and pass them along to the virtual machine's TCP serial port `5555`. It will also monitor the health-check port `5556` to ensure the PBot VM Guest server is alive. You may override any of the defaults by setting environment variables. For example, to use `pbot-test-vm` with a longer `30` second timeout, on different serial and health-check ports: host$ PBOTVM_DOMAIN="pbot-test-vm" PBOTVM_SERVER=9001 PBOTVM_SERIAL=7777 PBOTVM_HEALTH=7778 PBOTVM_TIMEOUT=30 ./vm-server ## Test PBot All done. Everything is set up now. PBot is already preconfigured with commands that invoke the `host/bin/vm-client` script to send VM commands to `vm-server` on the default port `9000`: factshow sh [global] sh: /call cc -lang=sh factshow cc [global] cc: /call vm-client {"nick":"$nick:json","channel":"$channel:json","code":"$args:json"} factshow vm-client [global] vm-client: pbot-vm/host/bin/vm-client [applet] In your instance of PBot, the `sh echo hello` command should output `hello`. sh echo hello hello ## Starting PBot VM again To start the PBot VM again, e.g. after a reboot: cd applets/pbot-vm/host/bin virsh snapshot-revert $PBOTVM_DOMAIN 1 # for each $PBOTVM_DOMAIN you need to start ./vm-server ## Adding additional VMs You may add as many virtual machines as you like. The virtual machines can exist on different physical machines. Edit the [`vm-exec.json`](../applets/pbot-vm/host/config/vm-exec.json) configuration file with the details (IP addresses of remote machines hosting the virtual machines, etc). Then use the `-vm=...` option to select the machine with the `sh`, `cc`, etc, commands. For instance, if you create a FreeBSD VM and have added it to [`vm-exec.json`](../applets/pbot-vm/host/config/vm-exec.json) with an alias of `freebsd`: sh -vm=freebsd echo testing testing You can create aliases to omit the `-vm=...` option: factalias bsd-sh sh -vm=freebsd $args bsd-sh foobar foobar ## QEMU command from libvirt This is the QEMU command-line arguments used by libvirt. Extract flags as needed, e.g. `-chardev`. /usr/bin/qemu-system-x86_64 -name guest=pbot-vm,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-pbot-vm/master-key.aes"} -machine pc-q35-6.2,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram -accel kvm -cpu IvyBridge-IBRS,ss=on,vmx=on,pdcm=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,aes=off,rdrand=off -m 2048 -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648} -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid ec9eebba-8ba1-4de3-8ec0-caa6fd808ad4 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=38,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device {"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"} -device {"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"} -device {"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"} -device {"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"} -device {"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"} -device {"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"} -device {"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"} -device {"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"} -device {"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"} -device {"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"} -device {"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"} -device {"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"} -device {"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"} -device {"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"} -device {"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"} -blockdev {"driver":"file","filename":"/home/pbot/pbot-vms/openSUSE-Tumbleweed-Minimal-VM.x86_64-kvm-and-xen.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null} -device {"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1} -netdev {"type":"tap","fd":"39","vhost":true,"vhostfd":"41","id":"hostnet0"} -device {"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:03:16:5a","bus":"pci.1","addr":"0x0"} -chardev pty,id=charserial0 -device {"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0} -chardev socket,id=charserial1,host=127.0.0.1,port=5555,server=on,wait=off -device {"driver":"isa-serial","chardev":"charserial1","id":"serial1","index":2} -chardev socket,id=charserial2,host=127.0.0.1,port=5556,server=on,wait=off -device {"driver":"isa-serial","chardev":"charserial2","id":"serial2","index":3} -chardev socket,id=charchannel0,fd=37,server=on,wait=off -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"} -chardev spicevmc,id=charchannel1,name=vdagent -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":2,"chardev":"charchannel1","id":"channel1","name":"com.redhat.spice.0"} -device {"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"} -audiodev {"id":"audio1","driver":"spice"} -spice port=5901,addr=127.0.0.1,disable-ticketing=on,image-compression=off,seamless-migration=on -device {"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"} -device {"driver":"ich9-intel-hda","id":"sound0","bus":"pcie.0","addr":"0x1b"} -device {"driver":"hda-duplex","id":"sound0-codec0","bus":"sound0.0","cad":0,"audiodev":"audio1"} -chardev spicevmc,id=charredir0,name=usbredir -device {"driver":"usb-redir","chardev":"charredir0","id":"redir0","bus":"usb.0","port":"2"} -chardev spicevmc,id=charredir1,name=usbredir -device {"driver":"usb-redir","chardev":"charredir1","id":"redir1","bus":"usb.0","port":"3"} -device {"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"} -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device {"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.6","addr":"0x0"} -loadvm 1 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -device {"driver":"vhost-vsock-pci","id":"vsock0","guest-cid":7,"vhostfd":"28","bus":"pci.7","addr":"0x0"} -msg timestamp=on