Troubleshooters.Com®, Linux Library and SSL Present:
Steve Litt's No Bullshit Guide to LAN-Peer Qemu VMs
Copyright © 2021 by Steve Litt
See the Troubleshooters.Com Bookstore.
CONTENTS
To prevent confusion and ambiguity, this document repeatedly and redundantly describes each element as being either on the host or the VM guest. The host is the physical computer (metal computer) on which the VM guest is run. The VM guest is a Virtual Machine running on the host. The downfall of most online Qemu documentation is they rarely make this distinction, which can confuse the daylights out of someone not already knowing Virtual Machine technology extremely well.
The examples in this document use a physical host that is running Void Linux. The VM guest runs Devuan Linux. This document's examples all use a LAN using the 192.168.0.0/24 network (subnet, actually), with a physical cable modem connection to the Internet at 192.168.0.1, the host at 192.168.0.2, a printer with HTML interface at 192.168.0.13, and a different Linux computer at 192.168.0.97. These four machines are all physical. On the LAN, the default route is the cable modem at 192.168.0.1, which also serves as a DHCP server, firewall, and router.
A LAN-peer is a VM guest that can interact with all the computers and devices on the host's LAN. Also, all the computers and devices on that LAN can interact with the VM guest. A LAN-peer VM guest is, from a networking standpoint, exactly like a physical computer on the LAN.
If parts of this document seem redundant and excessive, I've done this so that there's absolutely no confusion for anybody, regardless of their Virtual Machine or Qemu knowledge.
If you want to skip the theory and explanations of failures of other Qemu documentation, read this section.
The following represents the network configuration on the metal host machine:
[slitt@mydesk qemu]$ ip -4 link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp40s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 1000 link/ether 2c:f0:5d:7a:5c:1d brd ff:ff:ff:ff:ff:ff 3: enp42s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 2c:f0:5d:7a:5c:1c brd ff:ff:ff:ff:ff:ff 45: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 2c:f0:5d:7a:5c:1d brd ff:ff:ff:ff:ff:ff [slitt@mydesk qemu]$ ip -4 addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 45: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 192.168.0.2/24 scope global br0 valid_lft forever preferred_lft forever inet 192.168.0.102/24 scope global secondary br0 valid_lft forever preferred_lft forever [slitt@mydesk qemu]$ ip -4 route default via 192.168.0.1 dev br0 192.168.0.0/24 dev br0 proto kernel scope link src 192.168.0.2 [slitt@mydesk qemu]$
If you don't know how to achieve the preceding networking configuration on your host with your distribution, this will be discussed later. Suffice it to say that on your host, a software bridge called br0is created and attached to your host's physical network device. The software bridge has the IP addresses and routing, whereas the host's physical network device has neither addresses nor routing connected to it.
Your Addresses Will Be Different
Obviously, your IP addresses, including routing, will be different than the illustration in this section. That's fine, just change them in a consistent manner.
Also, note that there's a second physical network device (enp42s0) on the host. It's just an extra network device the motherboard manufacturer threw in: You can ignore it. Unless your host has more than one Ethernet network devices, you won't have this extra device on your host.
Because this example creates a VM guest whose network is DHCP derived, there's no specific Ethernet config on the VM guest.
Armed with the preceding network config on your host, the following shellscript should be able to create your VM guest as a LAN-peer:
#!/bin/sh dvddir=/scratch/linuxinst/devuan/devuan_beowulf/installer-iso qemu-system-x86_64 --enable-kvm \ -cdrom $dvddir/devuan_beowulf_3.0.0_amd64-desktop.iso \ -hda /scratch/qemu_images/beowulf.disk \ -m 4G \ -boot c \ -netdev bridge,id=mybridge0,br=br0 \ -device virtio-net-pci,netdev=mybridge0 \
In the preceding shellscript, notice that all lines of the qemu-system-x86_64 command end with backslashes, which are line continuations, meaning that the entire qemu-system-x86_64 command is interpreted as just one line. I backslashed all of them so I could move things in and out without needing to rearrange line continuations. However, this universal use of line continuations requires a blank line at the end to terminate the command.
Most of the preceding shellscript are common, ordinary Qemu stuff you can read about anywhere, but observe the last two lines:
-netdev bridge,id=mybridge0,br=br0 \ -device virtio-net-pci,netdev=mybridge0 \
The first of those two lines creates, on the host, a software bridge with device name br0 and id of mybridge0.
The second line creates, on the VM guest, a software Ethernet device, which is linked to the software bridge on the host by using that software bridge's id value as its netdev value.
On my Devuan Qemu VM guest, the software Ethernet device is named eth0. This is the name assigned it by the Linux kernel. Other distros name it other names (usually starting with en.
At this point, the network configuration on the host, and the VM guest producing shellscript on the host have both been extensively discussed. What remains is the network configuration on the VM guest, which because the VM guest is running Devuan Linux, is configured with /etc/network/interfaces. It turns out that for a DHCP created VM guest network, almost no configuration is done: Qemu does it for you. The following is the /etc/network/interfaces file on a DHCP configured VM guest running Devuan:
source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback
As you can see from the preceding, no effort is made, on the VM guest, to configure the Ethernet portion of the network. This is because, with this configuration, all VM guest network config is accomplished via the DHCP server on the physical cable modem at 192.168.0.1. The preceding VM guest network config file produces the following network situation on the VM guest:
slitt@beowulftest:~$ ip -4 link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff slitt@beowulftest:~$ ip -4 addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.0.204/24 brd 192.168.0.255 scope global dynamic noprefixroute eth0 valid_lft 2404sec preferred_lft 2404sec slitt@beowulftest:~$ ip -4 route default via 192.168.0.1 dev eth0 proto dhcp metric 100 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.204 metric 100 slitt@beowulftest:~$
Here are some other things that should be mentioned:
Because the preceding VM guest setup uses DHCP to determine all networking, including the DNS query addresses, it likely uses the DNS servers dictated by the Internet Service Provider (ISP). If the VM guest is hosted on a laptop that you carry around and use in different places, this is probably what you want.
But if your host is consistently used as just another computer on your LAN, and if your LAN has its own DNS server, then you probably want the VM guest to use the LAN's DNS. There are many distro-dependent and distro-correct ways of doing this, but personally, I just strongarm it. Assuming the DNS for my LAN (which also queries the Internet) is at 192.168.0.102, I create the following /etc/resolv.static, containing the following:
nameserver 192.168.0.102
I then copy the existing /etc/resolv.conf to /etc/resolv.mobile. Then, to use my LAN's DNS, I perform the following command:
cd /etc chattr -i resolv.conf; cp resolv.static resolv.conf; chattr +i resolv.conf
In the preceding, it's important to run them all as one command, so some silly program like network-manager doesn't change it in the middle of your task. Personally, I don't feel bad at all kludging resolv.conf immutable, because in my opinion some program like network-manager modifying resolv.conf is an unnecessarily bad solution to the problem of mobility.
The preceding section assembled a VM guest using DHCP. This is simplest and makes sense on a mobile laptop, but doesn't make sense when the host is on the same LAN most or all of the time. When the machine is on the same LAN most of the time, having a static IP address for the VM guest makes much more sense. This section explains how to accomplish this.
For a Devuan VM guest, all additional configuration necessary for a static IP for the VM guest occurs in the VM guest's /etc/network/interfaces file. For other distros, you need to do equivalent things. The point is, all the configuration necessary to make a VM guest with a static IP occurs on the VM guest. For a Devuan VM guest, the following /etc/network/interfaces creates a static IP:
source /etc/network/interfaces.d/* # The loopback network interface auto lo iface lo inet loopback # BELOW SLITT SETS ETH0 STATIC IP auto eth0 allow-hotplug eth0 iface eth0 inet static network 192.168.0.0 netmask 255.255.255.0 address 192.168.0.66 broadcast 192.168.0.255 gateway 192.168.0.1
In the preceding, let's talk about the gateway entry, which sets the default route. This address is set by the VM guest's DHCP, which queries the LAN's router (default gateway) and uses that for the VM guest's default route.
Oddly, you could set the VM guest's default route to anything in any private address block (10.0.0.0/8, 172.16.0.0/16, or 192.168.0.0/24), whether it's in the LAN's subnet or not. I set the gateway at 192.168.0.1, because that's the LAN's gateway to the Internet, so that the VM guest can access the Internet. If I had used a random address like 10.10.10.10, the VM guest could have accessed the entire LAN, but not the Internet.
As mentioned earlier, host runs Void Linux, a distro that has no /etc/network/interfaces or any other kind of declarative network configuration. Instead, Void Linux networking is configured imperatively, in other words, via shellscripts. I personally prefer the imperative approach because you don't need to learn yet another config language, and you can do anything your heart desires. Knowledge of ordinary shellscripting and the ip command is sufficient for any kind of networking arrangement. It is also portable between distros, as long as the distro has the ip command. It could even be used on Devuan if you comment out everything in /etc/network/interfaces, but Devuan's declarative network setup in /etc/network/interfaces is simple enough that I use the declarative network setup for Devuan.
Before viewing my 142 line host network setup shellscript, please understand that it has configuration and error checking. For instance, with trivial changes to variables at the top, you can get it just to configure only the physical network card enp40s0 complete with IP addresses and default route, or you can connect software bridge br0 to enp40s0, and attach the routing and IP addresses to br0 instead of enp40s0. If enp40s0 is connected, changing one more variable creates a TAP virtual network device, which is offtopic in this document but can be very useful in other situations. Also, the two network addresses (one for the host's main IP address and one for the host's DNS server. If you just wanted to hard-code everything, the script would be 15 to 30 lines long, depending on how much work it does.
The network-configuring shellscript, which is run at boot but can also be run any time because it undoes all the config before redoing it, is shown below:
#!/bin/sh use_bridge=1 use_tap=0 dev="enp40s0" ipaddr_major="192.168.0.2" ipaddr_minor="192.168.0.102" gateway="192.168.0.1" error_tap_without_bridge(){ echo -n "ERROR: Can\'t set TAP without " echo -n "BRIDGE! " echo Aborting... exit 1 } enable_ip_forwarding(){ echo 1 > /proc/sys/net/ipv4/ip_forward } unset_everything(){ dev=$1 ip_maj=$2 ip_min=$3 gateway=$4 ip link set dev lo down echo "Unsetting everything for $dev, $ip_maj and $ip_min" ip link set dev tap0 down brctl delif br0 tap0 ip link del tap0 ip link set dev br0 down ip addr del $ip_min/24 dev br0 ip addr del $ip_maj/24 dev br0 brctl delbr br0 ip link set dev $dev down ip addr del $ip_min/24 dev $dev ip addr del $ip_maj/24 dev $dev echo "" } set_hostname_and_localhost(){ echo "Setting hostname and localhost" hostname=`grep -v "^\s*#" /etc/hostname | head -n1` ip link set dev lo up echo "" } create_phys_device_link(){ dev=$1 echo Creating device link for $dev ip link set dev $dev up echo "" } set_phys_device_addr(){ dev=$1 ip_maj=$2 ip_min=$3 gateway=$4 echo -n "Setting physical device addresses " echo -n "$ip_maj " echo -n "and $ip_min " echo -n "for $physdev " echo "with gateway $gateway" ip link set dev $dev down ip addr add $ip_maj/24 dev $dev ip addr add $ip_min/24 dev $dev ip link set dev $dev up ip route add default via $gateway echo "" } set_bridge(){ dev=$1 ip_maj=$2 ip_min=$3 gateway=$4 echo Setting bridge for $dev echo -n "Creating and setting bridge addresses " echo -n "$ip_maj " echo -n "and $ip_min " echo -n "for $physdev " echo "with gateway $gateway" ip link add name br0 type bridge ip link set dev $dev master br0 ip addr add $ip_maj/24 dev br0 ip addr add $ip_min/24 dev br0 ip link set dev br0 up ip route add default via $gateway echo "" } set_tap(){ echo Setting tap ip tuntap add tap0 mode tap brctl addif br0 tap0 ip link set dev tap0 up echo "" } show_networking(){ echo -n "Networking follows in 3 seconds..." sleep 3 echo "\n" echo "========================================" echo "========================================" ip -4 link echo "......................" ip -4 addr echo "......................" ip -4 route echo "========================================" echo "========================================" } echo "\nBegin upnet.sh" [ "$use_tap" = "1" ] && [ "$use_bridge" != "1" ] && \ error_tap_without_bridge unset_everything $dev $ipaddr_major $ipaddr_minor $gateway enable_ip_forwarding set_hostname_and_localhost create_phys_device_link $dev $ipaddr_major $ipaddr_minor $gateway [ "$use_bridge" = "1" ] || \ set_phys_device_addr $dev $ipaddr_major $ipaddr_minor $gateway [ "$use_bridge" = "1" ] && set_bridge $dev \ $ipaddr_major $ipaddr_minor $gateway [ "$use_tap" = "1" ] && \ set_tap $dev $ipaddr_major $ipaddr_minor $gateway show_networking
The best way to explain the preceding script is to turn its main routine, the part that isn't contained in functions, into pseudocode:
Conceptually it's very simple.
Pay special attention to the following two lines in the set_bridge function:
ip link add name br0 type bridge ip link set dev $dev master br0
The preceding two lines establish the connection, on the host, between the physical network card and the software bridge.
The following is the block diagram of the system defined in the section called TL;DR Version:
Folks, you saw it here first. As of 2/23/2021 I couldn't find a single other block diagram of a LAN-peer VM guest in a host, anywhere on the Internet. Hundreds of ambiguous, anecdotal and often contradictory documents, but not one block diagram. If you ask me, documenting a Qemu LAN-peer VM guest without a diagram is a sure-fire way to cause mass confusion.
In the diagram, black entities are physical, and red entities are implemented in software. Later in this document the diagram will be thoroughly explained, and still later a clickable version of the diagram will be displayed, but I just wanted to illustrate the state of current (2/23/2021) Qemu LAN-peer documentation, and its effect on the general Tech public's understanding of Qemu LAN-peer VMs.
The following are some other Qemu LAN-peer documentation failures seen widely across the Internet in both documents and answers to questions:
The Internet being the Internet, it's a certainty that people will say this document is nonsense and all wrong. My response is simply this: I've tech edited everything in this document, and at least as of 2/23/2021, it all works. And it's simple and reproducible.
Critics will also say I covered only one architecture, the host's software bridge linked to the VM guest's virtio-net-pci device. Very true. The reason is that this is the simplest way to create a LAN-peer VM guest. With the current state of documentation, by far the greatest challenge in creating a LAN-peer VM guest is getting something, anything, to work. From there, a little stepwise experimentation can get you the exact setup you want, so starting with this document you can develop any LAN-peer architecture you want.
Some will say that this section is a rant. They're right. After spending five days in experimentation, asking questions on multiple venues, reviewing tens of online docs, I'm angry that such a simple thing as this was so complicated to learn, and so unknown to the general Tech public.
Some will say that I'm stupid for spending those five days, instead of learning it in a half a day. OK-fine, I'm stupid, but even stupid people deserve to be able to create LAN-peer VM guests. And as stupid as I appear to be, at least I created a block diagram of the situation and differentiated between host and VM guest. These are things most of the smart people forgot to do.
This section of this document was indeed a rant. The rest of the document will be just-the-facts information.
First, let's review the block diagram for the bridge based LAN-peer:
First, a definition. The bridge br0 with id mybridge0 is a software bridge, which is simply a software implementation of a software bridge. A software bridge is something like a network switch or hub in that it connects two distinct IP address ranges into one network. Another way of saying the same thing is that it splits one network into two distinct IP address ranges. So when you think of br0, visualize a software version of a physical box with lots of Ethernet ports, each of which is capable of connection to a network device, either physical or software.
In the preceding block diagram, the connection between the physical network card and bridge br0 is created completely and exclusively by the in host's networking configuration. In the case of Void Linux, it's created by the following two lines of the networking shellscript, where $dev is the name of the host's physical networking device, which on my computer is enp40s0:
ip link add name br0 type bridge ip link set dev $dev master br0
In the preceding code, the first line creates the bridge, and the second line connects the bridge to the host's physical network device.
Now let's turn to the connection between the host's bridge and the VM guest's eth0. Observe the following lines from the host's VM guest creation shellscript:
-netdev bridge,id=mybridge0,br=br0 \ -device virtio-net-pci,netdev=mybridge0 \
In the preceding code, the first line creates a software bridge on the host. The second line creates a software virtual network device on the VM guest, and connects that software virtual networking device to the bridge by means of a netdev value that matches the software bridge's id value.
The matching of a device on the VM guest and host is called "peering", so that the two devices are said to be peers of each other. If you ever get a warning, it means you haven't defined such a connection. Such a warning is serious, and if you see it, your VM guest won't be networked the way you hoped it would.
NOTE:
Please don't confuse the two very different uses of the word "peer" in this document. A LAN-peer is a VM guest that, from a networking standpoint, appears to be just another physical computer on the LAN. A peered pair of software devices is a pair with one software device in the VM guest and the other in the host. Each software device is said to be a "peer" of the other.
Notice there's no reference to enp40s0 in the preceding 2 lines of shellscript. Through Qemu magic, it figures out your configured physical network device, and connects that to the software bridge. I don't know a way to specify which physical device on the host to connect to the bridge. This is a piece of ambiguity in this document.
The following is the basic architecture of a VM guest created by a host, and please notice that you can click on any boxes or wide arrows to get more information in the box below the block diagram:
This section lists required packages and other prerequisites to create a Qemu constructed LAN-peer. Please note that the package names I give are for Void Linux. Your distro might have differently named packages.
On the host you need the bridge-utils package if you want to use the brctl program.
The Qemu documentation at https://wiki.qemu.org/Documentation/Networking says you need to do several things in order to run qemu-system-x86_64 as a non-root user and still be able to ping. On my setup, I've found these things to be unnecessary, probably because I don't use SLIRP networking, which comes into play when you use -netdev user in your host's shellscript to start your VM guest.
Obviously, on both the host and VM guest, you need the packages necessary to give you the ip and the various qemu commands.
One big disadvantage of a LAN-peer VM guest is it's right there on the LAN, not hiding behind NAT, which would be more secure. So you need to use a good firewall on your VM guest.
The Internet is rife with ambiguous, anecdotal and contradictory information about using Qemu to create a LAN-peer VM guest. This turns creating a LAN-peer VM guest into a bunch of experimentation, web search and asking people. It doesn't have to be that way.
This document is almost completely unambigous its Qemu LAN-peer VM guest creation instructions. Use it to create the simplest Qemu LAN-peer VM guest, and then step by step grow it into the exact setup you need.
If you like this page, please recommend it to your friends. If you find an error in this page (not a difference in priorities or a rant, but an error), please email me. Also, if you find any ambiguities in this document, please email me. I've ragged on the ambiguities in other documents, so I certainly don't want any in this document.
Last but not least, ponder the following block diagram to understand how simple LAN-peer VM guest creation really is, and don't despair from the inadequate info on the Internet.
[ Training | Troubleshooters.Com | Email Steve Litt ]
enp40s0 is the name of the HOST's physical network device (network card) in these examples. The name will probably be different on your HOST, so substitute as appropriate. On one side enp40s0 connects physically to an Ethernet cable on a LAN whose Internet gateway is 192.168.0.1. On the other end it connects via software to software bridge br0 on the HOST. Hover the arrow between enp40s0 and br0 to see how they are connected.
This is the Qemu software interface between HOST hardware network device enp40s0 and HOST software bridge br0. This connection is established either via a shellscript on the HOST (I call my shellscript upnet.sh) or a declarative syntax on the HOST such as (on Devuan, for instance) /etc/network/interfaces. The following two lines of shellscript code in upnet.sh create that connection:
ip link add name br0 type bridge ip link set dev enp40s0 master br0
Establishing the connection, however, is not enough. The HOST's br0 must also be given IP addresses and a default route, and set to an "on" condition. Once again, the HOST's upnet.sh provides the necessary, with the following lines of code:
ip addr add 192.168.0.2/24 dev br0 ip addr add 192.168.0.102/24 dev br0 ip link set dev br0 up ip route add default via 192.168.0.1
In the preceding shellscript code, 192.168.0.2 is the main IP address for enp40s0, 192.168.0.102 is an additional IP address for enp40s0, and 192.168.0.1 is the LAN's gateway to the Internet.
WARNING: Physical HOST network device should NOT be given IP addresses or routes in a software bridge configuration! The entirety of upnet.sh can be seen in section Host Networking Script.
br0 is a software bridge, on the HOST, connected on one end to the HOST's physical network card/device, in this case enp40s0, and on the other end to the VM GUEST's eth0 software network device. A bridge is sort of like a physical router or a hub: Lots of network devices can connect to each other though it. But in this case the bridge is software, not physical. Hover either the arrow coming out of br0 to see how the connection is established.
This software interface between HOST br0 and GUEST eth0 is created by Qemu according to the HOST's shellscript to launch the GUEST.
#!/bin/sh dvddir=/scratch/linuxinst/devuan/devuan_beowulf/installer-iso qemu-system-x86_64 --enable-kvm \ -cdrom $dvddir/devuan_beowulf_3.0.0_amd64-desktop.iso \ -hda /scratch/qemu_images/beowulf.disk \ -m 4G \ -boot c \ -netdev bridge,id=mybridge0,br=br0 \ -device virtio-net-pci,netdev=mybridge0 \
Note that the final blank line is mandatory because of the line continuations. The networking is established by the last two lines:
-netdev bridge,id=mybridge0,br=br0 \ -device virtio-net-pci,netdev=mybridge0 \
The first line gives the GUEST knowledge of software bridge br0, and gives it an id of "mybridge0". The second line declares a GUEST network device of type virtio-net-pci, which becomes eth0 on the GUEST. The part declaring "netdev=mybridge0" is what causes the connection between br0 and eth0.
This is a software equivalent of a network card, on the GUEST. In its simplest form, its network address(s), default route, and DNS are governed by DHCP. To give it a static IP, see TL;DR Version, Static IP.
This web page, nobs.htm, is constructed indirectly because it must include code from an SVG file. The nobs.htm file is created by running some AWK scripts on nobs_template.htm and redirecting the output to nobs.htm. You can read information about this process at http://troubleshooters.com/codecorn/svg_clickmaps/landmines.htm#imgfails, which also gives listings of the two AWK scripts needed for building this page. In summary, you need to:
If you have any questions at all, thoroughly review section http://troubleshooters.com/codecorn/svg_clickmaps/landmines.htm#imgfails, and if you still have questions, review the entire web page http://troubleshooters.com/codecorn/svg_clickmaps/landmines.htm
In the block diagram above, click any box or thick arrow to find out its function, usage, and how it's configured.
Remainder of article is below this box.