[Qemu-discuss] Port-forwarding to a QEMU/KVM VM

[Jaap Winius]

Hi folks,

For years I've maintained Debian Linux servers that run QEMU/KVM
virtual machines along with ppp, bridge-utils (brctl) and iptables. In
these cases it is simple to configure the latter to forward packets
from the Internet, arriving on interface ppp0, over to VMs on the
internal bridged interface, br0. This interface is configured like:

auto enp35s0
iface enp35s0 inet manual

auto br0
iface br0 inet static
address 10.1.1.5/24
gateway 10.1.1.1
bridge_ports enp35s0
bridge_stp off
bridge_fd 0
bridge_ageing 0
bridge_maxwait 2

The relevant iptables rules I use to forward HTTPS traffic on to the
VM, 10.1.1.10, look like:

iptables -t nat -A PREROUTING -i ppp0 \
-p tcp --dport 443 -j DNAT --to 10.1.1.10:443

iptables -A FORWARD -i ppp0 \
-p tcp -d 10.1.1.10 --dport 443 --syn -m state --state NEW -j ACCEPT

However, this forwarding configuration stopped working after ppp and
iptables were moved to a physically separate gateway machine. Now the
packets from outside are still forwarded on to the VM (that uses a
virtio network interface), which responds, but the replies never make
it out of the bridged network segment. Using tcpdump, the reply
packets can be detected on the VM and the host server, but not on the
gateway.

How can port-forwarding functionality best be restored in this case?

To be honest, this problem seems more like something to do with brctl
than with QEMU/KVM, but as brctl appears to be the bridge of choice in
these environments, surely someone here has already encountered this
problem and found a fix for it. And as I'm rather stumped on this one,
I'd be very grateful if someone were to share their solution here.

Thanks

Jaap

[Jakob Bohm]

Post by Jaap Winius
Hi folks,
For years I've maintained Debian Linux servers that run QEMU/KVM
virtual machines along with ppp, bridge-utils (brctl) and iptables. In
these cases it is simple to configure the latter to forward packets
from the Internet, arriving on interface ppp0, over to VMs on the
  auto enp35s0
  iface enp35s0 inet manual
  auto br0
  iface br0 inet static
        address 10.1.1.5/24
        gateway 10.1.1.1
        bridge_ports enp35s0
        bridge_stp off
        bridge_fd 0
        bridge_ageing 0
        bridge_maxwait 2
The relevant iptables rules I use to forward HTTPS traffic on to the
  iptables -t nat -A PREROUTING -i ppp0 \
    -p tcp --dport 443 -j DNAT --to 10.1.1.10:443
  iptables -A FORWARD -i ppp0 \
    -p tcp -d 10.1.1.10 --dport 443 --syn -m state --state NEW -j ACCEPT
However, this forwarding configuration stopped working after ppp and
iptables were moved to a physically separate gateway machine. Now the
packets from outside are still forwarded on to the VM (that uses a
virtio network interface), which responds, but the replies never make
it out of the bridged network segment. Using tcpdump, the reply
packets can be detected on the VM and the host server, but not on the
gateway.
How can port-forwarding functionality best be restored in this case?
To be honest, this problem seems more like something to do with brctl
than with QEMU/KVM, but as brctl appears to be the bridge of choice in
these environments, surely someone here has already encountered this
problem and found a fix for it. And as I'm rather stumped on this one,
I'd be very grateful if someone were to share their solution here.

Check the following:

Your VM MAC addresses are unique network wide and don't have
the multicast bit set (so packets can be routed correctly by
your layer 2 Ethernet switches).

Your IP tables allow ARP requests (for IPv4) and neighbor
discovery ICMPv6 (for IPv6) to travel between your VMs and
the physical network (so other machines, including the
gateway can find the IP-to-MAC mapping for your VMs).

Your physical Ethernet switch is configured to accept that
the cable to your machine behaves like a cable to another
Ethernet switch (this is the default for most switches with
a few annoying exceptions).

If you have a spare machine with two ethernet ports, configure
it to bridge the two ports with brctl and run wireshark on that,
then "splice" it into the connection between your machine and the
physical switch, then again between the physical switch and the
gateway.  Look if the packets are dropped on one side of the
physical switch or the other.  Also look for any Layer 2 and
layer 3 packets that negotiate the routing of IP and MAC
addresses.


Enjoy

Jakob

[Jaap Winius]

....

Post by Jakob Bohm
If you have a spare machine with two ethernet ports, configure
it to bridge the two ports with brctl and run wireshark on that,
then "splice" it into the connection between your machine and the
physical switch, then again between the physical switch and the
gateway.  Look if the packets are dropped on one side of the
physical switch or the other. ...

I first tried sniffing on the gateway-side of the switch, but just saw
the same thing. Then I did it on the server-side... and there saw that
the reply packets were making it out of the bridge! After that I
discovered the root cause of the problem: the default gateway on the
VM was still pointing to the wrong address. That was why the internal
tests were working, but not the external one. So, problem solved.

Thanks for your help!

Cheers,

Jaap