Setting up Open vSwitch with ipsec_gre

By , 21/12/2015 01:45

Lately I have been playing a lot with Docker and one of the things I want to try out is the multi-host network and perhaps even swarm support. To be able to do that however you need to expose the Docker daemon on your host machine. That’s all fine and dandy when you are working on a private network but what happens if you want to do this with hosts on multiple locations separated by public links? Even though you can secure the Docker daemon with TLS I don’t feel comfortable having service ports out in the open like that so I want to set up my own private network.

While I have set up OpenVPN links for these kind of things in the past I would now like something more flexible (easy adding of nodes, mesh topology, STP, …) so I started looking at Open vSwitch but I could not find a complete guide. Especially the part on using certificates instead of the pre-shared keys is lacking so I pieced this procedure together from different sources.

I performed these steps on two Debian Jessie hosts but you should be able to do this on any Linux distribution as long as you can get Open vSwitch installed. There are plenty of guides on that, including if you want a newer version than what is provided by the official repositories. Do this on both hosts.

  1. # apt-get install openvswitch-common openvswitch-switch openvswitch-ipsec

You will get a notice on the Racoon configuration, I chose the direct configuration.

Manual insecure setup

The first thing I tried to do was to setup a link without any security using just cli commands. Out of scope for this procedure is that you need to enable communication between the hosts on port 500 in your firewall configuration.

Perform these commands on host 1. I named the switch insecure, replace the remote_ip with your own.

  1. # ovs-vsctl add-br insecure
  2. # ip link set insecure up
  3. # ip addr add broadcast dev insecure
  4. # ovs-vsctl add-port insecure gre0 — set interface gre0 type=gre options:remote_ip=

Perform these commands on host 2, again replace the remote_ip with your own.

  1. # ovs-vsctl add-br insecure
  2. # ip link set insecure up
  3. # ip addr add broadcast dev insecure
  4. # ovs-vsctl add-port insecure gre0 — set interface gre0 type=gre options:remote_ip=

Now you should be able to ping between the hosts, if not you should investigate your firewall configuration. You can also use ovs-dpctl to do some debugging on the link. Note that you may experience issues if you try to actually do anything useful on this link like copying files. This is because the link MTU is not setup correctly yet.

To clean up this test perform this action on both hosts.

  1. # ovs-vsctl rem-br insecure

Automatic insecure setup

Now let’s create config files that create a link that actually works on every boot. Put this in /etc/network/interfaces on Host 1:

  1. allow-ovs insecure
  2. iface insecure inet static
  3.     address
  4.     netmask
  5.     ovs_type OVSBridge
  6.     ovs_ports gre1
  7.     ovs_extra set bridge ${IFACE} stp_enable=true
  8.     mtu 1400
  10. # Make GRE start together with OVS interface
  11. allow-insecure gre1
  12. # Starting from gre1 so that interfaces are not shown in ifconfig
  13. iface gre1 inet manual
  14.     ovs_type OVSPort
  15.     ovs_bridge insecure
  16.     ovs_extra set interface ${IFACE} type=gre options:remote_ip=

Create an equivalent file on Host 2. For the interfaces to come up nicely at reboot you also need to add this to /etc/default/openvswitch-switch:

  1. OVS_CTL_OPTS=‘–delete-bridges’

Reboot both machines and you should be able to ping between the hosts from the start. It might take a few seconds before the connection is initiated though.

You’ll notice 2 added things compared to the manual setup:

  • stp_enable enables the Spanning Tree Protocol which will reroute traffic when you have multiple hosts set-up and one breaks down. It also disables links that are not needed to prevent cyclic routing.
  • mtu 1400 lowers the MTU so that the overhead of GRE + IPSEC doesn’t result in packet fragmentation which will slow down your link or even result in packages not arriving

A little bit more secure

It is very easy to add a basic security to this setup. Just edit the GRE port in /etc/network/interfaces:

  1.    ovs_extra set interface ${IFACE} type=ipsec_gre options:remote_ip= options:psk=secret

Restart the interfaces or reboot an you will have encrypted traffic, you can verify with tcpdump if you like. Of course this type of setup is still not very secure because of the pre-shared key, but it is better than nothing!

Even more secure

The final action to take is to get rid of the pre-shared key and install certificates on the servers for authentication.

First generate on both hosts some certificates if you don’t have them yet. Note that these are self-signed and that they will stop working in 30 days. You can increase the time with the -days parameter or put some process in place to update them periodically.

  1. # cd /etc/openvswitch
  2. # openssl req -x509 -newkey rsa:2048 -keyout private.key -out cert.pem -nodes
  3. # openssl rsa –in private.key -out key.pem

Copy over the cert.pem file from the other host in the /etc/openvswitch/peers directory.

Next edit the OVSPort entry in /etc/network/interfaces (note I added some line breaks for readability).

  1.    ovs_extra set interface ${IFACE} type=ipsec_gre \
  2.         options:remote_ip= \
  3.         options:certificate=/etc/openvswitch/cert.pem \
  4.         options:private_key=/etc/openvswitch/key.pem \
  5.         options:peer_cert="`cat /etc/openvswitch/peers/host2.pem`"

For this to work I had to edit /etc/network/if-pre-up.d/openvswitch script slightly, there is an issue with how the options are evaluated by Bash. Find the entry for OVSPort and change the script like this:

  1.        OVSPort)
  2.                 eval ovs_vsctl — –may-exist add-port "${IF_OVS_BRIDGE}"\
  3.                     "${IFACE}" ${IF_OVS_OPTIONS} \
  4.                     ${IF_OVS_EXTRA+– $IF_OVS_EXTRA}

Restart your network interface or reboot and you should have an Open vSwitch setup with a GRE tunnel protected by an IPSEC configuration based on certificates.


I got most of the information from these pages:


Flattr this!

Leave a Reply


Panorama Theme by Themocracy