Open Source Routing GRE over IPSec with StrongSwan and Cisco IOS-XE

In my previous post about the Ansible Playbook for VyOS and BGP Routing, I wrote that I was looking for some Open Source alternatives for software routers to use in AWS Transit VPCs.

Here is the example using a Debian Linux,  FRR (Free Range Routing) and StrongSwan connecting over a GRE over IPSec tunnel to a Cisco IOS-XE (CSRv) router:

You can find the Vagrantfile in my Github repo https://github.com/berndonline/debian-router-vagrant. During the boot Ansible runs and pre-configures both nodes but continue reading about the detailed configuration:

sudo apt-get update
sudo apt-get upgrade -y

Enable IP routing by adding the following line to /etc/sysctl.conf:

sudo vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
sudo sysctl -p /etc/sysctl.conf

Download the latest FRR release for Debian 9 x86_64 from https://github.com/FRRouting/frr/releases

Install FRR and don’t worry about any dependency errors from the first command, the second command will install the missing dependencies. Next, enable the needed FRR daemons and start the service:

wget https://github.com/FRRouting/frr/releases/download/frr-3.0.3/frr_3.0.3-1_debian9.1_amd64.deb
wget https://github.com/FRRouting/frr/releases/download/frr-3.0.3/frr-pythontools_3.0.3-1_debian9.1_all.deb
wget https://github.com/FRRouting/frr/releases/download/frr-3.0.3/frr-doc_3.0.3-1_debian9.1_all.deb
sudo dpkg -i frr_3.0.3-1_debian9.1_amd64.deb frr-pythontools_3.0.3-1_debian9.1_all.deb frr-doc_3.0.3-1_debian9.1_all.deb
sudo apt-get install -f -y

sudo bash -c 'cat << EOF > /etc/frr/daemons
zebra=yes
bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
EOF'

sudo bash -c 'cat << EOF > /etc/frr/frr.conf
!
frr version 3.0.3
frr defaults traditional
no ipv6 forwarding
!
router bgp 65001
 neighbor 192.168.0.2 remote-as 65002
 !
 address-family ipv4 unicast
  network 10.255.0.1/32
 exit-address-family
 vnc defaults
  response-lifetime 3600
  exit-vnc
!
line vty
!
EOF'

sudo systemctl enable frr
sudo systemctl start frr

Install StrongSwan and change a few settings before you can enable and start the service:

sudo apt-get install -y strongswan-swanctl charon-systemd

sudo bash -c 'cat << EOF > /etc/strongswan.d/charon/connmark.conf
connmark {
 
    # Disable connmark plugin
    # Needed for tunnels - see https://wiki.strongswan.org/projects/strongswan/wiki/RouteBasedVPN
    load = no
 
}
EOF'
sudo bash -c 'cat << EOF > /etc/strongswan.d/charon.conf
charon {
 
    # Cisco IKEv2 wants to use reauth - need to set make_before_break otherwise
    # strongSwan will have a very brief outage during IKEv2 reauth
    make_before_break = yes
 
    # Needed for tunnels - see https://wiki.strongswan.org/projects/strongswan/wiki/RouteBasedVPN
    install_routes = no
 
}
EOF'

sudo systemctl enable strongswan-swanctl
sudo systemctl start strongswan-swanctl

Setting TCP MSS to path MTU with iptables:

sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -q iptables-persistent

sudo bash -c 'cat << EOF > /etc/iptables/rules.v4
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
COMMIT
EOF'

Let us continue with the Debian router interface configuration, here you also find the GRE tunnel settings:

sudo bash -c 'cat << EOF > /etc/network/interfaces
source-directory /etc/network/interfaces.d
auto lo
iface lo inet loopback

auto lo:1
iface lo:1 inet static
      address 10.255.0.1
      netmask 255.255.255.255

auto ens5
iface ens5 inet dhcp

auto ens6
iface ens6 inet static
      address 10.0.0.1
      netmask 255.255.255.0

auto gre1
iface gre1 inet tunnel
      address 192.168.0.1
      netmask 255.255.255.0
      mode gre
      endpoint 10.0.0.2
EOF'

sudo systemctl restart networking

In StrongSwan you configure the IPSec settings:

sudo bash -c 'cat << EOF > /etc/swanctl/swanctl.conf
connections {
    my-vpn {
        remote_addrs = 10.0.0.2
        version = 1
        proposals = aes256-sha1-modp1536
        reauth_time = 1440m
        local-1 {
            auth = psk
            id = debian-router.domain.com
        }
        remote-1 {
            # id field here is inferred from the remote address
            auth = psk
        }
        children {
            my-vpn-1 {
                local_ts = 10.0.0.1/32[gre]
                remote_ts = 10.0.0.2/32[gre]
                mode = transport
                esp_proposals = aes128-sha1-modp1536
                rekey_time = 60m
                start_action = trap
                dpd_action = restart
            }
        }
    }

}
secrets {
    ike-my-vpn-1 {
        id-1 = cisco-iosxe.domain.com
        id-2 = 10.0.0.2
        secret = "secret"
    }
}
EOF'

sudo systemctl restart strongswan-swanctl

We finished the Debian host configuration and continue with the Cisco  router configuration to connect the Debian router to the tunnel 0 interface on the Cisco router:

conf t
hostname cisco-iosxe

crypto keyring my-keyring  
  pre-shared-key address 10.0.0.1 key secret
  exit

crypto isakmp policy 20
 encr aes 256
 authentication pre-share
 group 5
crypto isakmp identity hostname
crypto isakmp profile my-isakmp-profile
   keyring my-keyring
   match identity host debian-router.domain.com
   exit
crypto ipsec transform-set my-transform-set esp-aes esp-sha-hmac
 mode transport
 exit
crypto ipsec profile my-ipsec-profile
 set transform-set my-transform-set
 set pfs group5
 set isakmp-profile my-isakmp-profile
 exit

interface Loopback1
 ip address 10.255.0.2 255.255.255.255
 exit

interface Tunnel0
 ip address 192.168.0.2 255.255.255.0
 tunnel source GigabitEthernet2
 tunnel destination 10.0.0.1
 tunnel protection ipsec profile my-ipsec-profile
 no shut
 exit

interface GigabitEthernet2
 ip address 10.0.0.2 255.255.255.0
 no shut
 exit

router bgp 65002
 bgp log-neighbor-changes
 neighbor 192.168.0.1 remote-as 65001
 address-family ipv4
  network 10.255.0.2 mask 255.255.255.255
  neighbor 192.168.0.1 activate
 exit-address-family
 exit

exit
wr mem

Clone my Github repo https://github.com/berndonline/debian-router-vagrant and boot the environment with “./vagrant_up.sh”. After the two VMs are booted wait a few seconds and run the validation playbook to check the connectivity between the nodes:

berndonline@lab:~/debian-router-vagrant$ ansible-playbook ./validate_connectivity.yml

PLAY [debian-router] ***********************************************************************************************************************************************

TASK [check connectivity from debian router] ***********************************************************************************************************************
changed: [debian-router]

PLAY [cisco-iosxe] *************************************************************************************************************************************************

TASK [check connectivity from cisco iosxe] *************************************************************************************************************************
ok: [cisco-iosxe]

PLAY RECAP *********************************************************************************************************************************************************
cisco-iosxe                : ok=1    changed=0    unreachable=0    failed=0
debian-router              : ok=1    changed=1    unreachable=0    failed=0

berndonline@lab:~/debian-router-vagrant$

I will continue improving the config, and do some more testing with AWS VPN gateways (VGW).

Please share your feedback.

Leave a comment

BGP EVPN and VXLAN with Cumulus Linux

I did some updates on my Cumulus Linux Vagrant topology and added new functions to my post about an Ansible Playbook for the Cumulus Linux BGP IP-Fabric.

To the Vagrant topology, I added 6x servers and per clag-pair, each server is connected to a VLAN and the second server is connected to a VXLAN.

Here are the links to the repositories where you find the Ansible Playbook https://github.com/berndonline/cumulus-lab-provision and the Vagrantfile https://github.com/berndonline/cumulus-lab-vagrant

In the Ansible Playbook, I added BGP EVPN and one VXLAN which spreads over all Leaf and Edge switches. VXLAN routing is happening on the Edge switches into the rest of the virtual data centre network.

Here is an example of the additional variables I added to edge-1 for BGP EVPN and VXLAN:

group_vars/edge.yml:

clagd_vxlan_anycast_ip: 10.255.100.1

The VXLAN anycast IP is needed in BGP for EVPN and the same IP is shared between edge-1 and edge-2. The same is for the other leaf switches, per clag pair they share the same anycast IP address.

host_vars/edge-1.yml:

---

loopback: 10.255.0.3/32

bgp_fabric:
  asn: 65001
  router_id: 10.255.0.3
  neighbor:
    - swp51
    - swp52
  networks:
    - 10.0.4.0/24
    - 10.255.0.3/32
    - 10.255.100.1/32
    - 10.0.255.0/28
  evpn: true
  advertise_vni: true

peerlink:
  bond_slaves: swp53 swp54
  mtu: 9216
  vlan: 4094
  address: 169.254.1.1/30
  clagd_peer_ip: 169.254.1.2
  clagd_backup_ip: 192.168.100.4
  clagd_sys_mac: 44:38:39:FF:40:94
  clagd_priority: 4096

bridge:
  ports: peerlink vxlan10201
  vids: 901 201

vlans:
  901:
    alias: edge-transit-901
    vipv4: 10.0.255.14/28
    vmac: 00:00:5e:00:09:01
    pipv4: 10.0.255.12/28
  201:
    alias: prod-server-10201
    vipv4: 10.0.4.254/24
    vmac: 00:00:00:00:02:01
    pipv4: 10.0.4.252/24
    vlan_id: 201
    vlan_raw_device: bridge

vxlans:
  10201:
    alias: prod-server-10201
    vxlan_local_tunnelip: 10.255.0.3
    bridge_access: 201
    bridge_learning: 'off'
    bridge_arp_nd_suppress: 'on'

On the Edge switches, because of VXLAN routing, you find a mapping between VXLAN 10201 to VLAN 201 which has VRR running.

I needed to do some modifications to the interfaces template interfaces_config.j2:

{% if loopback is defined %}
auto lo
iface lo inet loopback
    address {{ loopback }}
{% if clagd_vxlan_anycast_ip is defined %}
    clagd-vxlan-anycast-ip {{ clagd_vxlan_anycast_ip }}
{% endif %}
{% endif %}
...
{% if bridge is defined %}
{% for vxlan_id, value in vxlans.items() %}
auto vxlan{{ vxlan_id }}
iface vxlan{{ vxlan_id }}
    alias {{ value.alias }}
    vxlan-id {{ vxlan_id }}
    vxlan-local-tunnelip {{ value.vxlan_local_tunnelip }}
    bridge-access {{ value.bridge_access }}
    bridge-learning {{ value.bridge_learning }}
    bridge-arp-nd-suppress {{ value.bridge_arp_nd_suppress }}
    mstpctl-bpduguard yes
    mstpctl-portbpdufilter yes

{% endfor %}
{% endif %}

There were also some modifications needed to the FRR template frr.j2 to add EVPN to the BGP configuration:

...
{% if bgp_fabric.evpn is defined %}
 address-family ipv6 unicast
  neighbor fabric activate
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor fabric activate
{% if bgp_fabric.advertise_vni is defined %}
  advertise-all-vni
{% endif %}
 exit-address-family
{% endif %}
{% endif %}
...

For more detailed information about EVPN and VXLAN routing on Cumulus Linux, I recommend reading the documentation Ethernet Virtual Private Network – EVPN and VXLAN Routing.

Have fun testing the new features in my Ansible Playbook and please share your feedback.

Leave a comment

Ansible Playbook for Cisco BGP Routing Topology

This is my Ansible Playbook for a simple Cisco BGP routing topology and using a CICD pipeline for integration testing. The virtual network environment is created on-demand by using Vagrant, see my post about Cisco IOSv and XE network simulation using Vagrant.

Network overview:

Here’s my Github repository where you can find the complete Ansible Playbook: https://github.com/berndonline/cisco-lab-provision

You can find all the variables for the interface and routing configuration under host_vars. Below is an example for router rtr-1:

---

hostname: rtr-1
domain_name: lab.local

loopback:
  address: 10.255.0.1
  mask: 255.255.255.255

interfaces:
  0/1:
    alias: connection rtr-2
    address: 10.0.255.1
    mask: 255.255.255.252

  0/2:
    alias: connection rtr-3
    address: 10.0.255.5
    mask: 255.255.255.252

bgp:
  asn: 65001
  neighbor:
    - {address: 10.0.255.2, remote_as: 65000}
    - {address: 10.0.255.6, remote_as: 65000}
  networks:
    - {network: 10.0.255.0, mask: 255.255.255.252}
    - {network: 10.0.255.4, mask: 255.255.255.252}
    - {network: 10.255.0.1, mask: 255.255.255.255}
  maxpath: 2

Roles:

  • Hostname: The task in main.yml uses the Ansible module ios_system and configures hostname, domain name and disables dns lookups.
  • Interfaces: This role uses the Ansible module ios_config to deploy the template interfaces.j2 to configure the interfaces. In the main.yml is a second task to enable the interfaces when the previous template applied the configuration.
  • Routing: Very similar to the interfaces role and uses also the ios_config module to deploy the template routing.j2 for the BGP routing configuration.

Main Ansible Playbook site.yml:

---

- hosts: all

  connection: local
  user: vagrant
  gather_facts: 'no'

  roles:
    - hostname
    - interfaces
    - routing

When a change triggers the gitlab-ci pipeline it spins up the Vagrant instances and executes the main Ansible Playbook:

After the main site.yml ran, a second Playbook is executed for basic connectivity testing cisco_check_icmp.yml. This uses the Ansible module ios_ping and can be useful in my case to validate if the configuration was correctly applied:

If everything goes well, like in this example, the job is successful:

I will continue to improve the Playbook and the CICD pipeline so come back later to check it out.

Leave a comment

Ansible Playbook for Cumulus Linux BGP IP-Fabric and Cumulus NetQ Validation

This is my Ansible Playbook for a Cumulus Linux BGP IP-Fabric using BGP unnumbered and Cumulus NetQ to validate the configuration in a CICD pipeline. I use the same CICD pipeline from my previous post about Continuous Integration and Delivery for Networking with Cumulus Linux but added the Cumulus NetQ validation in the production stage to check BGP and CLAG configuration.

Network overview:

Here’s my Github repository where you find the complete Ansible Playbook: https://github.com/berndonline/cumulus-lab-provision

The variables are split between group_vars and host_vars. Still need to see if I can find a better way for the variables because interface settings for spine and edge switches are in group_vars, and for leaf switches the interface configuration is per host in host_vars. Not ideal at the moment, it should be the same for all devices.

Roles:

  • Hostname: This task changes the hostname
  • Interfaces: This creates the interfaces and bridge (only leafs and edges) configuration. The task uses templates interfaces.j2 and interfaces_config.j2 to create the configuration files under /etc/network/…
  • Routing: The template frr.j2 creates the FRR (Free Range Routing) configuration file. FRR replaces Quagga since Cumulus Linux version 3.4.x
  • PTM: Uses as well an template topology.j2 to generate the topology file for the Prescriptive Topology Manager (PTM)
  • NTP: Ntp and timezone settings

In most of the cases I use Jinja2 templates to generate configuration files. The site.yml is otherwise very simple. It executes the different roles, and triggers the handlers if a change is made by a role.

---

- hosts: network
  strategy: free

  user: cumulus
  become: 'True'
  gather_facts: 'False'

  handlers:
    - name: reload networking
      command: "{{item}}"
      with_items:
        - ifreload -a
        - sleep 10

    - name: reload frr
      service: name=frr state=reloaded

    - name: apply hostname
      command: hostname -F /etc/hostname

    - name: restart netq agent
      command: netq config agent restart

    - name: reload ptmd
      service: name=ptmd state=reloaded

    - name: apply timezone
      command: /usr/sbin/dpkg-reconfigure --frontend noninteractive tzdata

    - name: restart ntp
      service: name=ntp state=restarted

  roles:
    - hostname
    - interfaces
    - routing
    - ptm
    - ntp

Like mentioned in previous posts, I use Gitlab-CI for my Continuous Integration / Continuous Delivery (CICD) pipeline to simulate changes against a virtual Cumulus Linux network using Vagrant. You can find more information about the pipeline configuration in the .gitlab-ci.yml.

Changes in the staging branch will spin-up the Vagrant environment but only executes the the Ansible Playbook:

Cumulus NetQ configuration validation in production:

The production stage in the pipeline spins-up the Vagrant environment and executes the Ansible Playbook, then continues executing the two NetQ checks netq_check_bgp.yml and netq_check_clag.yml to validate the BGP and CLAG configuration:

The result will look like this when all stages finish successfully:

I will continue to improve the Playbook and the CICD pipeline so come back later to check it out.

In my repository I have some other useful Playbooks for config backup and restore but also to collect and remove cl-support.

config_backup.yml

config_restore.yml

cl-support_get.yml

cl-support_remove.yml

Please tell me if you like it and share your feedback.

See my new post about BGP EVPN and VXLAN with Cumulus Linux

Leave a comment

VMware NSX Edge Routing

I recently deployed VMware NSX (Software defined Network) in our datacentre.

About the NSX Edge cluster there are some some specific requirements when it comes to physical connectivity. All the information you find as well in the VMware NSX reference design guide.

On Cumulus Linux side I am using BGP in Quagga and the traffic is distributed via ECMP (Equal-cost multi-path) over multiple Edge nodes within NSX.

See below the overview:

Very important to have an dedicated VLAN per core switch to the Edge Nodes. In my tests it didn’t work with a shared VLAN via the Cumulus core, the BGP neighbor relationships were correctly established but there was a problem with the packet forwarding via the Peerlink.

Here the example Quagga BGP config from spine-1:

router bgp 65001 vrf vrf-nsx
 neighbor 10.100.254.1 remote-as 65002
 neighbor 10.100.254.1 password verystrongpassword!!
 neighbor 10.100.254.1 timers 1 3
 neighbor 10.100.254.2 remote-as 65002
 neighbor 10.100.254.2 password verystrongpassword!!
 neighbor 10.100.254.2 timers 1 3
 neighbor 10.100.254.3 remote-as 65002
 neighbor 10.100.254.3 password verystrongpassword!!
 neighbor 10.100.254.3 timers 1 3
 neighbor 10.100.254.4 remote-as 65002
 neighbor 10.100.254.4 password verystrongpassword!!
 neighbor 10.100.254.4 timers 1 3
 neighbor 10.100.255.2 remote-as 65001
 neighbor 10.100.255.2 password verystrongpassword!!

 address-family ipv4 unicast
  network 0.0.0.0/0
  neighbor 10.100.254.1 route-map bgp-in in
  neighbor 10.100.254.2 route-map bgp-in in
  neighbor 10.100.254.3 route-map bgp-in in
  neighbor 10.100.254.4 route-map bgp-in in
  neighbor 10.100.255.2 next-hop-self
  neighbor 10.100.255.2 route-map bgp-in in
 exit-address-family

ip route 0.0.0.0/0 10.100.255.14 vrf vrf_prod-nsx

access-list bgp-in permit 10.100.0.0/17

route-map bgp-in permit 10
 match ip address bgp-in

The second core switch, spine-2 looks exactly the same only different IP addresses are used.

More about my experience with VMware NSX will follow soon.