Like promised, here a basic Ansible Playbook for a Cumulus Linux Layer 3 Fabric running BGP which you see in large-scale data centre deployments.
You push the layer 2 network as close as possible to the server and use ECMP (Equal-cost multi-path) routing to distribute your traffic via multiple uplinks.
These kind of network designs are highly scalable and in my example a 2-Tier deployment but you can easily use 3-Tiers where the Leaf switches become the distribution layer and you add additional ToR (Top of Rack) switches.
Here some interesting information about Facebook’s next-generation data centre fabric: Introducing data center fabric, the next-generation Facebook data center network
I use the same hosts file like from my previous blog post Ansible Playbook for Cumulus Linux (Layer 2 Fabric)
Hosts file:
[spine] spine-1 spine-2 [leaf] leaf-1 leaf-2
Ansible Playbook:
---
- hosts: all
remote_user: cumulus
gather_facts: no
become: yes
vars:
ansible_become_pass: "CumulusLinux!"
spine_interfaces:
- { port: swp1, desc: leaf-1, address: "{{ swp1_address}}" }
- { port: swp2, desc: leaf-2, address: "{{ swp2_address}}" }
- { port: swp6, desc: layer3_peerlink, address: "{{ peer_address}}" }
leaf_interfaces:
- { port: swp1, desc: spine-1, address: "{{ swp1_address}}" }
- { port: swp2, desc: spine-2, address: "{{ swp2_address}}" }
handlers:
- name: ifreload
command: ifreload -a
- name: restart quagga
service: name=quagga state=restarted
tasks:
- name: deploys spine interface configuration
template: src=templates/spine_routing_interfaces.j2 dest=/etc/network/interfaces
when: "'spine' in group_names"
notify: ifreload
- name: deploys leaf interface configuration
template: src=templates/leaf_routing_interfaces.j2 dest=/etc/network/interfaces
when: "'leaf' in group_names"
notify: ifreload
- name: deploys quagga configuration
template: src=templates/quagga.conf.j2 dest=/etc/quagga/Quagga.conf
notify: restart quagga
Let’s run the Playbook and see the output:
[root@ansible cumulus]$ ansible-playbook routing.yml -i hosts PLAY [all] ********************************************************************* TASK [deploys spine interface configuration] *********************************** skipping: [leaf-2] skipping: [leaf-1] changed: [spine-2] changed: [spine-1] TASK [deploys leaf interface configuration] ************************************ skipping: [spine-1] skipping: [spine-2] changed: [leaf-2] changed: [leaf-1] TASK [deploys quagga configuration] ******************************************** changed: [leaf-2] changed: [spine-2] changed: [spine-1] changed: [leaf-1] RUNNING HANDLER [ifreload] ***************************************************** changed: [leaf-2] changed: [leaf-1] changed: [spine-2] changed: [spine-1] RUNNING HANDLER [restart quagga] *********************************************** changed: [leaf-1] changed: [leaf-2] changed: [spine-1] changed: [spine-2] PLAY RECAP ********************************************************************* leaf-1 : ok=4 changed=4 unreachable=0 failed=0 leaf-2 : ok=4 changed=4 unreachable=0 failed=0 spine-1 : ok=4 changed=4 unreachable=0 failed=0 spine-2 : ok=4 changed=4 unreachable=0 failed=0 [roote@ansible cumulus]$
To verify the configuration let’s look at the BGP routes on the leaf switches:
root@leaf-1:/home/cumulus# net show route bgp
RIB entry for bgp
=================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,
V - VPN,
> - selected route, * - FIB route
B>* 10.0.0.0/30 [20/0] via 10.0.1.1, swp1, 00:02:14
* via 10.0.1.5, swp2, 00:02:14
B 10.0.1.0/30 [20/0] via 10.0.1.1 inactive, 00:02:14
via 10.0.1.5, swp2, 00:02:14
B 10.0.1.4/30 [20/0] via 10.0.1.5 inactive, 00:02:14
via 10.0.1.1, swp1, 00:02:14
B>* 10.0.2.0/30 [20/0] via 10.0.1.5, swp2, 00:02:14
* via 10.0.1.1, swp1, 00:02:14
B>* 10.0.2.4/30 [20/0] via 10.0.1.1, swp1, 00:02:14
* via 10.0.1.5, swp2, 00:02:14
B>* 10.200.0.0/24 [20/0] via 10.0.1.1, swp1, 00:02:14
* via 10.0.1.5, swp2, 00:02:14
root@leaf-1:/home/cumulus#
root@leaf-2:/home/cumulus# net show route bgp
RIB entry for bgp
=================
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC,
V - VPN,
> - selected route, * - FIB route
B>* 10.0.0.0/30 [20/0] via 10.0.2.5, swp1, 00:02:22
* via 10.0.2.1, swp2, 00:02:22
B>* 10.0.1.0/30 [20/0] via 10.0.2.5, swp1, 00:02:22
* via 10.0.2.1, swp2, 00:02:22
B>* 10.0.1.4/30 [20/0] via 10.0.2.1, swp2, 00:02:22
* via 10.0.2.5, swp1, 00:02:22
B 10.0.2.0/30 [20/0] via 10.0.2.1 inactive, 00:02:22
via 10.0.2.5, swp1, 00:02:22
B 10.0.2.4/30 [20/0] via 10.0.2.5 inactive, 00:02:22
via 10.0.2.1, swp2, 00:02:22
B>* 10.100.0.0/24 [20/0] via 10.0.2.5, swp1, 00:02:22
* via 10.0.2.1, swp2, 00:02:22
root@leaf-2:/home/cumulus#
Have fun!
Read my new post about an Ansible Playbook for Cumulus Linux BGP IP-Fabric and Cumulus NetQ Validation.
