Cumulus Networks NetQ telemetry-based validation system

I had some time to play around with the new NetQ tool from Cumulus which checks your Cumulus Linux switch fabric.

I did some testing with my Cumulus Layer 2 Fabric example: Ansible Playbook for Cumulus Linux (Layer 2 Fabric)

You need to download the NetQ VM from Cumulus as VMware or VirtualBox template: here

It is a great tool to centrally check your Cumulus switches and keep history about changes in your environment. NetQ can send out notification about changes in your fabric which is nice because you are always up-to-date what is going on in your network.

Installing NetQ agent on a Cumulus Linux Switch:

[email protected]:~$ sudo apt-get update
[email protected]:~$ sudo apt-get install cumulus-netq -y

Configuring the NetQ Agent on a switch:

[email protected]:~$ sudo systemctl restart rsyslog
[email protected]:~$ netq add server 192.168.100.133
[email protected]:~$ netq agent restart

I will write a small Ansible script in the next days to automate the agent installation and configuration.

Connect to Cumulus NetQ VM and check agent connectivity

[email protected]:~$ netq-shell

Welcome to Cumulus (R) NetQ Command Line Interface
TIP: Type `netq help` to get started.

[email protected]:/$ netq show agents
Node     Status    Sys Uptime    Agent Uptime
-------  --------  ------------  --------------
leaf-1   Fresh     1h ago        1h ago
leaf-2   Fresh     1h ago        1h ago
spine-1  Fresh     1h ago        1h ago
spine-2  Fresh     1h ago        1h ago
[email protected]:/$

Basic Show Commands:

[email protected]:/$ netq show clag
Matching CLAG session records are:
Node             Peer             SysMac            State Backup #Links #Dual Last Changed
---------------- ---------------- ----------------- ----- ------ ------ ----- --------------
leaf-1           leaf-2(P)        44:38:39:ff:40:93 up    up     1      1     8m ago
leaf-2(P)        leaf-1           44:38:39:ff:40:93 up    up     1      1     8m ago
spine-1(P)       spine-2          44:38:39:ff:40:94 up    up     1      1     8m ago
spine-2          spine-1(P)       44:38:39:ff:40:94 up    up     1      1     9m ago
[email protected]:/$
 
[email protected]:/$ netq show lldp
LLDP peer info for *:*
Node     Interface    LLDP Peer    Peer Int    Last Changed
-------  -----------  -----------  ----------  --------------
leaf-1   eth0         cumulus      eth0        1h ago
leaf-1   eth0         leaf-2       eth0        1h ago
leaf-1   eth0         spine-1      eth0        1h ago
leaf-1   eth0         spine-2      eth0        1h ago
leaf-1   swp1         spine-1      swp1        1h ago
leaf-1   swp11        leaf-2       swp11       9m ago
leaf-1   swp2         spine-2      swp1        1h ago
leaf-2   eth0         cumulus      eth0        1h ago
leaf-2   eth0         leaf-1       eth0        1h ago
leaf-2   eth0         spine-1      eth0        1h ago
leaf-2   eth0         spine-2      eth0        1h ago
leaf-2   swp1         spine-2      swp2        1h ago
leaf-2   swp11        leaf-1       swp11       8m ago
leaf-2   swp2         spine-1      swp2        1h ago
spine-1  eth0         cumulus      eth0        1h ago
spine-1  eth0         leaf-1       eth0        1h ago
spine-1  eth0         leaf-2       eth0        1h ago
spine-1  eth0         spine-2      eth0        1h ago
spine-1  swp1         leaf-1       swp1        1h ago
spine-1  swp11        spine-2      swp11       1h ago
spine-1  swp2         leaf-2       swp2        8m ago
spine-2  eth0         cumulus      eth0        1h ago
spine-2  eth0         leaf-1       eth0        1h ago
spine-2  eth0         leaf-2       eth0        1h ago
spine-2  eth0         spine-1      eth0        1h ago
spine-2  swp1         leaf-1       swp2        1h ago
spine-2  swp11        spine-1      swp11       1h ago
spine-2  swp2         leaf-2       swp1        8m ago
[email protected]:/$
 
[email protected]:/$ netq show interfaces type bond
Matching interface records are:
Node             Interface        Type     State Details                     Last Changed
---------------- ---------------- -------- ----- --------------------------- --------------
leaf-1           bond1            bond     up    Slave: swp1(spine-1:swp1),  10m ago
                                                 Slave: swp2(spine-2:swp1),
                                                 VLANs:  100-199, PVID: 1,
                                                 Master: bridge, MTU: 1500
leaf-1           peerlink         bond     up    Slave: swp11(leaf-2:swp11), 10m ago
                                                 VLANs: , PVID: 0,
                                                 Master: peerlink, MTU: 1500
leaf-2           bond1            bond     up    Slave: swp1(spine-2:swp2),  10m ago
                                                 Slave: swp2(spine-1:swp2),
                                                 VLANs:  100-199, PVID: 1,
                                                 Master: bridge, MTU: 1500
leaf-2           peerlink         bond     up    Slave: swp11(leaf-1:swp11), 10m ago
                                                 VLANs: , PVID: 0,
                                                 Master: peerlink, MTU: 1500
spine-1          bond1            bond     up    Slave: swp1(leaf-1:swp1),   10m ago
                                                 Slave: swp2(leaf-2:swp2),
                                                 VLANs:  100-199, PVID: 1,
                                                 Master: bridge, MTU: 1500
spine-1          peerlink         bond     up    Slave: swp11(spine-2:swp11) 1h ago
                                                 , VLANs:  100-199,
                                                 PVID: 1, Master: bridge,
                                                 MTU: 1500
spine-2          bond1            bond     up    Slave: swp1(leaf-1:swp2),   10m ago
                                                 Slave: swp2(leaf-2:swp1),
                                                 VLANs:  100-199, PVID: 1,
                                                 Master: bridge, MTU: 1500
spine-2          peerlink         bond     up    Slave: swp11(spine-1:swp11) 1h ago
                                                 , VLANs:  100-199,
                                                 PVID: 1, Master: bridge,
                                                 MTU: 1500
[email protected]:/$
 
[email protected]:/$ netq show ip routes
Matching IP route records are:
Origin Table            IP               Node             Nexthops                   Last Changed
------ ---------------- ---------------- ---------------- -------------------------- ----------------
1      default          169.254.1.0/30   leaf-1           peerlink.4093              11m ago
1      default          169.254.1.0/30   leaf-2           peerlink.4093              11m ago
1      default          169.254.1.0/30   spine-1          peerlink.4094              1h ago
1      default          169.254.1.0/30   spine-2          peerlink.4094              1h ago
1      default          169.254.1.1/32   leaf-1           peerlink.4093              11m ago
1      default          169.254.1.1/32   spine-1          peerlink.4094              1h ago
1      default          169.254.1.2/32   leaf-2           peerlink.4093              11m ago
1      default          169.254.1.2/32   spine-2          peerlink.4094              1h ago
1      default          192.168.100.0/24 leaf-1           eth0                       1h ago
1      default          192.168.100.0/24 leaf-2           eth0                       1h ago
1      default          192.168.100.0/24 spine-1          eth0                       1h ago
1      default          192.168.100.0/24 spine-2          eth0                       1h ago
1      default          192.168.100.205/ spine-1          eth0                       1h ago
                        32
1      default          192.168.100.206/ spine-2          eth0                       1h ago
                        32
1      default          192.168.100.207/ leaf-1           eth0                       1h ago
                        32
1      default          192.168.100.208/ leaf-2           eth0                       1h ago
                        32
0      vrf-prod         0.0.0.0/0        spine-1          Blackhole                  1h ago
0      vrf-prod         0.0.0.0/0        spine-2          Blackhole                  1h ago
1      vrf-prod         10.1.0.0/24      spine-1          bridge.100                 1h ago
1      vrf-prod         10.1.0.0/24      spine-2          bridge.100                 1h ago
1      vrf-prod         10.1.0.252/32    spine-1          bridge.100                 1h ago
1      vrf-prod         10.1.0.253/32    spine-2          bridge.100                 1h ago
1      vrf-prod         10.1.0.254/32    spine-1          bridge.100                 1h ago
1      vrf-prod         10.1.0.254/32    spine-2          bridge.100                 1h ago
1      vrf-prod         10.1.1.0/24      spine-1          bridge.101                 1h ago
1      vrf-prod         10.1.1.0/24      spine-2          bridge.101                 1h ago
1      vrf-prod         10.1.1.252/32    spine-1          bridge.101                 1h ago
1      vrf-prod         10.1.1.253/32    spine-2          bridge.101                 1h ago
1      vrf-prod         10.1.1.254/32    spine-1          bridge.101                 1h ago
1      vrf-prod         10.1.1.254/32    spine-2          bridge.101                 1h ago
1      vrf-prod         10.1.2.0/24      spine-1          bridge.102                 1h ago
1      vrf-prod         10.1.2.0/24      spine-2          bridge.102                 1h ago
1      vrf-prod         10.1.2.252/32    spine-1          bridge.102                 1h ago
1      vrf-prod         10.1.2.253/32    spine-2          bridge.102                 1h ago
1      vrf-prod         10.1.2.254/32    spine-1          bridge.102                 1h ago
1      vrf-prod         10.1.2.254/32    spine-2          bridge.102                 1h ago
[email protected]:/$

See Changes in Switch Fabric:

[email protected]:/$ netq leaf-1 show interfaces type bond
Matching interface records are:
Node             Interface        Type     State Details                     Last Changed
---------------- ---------------- -------- ----- --------------------------- --------------
leaf-1           bond1            bond     up    Slave: swp1(spine-1:swp1),  2s ago
                                                 Slave: swp2(spine-2:swp1),
                                                 VLANs:  100-199, PVID: 1,
                                                 Master: bridge, MTU: 1500
leaf-1           peerlink         bond     up    Slave: swp11(leaf-2:swp11), 21m ago
                                                 VLANs: , PVID: 0,
                                                 Master: peerlink, MTU: 1500
[email protected]:/$
 
[email protected]:~$ sudo ifdown bond1
[email protected]:~$
 
[email protected]:/$ netq leaf-1 show interfaces type bond
Matching interface records are:
Node             Interface        Type     State Details                     Last Changed
---------------- ---------------- -------- ----- --------------------------- --------------
leaf-1           peerlink         bond     up    Slave: swp11(leaf-2:swp11), 22m ago
                                                 VLANs: , PVID: 0,
                                                 Master: peerlink, MTU: 1500
[email protected]:/$
 
[email protected]:/$ netq leaf-1 show interfaces type bond changes
Matching interface records are:
Node             Interface        Type     State Details                     DbState Last Changed
---------------- ---------------- -------- ----- --------------------------- ------- --------------
leaf-1           bond1            bond     down  VLANs: , PVID: 0,           Del     21s ago
                                                 Master: bridge, MTU: 1500
leaf-1           bond1            bond     down  Slave: swp1(),              Add     21s ago
                                                 Slave: swp2(),
                                                 VLANs:  100-199, PVID: 1,
                                                 Master: bridge, MTU: 1500
leaf-1           bond1            bond     up    Slave: swp1(spine-1:swp1),  Add     1m ago
                                                 Slave: swp2(spine-2:swp1),
                                                 VLANs:  100-199, PVID: 1,
                                                 Master: bridge, MTU: 1500 

More information you can find in the Cumulus NetQ documentation: https://docs.cumulusnetworks.com/display/NETQ/NetQ

Ansible Semaphore

I spend lot of time working with Ansible in the last weeks to automate the deployment of Cisco router or Cumulus switches. (Waiting for Ansible 2.2 to support Cisco ASA devices..)
Ansible is a great tool but if you have multiple YAML files and various roles it can get pretty messy and would be nice to have central tool to trigger your tasks and structure your environment variables or inventories.

I exactly found this tool with Ansible Semaphore: https://github.com/ansible-semaphore/semaphore

The install is pretty easy and provides an API to trigger your tasks remotely.

You can create different projects and include your Ansible YAML files.

screen-shot-2016-10-15-at-21-59-57

The source is a Git repository where your files are stored:

screen-shot-2016-10-15-at-21-58-18

Here your environment variables:

screen-shot-2016-10-15-at-21-58-43

Inventory definition:

screen-shot-2016-10-15-at-21-59-02

Finally the you can execute your Ansible YAML files via the Web UI or API:

screen-shot-2016-10-15-at-22-00-26

screen-shot-2016-10-15-at-22-00-53

Have fun playing around with Semaphore 🙂

Cisco IOS automation with Ansible

Bin a long time since I wrote my last post, I am pretty busy with work redesigning the data centres for my employer. Implementing as well an SDN Software-defined Network from VMware NSX but more about this later.

Ansible released some weeks ago new core modules which allows you to push directly configuration to Cisco IOS devices. More information you find here: https://docs.ansible.com/ansible/list_of_network_modules.html

I created a small automation lab in GNS3 to test the deployment of configs via Ansible to the two Cisco routers you see below. I am running VMware Fusion and used the vmnet2 (192.168.100.0/24) network for management because I run there my CentOS VM from where I deploy the configuration.

Don’t forget you need to pre-configure your Cisco router that you can connect via SSH to deploy the configuration.

Here the folder and file script structure of my Ansible example, under roles you have the different tasks I would like to execute common and logging but as well dependencies writecfg which saves the running-config to startup-config:

site.yml
hosts
group_vars/all.yml
roles/common/meta/main.yml
roles/common/task/main.yml
roles/common/templates/common.j2
roles/logging/meta/main.yml
roles/logging/tasks/main.yml
roles/logging/templates/common.j2
roles/writecfg/handlers/main.yml

The site.yml is the main script which I execute with Ansible which includes different roles for common and logging configuration:

- name: Cisco baseline configuration
  connection: local
  hosts: ios 
  gather_facts: false

  roles:
    - role: common
      tags: common
    - role: logging
      tags: logging

In the hosts file, I define the hostname and IP addresses of my IOS devices

[ios]
rtr01 device_ip=192.168.100.130
rtr02 device_ip=192.168.100.132

The file group_vars/all.yml defines variables which I used when the script is executed:

---
username: "ansible"
password: "cisco"
secret: "cisco"
logserver: 192.168.100.131

Under the roles/../meta/main.yml I set a dependency on the writecfg handler to save the configuration later when I change anything on the device.

Under the roles/../tasks/main.yml I define the module which I want to execute and the template I would like to deploy

Under the roles/../templates/.. you find the Jinja2 template files which include the commands.

Under roles/writecfg/handler/main.yml is the dependencies I have with the two roles common and logging to save the configuration if something is changed on the router.

To execute the cisco-baseline Ansible script just execute the following command and see the result:

[[email protected] cisco-baseline]$ ansible-playbook site.yml -i hosts

PLAY [Ensure basic configuration of switches] **********************************

TASK [common : ensure common configuration exists] *****************************
ok: [rtr02]
ok: [rtr01]

TASK [logging : ensure logging configuration exists] ***************************
changed: [rtr02]
changed: [rtr01]

RUNNING HANDLER [writecfg : write config] **************************************
ok: [rtr01]
ok: [rtr02]

PLAY RECAP *********************************************************************
rtr01                      : ok=3    changed=1    unreachable=0    failed=0
rtr02                      : ok=3    changed=1    unreachable=0    failed=0

[[email protected] cisco-baseline]$

Read my new posts about Ansible Playbook for Cisco ASAv Firewall Topology or Ansible Playbook for Cisco BGP Routing Topology.

Uptime – simple http monitoring utility

I found an very interesting http monitoring tool called Uptime using node.js and mongoDB. I directly installed Uptime on one of my Linux servers and from the first look I find it really cool 🙂 before you start you need to get node.js and mongoDB installed on your server and the rest is then very easy.

Ones the Uptime is running you can access the web interface and create the first checks, here some screenshots:

Here you create your http checks and define some settings:

Detailed check overview with graphs:

If you are interested then have a look here: http://fzaninotto.github.com/uptime/

GNS3 Network Simulator

Found something really cool today 🙂

GNS3 is a graphical network simulator where you can set-up complex virtual networks and run Cisco and Juniper routers or switches. The best is that you can also integrate Qemu and Virtualbox into your virtual lab environment what I really love. You can easily test new configurations on devices without having to set-up all these in hardware.

The only little problem is that you need a quite power system to do all of that. Otherwise I tested GNS3 on an 3 year old laptop with Intel Core2Duo and 4 GB RAM and run up to 6 Cisco routers without any big problems what’s enough for me at the moment.

Ah I forgot, you can of course also use Wireshark to capture packets on an link between two devices.

Here the link to the website: www.gns3.net