Strange ARP issue between ASA and Cisco router

Recently I had a strange ARP problem between an Cisco ASA firewall and an Cisco router (provider router) on an internet line in one of our remote offices. Periodically the office lost the network connectivity.

From the first look the ARP table seemed fine:

# sh arp | i OUTSIDE
OUTSIDE 212.0.107.169 000f.e28a.1f7a 348

The ARP resolution was not working properly, the firewall was waiting for responses or even lost the ARP entry from the router. From the debugging output you can see that the firewall was in pending state and waiting for the router to respond:

# clear arp OUTSIDE 212.0.107.169
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-send: arp request built from 212.0.107.170 0a00.0a00.0010 for 212.0.107.169 at 3637391690
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-in: response at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.170 0a00.0a00.0010
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3637391710
arp-in: resp from 212.0.107.169 for 212.0.107.170 on OUTSIDE at 3637391710
arp-send: sending all saved block to OUTSIDE 212.0.107.169 at 3637391710

The same happen to normal ARP updates and the reason why we lost periodically the connectivity because the router didnt respond at all.

Our provider quickly figured out that there was a problem with the device and replaced the router.

ARP table output:

# sh arp | i OUTSIDE
OUTSIDE 212.0.107.169 000f.e28a.1f7a 303

Here the normal ARP behaviour ones the router was replaced,  the router responded directly to ARP requests:

# clear arp OUTSIDE 212.0.107.169
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-send: arp request built from 212.0.107.170 0a00.0a00.0010 for 212.0.107.169 at 3717553710
arp-in: response at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.170 0a00.0a00.0010
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3717553710
arp-in: resp from 212.0.107.169 for 212.0.107.170 on OUTSIDE at 3717553710

Normal ARP updates:

arp-in: request at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.171 0000.0000.0000
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3717983740

 

Ansible Playbook for Cisco Lab

From my recent posts, you can see that I use Ansible a lot for automating the device configuration deployment. Here my firewall lab (Cisco routers and Cisco ASA firewall) which I use to test different things in GNS3:

Before you can start deploying configs via Ansible you need to manually configure your management interfaces and device remote access. I run VMware Fusion Pro and use my VMNET2 network as management network because I have additional VMs for Ansible and Monitoring.

Here the config to prep your Cisco routers that you can afterwards deploy the rest of the config via Ansible:

conf t
ip vrf vrf-mgmt
	rd 1:1
	exit

interface Ethernet1/0
 description management
 ip vrf forwarding vrf-mgmt
 ip address 192.168.100.201 255.255.255.0
 no shutdown
 exit

ip domain-name localdomain

aaa new-model
aaa authentication login default local
aaa authorization exec default local 

username ansible privilege 15 secret 5 $1$xAJX$D99QcH02Splr1L3ktrvh41

crypto key generate rsa general-keys modulus 2048 

ip ssh version 2
ip ssh authentication-retries 5

line vty 0 4
 transport input ssh
 exit

exit
write mem

The same you need to do for your Cisco ASA firewall:

conf t
enable password 2KFQnbNIdI.2KYOU encrypted

interface Management0/0
 nameif management
 security-level 0
 ip address 192.168.100.204 255.255.255.0
 
aaa authentication ssh console LOCAL

ssh 0.0.0.0 0.0.0.0 management

username ansible password xsxRJKdxDzf9Ctr8 encrypted privilege 15
exit
write mem

Now you are ready to deploy the basic lab configuration to all the devices but before we start we need hosts and vars files and the main Ansible Playbook (yaml) file.

In the host’s file I define all the interface variables, there are different ways of doing it but this one is the easiest.

./hosts

[router]
inside
dmz
outside
[firewall]
firewall

In the group_vars file is the global variables.

./group_vars/all.yml

---
username: "ansible"
password: "cisco"
secret: "cisco"
default_gw_inside: "10.1.255.1"
default_gw_dmz: "10.1.255.33"
default_gw_firewall: "217.110.110.254"

Here the Ansible Playbook with the basic device configuration:

./interfaces.yml

- name: Deploy Cisco lab configuration part 1
  connection: local
  hosts: router
  gather_facts: false
  vars:
    cli:
      username: "{{ username }}"
      password: "{{ password }}"
      host: "{{ device_ip }}"
  tasks:
    - name: deploy inside router configuration
      when: ansible_host not in "outside"
      ios_config:
        provider: "{{ cli }}"
        before:
          - "default interface {{ item.interface }}"
        lines:
          - "ip address {{ item.address }}"
        after:
          - no shutdown
        parents: "interface {{ item.interface }}"
        match: strict
      with_items:
        - { interface : Ethernet0/0, address : "{{ eth_00_ip }} {{ eth_00_mask }}" }
        - { interface : Ethernet0/1, address : "{{ eth_01_ip }} {{ eth_01_mask }}" }
    - name: deploy outside router configuration
      when: ansible_host not in "inside,dmz"
      ios_config:
        provider: "{{ cli }}"
        before:
          - "default interface {{ item.interface }}"
        lines:
          - "ip address {{ item.address }}"
        after:
          - no shutdown
        parents: "interface {{ item.interface }}"
        match: strict
      with_items:
        - { interface : Ethernet0/0, address : "{{ eth_00_ip }} {{ eth_00_mask }}" }
        - { interface : Ethernet0/1, address : "{{ eth_01_ip }}" }

- name: Deploy Cisco lab configuration part 2
  connection: local
  hosts: firewall
  gather_facts: false
  vars:
      cli:
       username: "{{ username }}"
       password: "{{ password }}"
       auth_pass: "{{ secret }}"
       authorize: yes
       host: "{{ device_ip }}"
  tasks:
    - name: deploy firewall configuration
      when: ansible_host not in "inside,dmz,outside"
      asa_config:
        provider: "{{ cli }}"
        lines:
          - "nameif {{ item.nameif }}"
          - "ip address {{ item.address }}"
        after:
          - no shutdown
        parents: "interface {{ item.interface }}"
        match: line
      with_items:
        - { interface : GigabitEthernet0/0, nameif : "{{ eth_00_nameif }}", address : "{{ eth_00_ip }} {{ eth_00_mask }}" }
        - { interface : GigabitEthernet0/1, nameif : "{{ eth_01_nameif }}", address : "{{ eth_01_ip }} {{ eth_01_mask }}" }
        - { interface : GigabitEthernet0/2, nameif : "{{ eth_02_nameif }}", address : "{{ eth_02_ip }} {{ eth_02_mask }}" }

In the playbook, I needed to separate the outside router because one interface is configured to dhcp otherwise I could have used only one task for all three routers.

The 2nd part is for the Cisco ASA firewall configuration because it uses a different Ansible module and variables.

Now let us deploy the config and see the output from Ansible:

[berndonline@ansible firewall]$ ansible-playbook interfaces.yml -i hosts

PLAY [Deploy firewall lab configuration part 1] ********************************

TASK [deploy inside router configuration] **************************************
skipping: [outside] => (item={u'interface': u'Ethernet0/1', u'address': u'dhcp '})
skipping: [outside] => (item={u'interface': u'Ethernet0/0', u'address': u'217.110.110.254 255.255.255.0'})
changed: [dmz] => (item={u'interface': u'Ethernet0/0', u'address': u'10.1.255.34 255.255.255.240'})
changed: [inside] => (item={u'interface': u'Ethernet0/0', u'address': u'10.1.255.2 255.255.255.240'})
changed: [dmz] => (item={u'interface': u'Ethernet0/1', u'address': u'10.1.1.254 255.255.255.0'})
changed: [inside] => (item={u'interface': u'Ethernet0/1', u'address': u'10.1.0.254 255.255.255.0'})

TASK [deploy outside router configuration] *************************************
skipping: [inside] => (item={u'interface': u'Ethernet0/1', u'address': u'10.1.0.254'})
skipping: [inside] => (item={u'interface': u'Ethernet0/0', u'address': u'10.1.255.2 255.255.255.240'})
skipping: [dmz] => (item={u'interface': u'Ethernet0/1', u'address': u'10.1.1.254'})
skipping: [dmz] => (item={u'interface': u'Ethernet0/0', u'address': u'10.1.255.34 255.255.255.240'})
changed: [outside] => (item={u'interface': u'Ethernet0/0', u'address': u'217.110.110.254 255.255.255.0'})
changed: [outside] => (item={u'interface': u'Ethernet0/1', u'address': u'dhcp'})

PLAY [Deploy firewall lab configuration part 2] ********************************

TASK [deploy firewall configuration] *******************************************
changed: [firewall] => (item={u'interface': u'GigabitEthernet0/0', u'nameif': u'inside', u'address': u'10.1.255.1 255.255.255.240'})
changed: [firewall] => (item={u'interface': u'GigabitEthernet0/1', u'nameif': u'dmz', u'address': u'10.1.255.33 255.255.255.240'})
changed: [firewall] => (item={u'interface': u'GigabitEthernet0/2', u'nameif': u'outside', u'address': u'217.110.110.1 255.255.255.0'})

PLAY RECAP *********************************************************************
dmz                        : ok=1    changed=1    unreachable=0    failed=0
firewall                   : ok=1    changed=1    unreachable=0    failed=0
inside                     : ok=1    changed=1    unreachable=0    failed=0
outside                    : ok=1    changed=1    unreachable=0    failed=0

[berndonline@ansible firewall]$

Quick check if Ansible deployed the interface configuration:

inside#sh ip int brief
Interface                  IP-Address      OK? Method Status                Protocol
Ethernet0/0                10.1.255.2      YES manual up                    up
Ethernet0/1                10.1.0.254      YES manual up                    up
Ethernet1/0                192.168.100.201 YES NVRAM  up                    up
inside#

dmz#sh ip int brief
Interface                  IP-Address      OK? Method Status                Protocol
Ethernet0/0                10.1.255.34     YES manual up                    up
Ethernet0/1                10.1.1.254      YES manual up                    up
Ethernet1/0                192.168.100.202 YES NVRAM  up                    up
dmz#

outside#sh ip int brief
Interface                  IP-Address      OK? Method Status                Protocol
Ethernet0/0                217.110.110.254 YES manual up                    up
Ethernet0/1                172.16.191.23   YES DHCP   up                    up
Ethernet1/0                192.168.100.203 YES NVRAM  up                    up
outside#

firewall# sho ip address
Current IP Addresses:
Interface                Name                   IP address      Subnet mask     Method
GigabitEthernet0/0       inside                 10.1.255.1      255.255.255.240 manual
GigabitEthernet0/1       dmz                    10.1.255.33     255.255.255.240 manual
GigabitEthernet0/2       outside                217.110.110.1   255.255.255.0   manual
Management0/0            management             192.168.100.204 255.255.255.0   CONFIG
firewall#

As you can see Ansible deployed the interface configuration correctly. If I run Ansible again nothing will be deployed because the configuration is already present:

[berndonline@ansible firewall]$ ansible-playbook interfaces.yml -i hosts

PLAY [Deploy firewall lab configuration part 1] ********************************

TASK [deploy inside router configuration] **************************************
skipping: [outside] => (item={u'interface': u'Ethernet0/1', u'address': u'dhcp '})
skipping: [outside] => (item={u'interface': u'Ethernet0/0', u'address': u'217.110.110.254 255.255.255.0'})
ok: [dmz] => (item={u'interface': u'Ethernet0/0', u'address': u'10.1.255.34 255.255.255.240'})
ok: [dmz] => (item={u'interface': u'Ethernet0/1', u'address': u'10.1.1.254 255.255.255.0'})
ok: [inside] => (item={u'interface': u'Ethernet0/0', u'address': u'10.1.255.2 255.255.255.240'})
ok: [inside] => (item={u'interface': u'Ethernet0/1', u'address': u'10.1.0.254 255.255.255.0'})

TASK [deploy outside router configuration] *************************************
skipping: [inside] => (item={u'interface': u'Ethernet0/1', u'address': u'10.1.0.254'})
skipping: [inside] => (item={u'interface': u'Ethernet0/0', u'address': u'10.1.255.2 255.255.255.240'})
skipping: [dmz] => (item={u'interface': u'Ethernet0/1', u'address': u'10.1.1.254'})
skipping: [dmz] => (item={u'interface': u'Ethernet0/0', u'address': u'10.1.255.34 255.255.255.240'})
ok: [outside] => (item={u'interface': u'Ethernet0/0', u'address': u'217.110.110.254 255.255.255.0'})
ok: [outside] => (item={u'interface': u'Ethernet0/1', u'address': u'dhcp'})

PLAY [Deploy firewall lab configuration part 2] ********************************

TASK [deploy firewall configuration] *******************************************
ok: [firewall] => (item={u'interface': u'GigabitEthernet0/0', u'nameif': u'inside', u'address': u'10.1.255.1 255.255.255.240'})
ok: [firewall] => (item={u'interface': u'GigabitEthernet0/1', u'nameif': u'dmz', u'address': u'10.1.255.33 255.255.255.240'})
ok: [firewall] => (item={u'interface': u'GigabitEthernet0/2', u'nameif': u'outside', u'address': u'217.110.110.1 255.255.255.0'})

PLAY RECAP *********************************************************************
dmz                        : ok=1    changed=0    unreachable=0    failed=0
firewall                   : ok=1    changed=0    unreachable=0    failed=0
inside                     : ok=1    changed=0    unreachable=0    failed=0
outside                    : ok=1    changed=0    unreachable=0    failed=0

[berndonline@ansible firewall]$

In my GNS3 labs, I normally not save the device configuration except the management IPs because with Ansible I can deploy everything again within seconds and use different Playbooks depending what I want to test. It gets even cooler if you use Semaphore (see my blog post: Ansible Semaphore) because you just click ones on the Playbook you want to deploy.

Comment below if you have questions or problems.

Read my new posts about Ansible Playbook for Cisco ASAv Firewall Topology or Ansible Playbook for Cisco BGP Routing Topology.

Cisco ASA and IOS-XE embedded packet capturing

This is a short post about a step-by-step procedure to configure packet capturing on Cisco ASA or IOS XE using the CLI.

Cisco ASA embedded packet capturing:

access-list acl_capin extended permit ip host 217.100.100.254 host 10.0.255.254
access-list acl_capin extended permit ip host 10.0.255.254 host 217.100.100.254
capture capin interface inside access-list acl_capin

or

capture capin interface inside match ip host 10.0.255.254 host 217.100.100.254
[possible in asa 8.x and later]

Show captured packets:

asa-1(config)#  show capture capin

10 packets captured

   1: 15:11:12.760092       10.0.255.254 > 217.100.100.254: icmp: echo request
   2: 15:11:12.761755       217.100.100.254 > 10.0.255.254: icmp: echo reply
   3: 15:11:12.764196       10.0.255.254 > 217.100.100.254: icmp: echo request
   4: 15:11:12.765615       217.100.100.254 > 10.0.255.254: icmp: echo reply
   5: 15:11:12.768072       10.0.255.254 > 217.100.100.254: icmp: echo request
   6: 15:11:12.769354       217.100.100.254 > 10.0.255.254: icmp: echo reply
   7: 15:11:12.771612       10.0.255.254 > 217.100.100.254: icmp: echo request
   8: 15:11:12.773077       217.100.100.254 > 10.0.255.254: icmp: echo reply
   9: 15:11:12.775548       10.0.255.254 > 217.100.100.254: icmp: echo request
  10: 15:11:12.777150       217.100.100.254 > 10.0.255.254: icmp: echo reply
10 packets shown

asa-1(config)#

asa-1(config)#  show capture capinside detail

20 packets captured

   1: 15:11:12.760092 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 5)
   2: 15:11:12.761755 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 5)
   3: 15:11:12.764196 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 6)
   4: 15:11:12.765615 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 6)
   5: 15:11:12.768072 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 7)
   6: 15:11:12.769354 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 7)
   7: 15:11:12.771612 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 8)
   8: 15:11:12.773077 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 8)
   9: 15:11:12.775548 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 9)
  10: 15:11:12.777150 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 9)
20 packets shown

asa-1(config)#
Browser capture:
https://10.255.1.203/admin/capture/capin

Download pcap:
https://10.255.1.203/capture/capin/pcap

Disable capture and remove access-list:

no capture capin
no capture capout
clear configure access-list acl_capin
clear configure access-list acl_capout

Cisco ASR embedded packet capturing:

ip access-list extended acl_cap
 permit ip any any
 permit icmp any any
 exit
 
monitor capture mycap access-list acl_cap 
monitor capture mycap limit duration 1000
monitor capture mycap interface GigabitEthernet3 both
monitor capture mycap buffer circular size 10
monitor capture mycap start
monitor capture mycap export tftp://10.255.1.87/mycap.pcap

Show captured packets:

rtr-2#show monitor capture mycap buffer dump
0
  0000:  A0000000 0004A000 00000001 08004500   ..............E.
  0010:  00640041 0000FF01 A9530A00 FF010A00   .d.A.....S......
  0020:  FF020800 0B62000D 00000000 0000001E   .....b..........
  0030:  72BDABCD ABCDABCD ABCDABCD ABCDABCD   r...............
  0040:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0050:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0060:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0070:  ABCD                                  ..

1
  0000:  A0000000 0001A000 00000004 08004500   ..............E.
  0010:  00640041 0000FF01 A9530A00 FF020A00   .d.A.....S......
  0020:  FF010000 1362000D 00000000 0000001E   .....b..........
  0030:  72BDABCD ABCDABCD ABCDABCD ABCDABCD   r...............
  0040:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0050:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0060:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0070:  ABCD                                  ..
...

rtr-2#show monitor capture mycap buffer
 buffer size (KB) : 10240
 buffer used (KB) : 128
 packets in buf   : 14
 packets dropped  : 0
 packets per sec  : 1
...

Disable capture and remove access-list:

monitor capture mycap stop
no monitor capture mycap
no ip access-list extended acl_cap

Ansible ASA Playbook (asa_config and asa_acl): Cisco ASA access-list

Like in my previous post in the new development version 2.2. from Ansible are new IOS and ASA core modules.

Here an example of the asa_config and asa_acl module to create and object-group in the first step and create the inside create access-list:

- name: Cisco ASA access-list config
  connection: local
  hosts: firewall
  gather_facts: false
  vars:
    cli:
      username: "{{ username }}"
      password: "{{ password }}"
      host: "{{ device_ip }}"
      authorize: yes
      auth_pass: cisco
  tasks:
    - name: create object group
      asa_config:
        lines:
          - network-object host 10.1.0.1
          - network-object host 10.1.0.2
          - network-object host 10.1.0.3
        parents: ['object-group network dummy-group']
        provider: "{{ cli }}"
#      register: result

    - name: configure access-list
      asa_acl:
        lines:
          - access-list acl_inside extended permit tcp object-group dummy-group any eq www
          - access-list acl_inside extended permit udp object-group dummy-group any eq domain
          - access-list acl_inside extended deny ip any any
        before: clear configure access-list acl_inside
        match: strict
        replace: block
        provider: "{{ cli }}" 
#      register: result

    - debug: var=result

Here output when you run the playbook the first time:

ansible-playbook cisco/asa_access-list_config.yml -i cisco/hosts

PLAY [Cisco ASA access-list config] ********************************************

TASK [create object group] *****************************************************
changed: [fw1]

TASK [configure access-list] ***************************************************
changed: [fw1]

TASK [debug] *******************************************************************
ok: [fw1] => {
    "result": "VARIABLE IS NOT DEFINED!"
}

PLAY RECAP *********************************************************************
fw1                        : ok=3    changed=2    unreachable=0    failed=0

Here the output then you run the playbook a second time, you see nothing is changed:

ansible-playbook cisco/asa_access-list_config.yml -i cisco/hosts

PLAY [Cisco ASA access-list config] ********************************************

TASK [create object group] *****************************************************
ok: [fw1]

TASK [configure access-list] ***************************************************
ok: [fw1]

TASK [debug] *******************************************************************
ok: [fw1] => {
    "result": "VARIABLE IS NOT DEFINED!"
}

PLAY RECAP *********************************************************************
fw1                        : ok=3    changed=0    unreachable=0    failed=0

Read my new post about an Ansible Playbook for Cisco ASAv Firewall Topology

Ansible Interface Playbook (ios_config): Cisco interface config

Here an Ansible Playbook with three different examples how to configure Cisco router interfaces:

1. Static IP address configuration in playbook
2. Configuration comes out of Jinja2 template, can be run dynamic with variables
3. Loop in playbook configures multiple interfaces

There are some disadvantages to work with templates, if you use commands like “no shutdown” to enable the interface. They are not shown in the running-configuration which means Ansible will assume that the configuration is not matching and execute the template again.
Another disadvantage with templates is that you cannot run “before” or “after” commands to remove existing configuration all this needs to be implemented in your Jinja2 template.

Here the Ansible Playbook:

- name: Cisco interface config
  connection: local
  hosts: all
  gather_facts: false
  vars:
    cli:
      username: "{{ username }}"
      password: "{{ password }}"
      host: "{{ device_ip }}"
  tasks:
    - name: configure IP address
      ios_config:
        before: 
          - default interface FastEthernet1/0
        lines: 
          - ip address 10.1.1.1 255.255.255.0
        after: 
          - no shutdown
        match: strict
        parents: interface FastEthernet1/0
        provider: "{{ cli }}"

    - name: configure IP out of template
      ios_config:
        src: "interfaces.j2"
        provider: "{{ cli }}"

    - name: configure IP with loop
      ios_config:
        provider: "{{ cli }}"
        before:
          - "default interface {{ item.interface }}"
        lines:
          - "ip address {{ item.address }} 255.255.255.0"
        after:
          - no shutdown
        parents: "interface {{ item.interface }}"
      with_items:
        - { interface : FastEthernet2/0, address : 10.3.3.3 }
        - { interface : FastEthernet2/1, address : 10.4.4.4 }

Read my new posts about Ansible Playbook for Cisco ASAv Firewall Topology or Ansible Playbook for Cisco BGP Routing Topology.