Strange ARP issue between ASA and Cisco router

Recently I had a strange ARP problem between an Cisco ASA firewall and an Cisco router (provider router) on an internet line in one of our remote offices. Periodically the office lost the network connectivity.

From the first look the ARP table seemed fine:

# sh arp | i OUTSIDE
OUTSIDE 212.0.107.169 000f.e28a.1f7a 348

The ARP resolution was not working properly, the firewall was waiting for responses or even lost the ARP entry from the router. From the debugging output you can see that the firewall was in pending state and waiting for the router to respond:

# clear arp OUTSIDE 212.0.107.169
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-send: arp request built from 212.0.107.170 0a00.0a00.0010 for 212.0.107.169 at 3637391690
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-in: response at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.170 0a00.0a00.0010
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3637391710
arp-in: resp from 212.0.107.169 for 212.0.107.170 on OUTSIDE at 3637391710
arp-send: sending all saved block to OUTSIDE 212.0.107.169 at 3637391710

The same happen to normal ARP updates and the reason why we lost periodically the connectivity because the router didnt respond at all.

Our provider quickly figured out that there was a problem with the device and replaced the router.

ARP table output:

# sh arp | i OUTSIDE
OUTSIDE 212.0.107.169 000f.e28a.1f7a 303

Here the normal ARP behaviour ones the router was replaced,  the router responded directly to ARP requests:

# clear arp OUTSIDE 212.0.107.169
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-send: arp request built from 212.0.107.170 0a00.0a00.0010 for 212.0.107.169 at 3717553710
arp-in: response at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.170 0a00.0a00.0010
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3717553710
arp-in: resp from 212.0.107.169 for 212.0.107.170 on OUTSIDE at 3717553710

Normal ARP updates:

arp-in: request at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.171 0000.0000.0000
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3717983740

 

Cisco ASA and IOS-XE embedded packet capturing

This is a short post about a step-by-step procedure to configure packet capturing on Cisco ASA or IOS XE using the CLI.

Cisco ASA embedded packet capturing:

access-list acl_capin extended permit ip host 217.100.100.254 host 10.0.255.254
access-list acl_capin extended permit ip host 10.0.255.254 host 217.100.100.254
capture capin interface inside access-list acl_capin

or

capture capin interface inside match ip host 10.0.255.254 host 217.100.100.254
[possible in asa 8.x and later]

Show captured packets:

asa-1(config)#  show capture capin

10 packets captured

   1: 15:11:12.760092       10.0.255.254 > 217.100.100.254: icmp: echo request
   2: 15:11:12.761755       217.100.100.254 > 10.0.255.254: icmp: echo reply
   3: 15:11:12.764196       10.0.255.254 > 217.100.100.254: icmp: echo request
   4: 15:11:12.765615       217.100.100.254 > 10.0.255.254: icmp: echo reply
   5: 15:11:12.768072       10.0.255.254 > 217.100.100.254: icmp: echo request
   6: 15:11:12.769354       217.100.100.254 > 10.0.255.254: icmp: echo reply
   7: 15:11:12.771612       10.0.255.254 > 217.100.100.254: icmp: echo request
   8: 15:11:12.773077       217.100.100.254 > 10.0.255.254: icmp: echo reply
   9: 15:11:12.775548       10.0.255.254 > 217.100.100.254: icmp: echo request
  10: 15:11:12.777150       217.100.100.254 > 10.0.255.254: icmp: echo reply
10 packets shown

asa-1(config)#

asa-1(config)#  show capture capinside detail

20 packets captured

   1: 15:11:12.760092 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 5)
   2: 15:11:12.761755 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 5)
   3: 15:11:12.764196 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 6)
   4: 15:11:12.765615 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 6)
   5: 15:11:12.768072 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 7)
   6: 15:11:12.769354 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 7)
   7: 15:11:12.771612 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 8)
   8: 15:11:12.773077 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 8)
   9: 15:11:12.775548 a000.0000.0001 a000.0000.0021 0x0800 Length: 114
      10.0.255.254 > 217.100.100.254: icmp: echo request (ttl 255, id 9)
  10: 15:11:12.777150 a000.0000.0021 a000.0000.0001 0x0800 Length: 114
      217.100.100.254 > 10.0.255.254: icmp: echo reply (ttl 255, id 9)
20 packets shown

asa-1(config)#
Browser capture:
https://10.255.1.203/admin/capture/capin

Download pcap:
https://10.255.1.203/capture/capin/pcap

Disable capture and remove access-list:

no capture capin
no capture capout
clear configure access-list acl_capin
clear configure access-list acl_capout

Cisco ASR embedded packet capturing:

ip access-list extended acl_cap
 permit ip any any
 permit icmp any any
 exit
 
monitor capture mycap access-list acl_cap 
monitor capture mycap limit duration 1000
monitor capture mycap interface GigabitEthernet3 both
monitor capture mycap buffer circular size 10
monitor capture mycap start
monitor capture mycap export tftp://10.255.1.87/mycap.pcap

Show captured packets:

rtr-2#show monitor capture mycap buffer dump
0
  0000:  A0000000 0004A000 00000001 08004500   ..............E.
  0010:  00640041 0000FF01 A9530A00 FF010A00   .d.A.....S......
  0020:  FF020800 0B62000D 00000000 0000001E   .....b..........
  0030:  72BDABCD ABCDABCD ABCDABCD ABCDABCD   r...............
  0040:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0050:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0060:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0070:  ABCD                                  ..

1
  0000:  A0000000 0001A000 00000004 08004500   ..............E.
  0010:  00640041 0000FF01 A9530A00 FF020A00   .d.A.....S......
  0020:  FF010000 1362000D 00000000 0000001E   .....b..........
  0030:  72BDABCD ABCDABCD ABCDABCD ABCDABCD   r...............
  0040:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0050:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0060:  ABCDABCD ABCDABCD ABCDABCD ABCDABCD   ................
  0070:  ABCD                                  ..
...

rtr-2#show monitor capture mycap buffer
 buffer size (KB) : 10240
 buffer used (KB) : 128
 packets in buf   : 14
 packets dropped  : 0
 packets per sec  : 1
...

Disable capture and remove access-list:

monitor capture mycap stop
no monitor capture mycap
no ip access-list extended acl_cap

Cisco ASA TCP Connection Flags

I got asked to look into a problem where two servers where not able to communicate with each other, ping didnt work and the application could not connect to the server. Firewall rules and routing was fine and my colleague spend already over an hour but couldnt find something. The first thing I asked, do you see a TCP connection? He told me yes over the ASDM logging I see something…. I double check and connect to the console and run:

show conn address 10.20.100.21

Show conn output:

TCP DMZ 10.10.127.29:2222 TRANSFER 10.20.100.21:42799, idle 0:00:00, bytes 0, flags saA
TCP DMZ 10.10.127.29:2223 TRANSFER 10.20.100.21:63554, idle 0:00:00, bytes 0, flags saA
TCP DMZ 10.5.63.29:2220 TRANSFER 10.20.100.21:59274, idle 0:00:00, bytes 0, flags saA
TCP DMZ 10.5.63.29:2221 TRANSFER 10.20.100.21:55782, idle 0:00:00, bytes 0, flags saA

I saw directly that the TCP connection was not open because the connection flag was “saA” what means outbound SYN was send and a connection is reserved but no SYN ACK came back. The problem in the end was that between these two servers was a VPN and that the IP network was missing in both crypto map’s what was then easy to find and solve in the end.

The command “show conn ?” give you enough information and when it comes to troubleshooting that you need to know basic troubleshooting skills because some stuff is not easy to find out over the ASDM and to use the command line instead.

Here the overview over the ASA TCP connection flags which are important to know or at least to know where to look them up 😉

Here the document from Cisco: ASA TCP Connection Flags (Connection build-up and teardown)