Recently I had a strange ARP problem between an Cisco ASA firewall and an Cisco router (provider router) on an internet line in one of our remote offices. Periodically the office lost the network connectivity.
From the first look the ARP table seemed fine:
# sh arp | i OUTSIDE OUTSIDE 212.0.107.169 000f.e28a.1f7a 348
The ARP resolution was not working properly, the firewall was waiting for responses or even lost the ARP entry from the router. From the debugging output you can see that the firewall was in pending state and waiting for the router to respond:
# clear arp OUTSIDE 212.0.107.169 arp-req: generating request for 212.0.107.169 at interface OUTSIDE arp-send: arp request built from 212.0.107.170 0a00.0a00.0010 for 212.0.107.169 at 3637391690 arp-req: generating request for 212.0.107.169 at interface OUTSIDE arp-req: request for 212.0.107.169 still pending arp-req: generating request for 212.0.107.169 at interface OUTSIDE arp-req: request for 212.0.107.169 still pending arp-req: generating request for 212.0.107.169 at interface OUTSIDE arp-req: request for 212.0.107.169 still pending arp-in: response at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.170 0a00.0a00.0010 arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3637391710 arp-in: resp from 212.0.107.169 for 212.0.107.170 on OUTSIDE at 3637391710 arp-send: sending all saved block to OUTSIDE 212.0.107.169 at 3637391710
The same happen to normal ARP updates and the reason why we lost periodically the connectivity because the router didnt respond at all.
Our provider quickly figured out that there was a problem with the device and replaced the router.
ARP table output:
# sh arp | i OUTSIDE OUTSIDE 212.0.107.169 000f.e28a.1f7a 303
Here the normal ARP behaviour ones the router was replaced, the router responded directly to ARP requests:
# clear arp OUTSIDE 212.0.107.169 arp-req: generating request for 212.0.107.169 at interface OUTSIDE arp-send: arp request built from 212.0.107.170 0a00.0a00.0010 for 212.0.107.169 at 3717553710 arp-in: response at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.170 0a00.0a00.0010 arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3717553710 arp-in: resp from 212.0.107.169 for 212.0.107.170 on OUTSIDE at 3717553710
Normal ARP updates:
arp-in: request at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.171 0000.0000.0000 arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3717983740