Strange ARP issue between ASA and Cisco router

Recently I had a strange ARP problem between an Cisco ASA firewall and an Cisco router (provider router) on an internet line in one of our remote offices. Periodically the office lost the network connectivity.

From the first look the ARP table seemed fine:

# sh arp | i OUTSIDE
OUTSIDE 212.0.107.169 000f.e28a.1f7a 348

The ARP resolution was not working properly, the firewall was waiting for responses or even lost the ARP entry from the router. From the debugging output you can see that the firewall was in pending state and waiting for the router to respond:

# clear arp OUTSIDE 212.0.107.169
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-send: arp request built from 212.0.107.170 0a00.0a00.0010 for 212.0.107.169 at 3637391690
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-req: request for 212.0.107.169 still pending
arp-in: response at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.170 0a00.0a00.0010
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3637391710
arp-in: resp from 212.0.107.169 for 212.0.107.170 on OUTSIDE at 3637391710
arp-send: sending all saved block to OUTSIDE 212.0.107.169 at 3637391710

The same happen to normal ARP updates and the reason why we lost periodically the connectivity because the router didnt respond at all.

Our provider quickly figured out that there was a problem with the device and replaced the router.

ARP table output:

# sh arp | i OUTSIDE
OUTSIDE 212.0.107.169 000f.e28a.1f7a 303

Here the normal ARP behaviour ones the router was replaced,  the router responded directly to ARP requests:

# clear arp OUTSIDE 212.0.107.169
arp-req: generating request for 212.0.107.169 at interface OUTSIDE
arp-send: arp request built from 212.0.107.170 0a00.0a00.0010 for 212.0.107.169 at 3717553710
arp-in: response at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.170 0a00.0a00.0010
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3717553710
arp-in: resp from 212.0.107.169 for 212.0.107.170 on OUTSIDE at 3717553710

Normal ARP updates:

arp-in: request at OUTSIDE from 212.0.107.169 000f.e28a.1f7a for 212.0.107.171 0000.0000.0000
arp-set: added arp OUTSIDE 212.0.107.169 000f.e28a.1f7a and updating NPs at 3717983740

 

Cisco Catalyst static CAM table entries

That’s maybe not interesting for everybody but when you use Windows Network Load Balancing in your network you should definitively configure static CAM table entries otherwise your VLAN will be flood with multicast traffic. You can create different VLANs for your Windows NLB instances to separate the traffic but that’s more an work around and with static CAM entries nicer from the design.

The static CAM table entries just restrict the multicast traffic to specific ports on your switch where your Servers are located and keep otherwise the network free.

On your router you will have an entry like that:

arp 10.0.0.100 0300.5e11.1111

An static ARP entry with an multicast MAC address

Now to restrict the multicast traffic you use the following command:

mac-address-table static 0300.5e11.1111 vlan 100 interface gi0/10 gi0/11

Which just means that the multicast traffic for VLAN 100 will be flood through the interfaces Gi0/10 and Gi0/11 and all other interfaces will not see the multicast traffic. Its a bit an administrative overhead and you have to think a little bit about that before you can implement but an clean traffic flow within your network.

If you want to read more about it have a look here: Catalyst Switches for Microsoft Network Load Balancing Configuration Example