Proxy ARP & ICMP Redirect in vShield Edge NIC – Explained

Its been quite some time when I wrote my last post. I am still not able to get time to update my blog. Hopefully changing the time zone will settle it after this month.

Well, its pretty good weather outside in Palo Alto and I found some time to show you something interesting. Today I want to talk about Proxy ARP Enablement and ICMP Redirection features in vShield Edge devices.

Now before I talk about the tradeoff and what it does, let me know when do you get to this point.

After you deploy your vSM, you now want to deploy your Edge Devices to make it ready either for a standard deployment or for a Cloud deployment. Now, most people today use it with combination of vCloud.

So, by now we all know that today our vShield Edge device can support up to 10 Internal & External interfaces. By default, when you deploy your first Edge, it get deployed with one external interface and then you may want to add other interfaces as well, either internal or external. Now when you start adding another leg on this Edge, it will look like as below:

Edge-NIC-Add

If you further scroll it down, there you will see this:

Proxy-ARP

Here you see two options and those are:

  1. Enable Proxy ARP
  2. Send ICMP Redirect

You might wonder as to use these options or take the default or not to use these at all. Now let me tell you what it is and how should you tackle these in your environment. I will use a Use Case for this and will chuck it out on two different options.

Firstly I will talk about the Proxy ARP, it’s Use Case, a sample design and a walkthrough of this technology.

When you enable proxy ARP it will allow VMs on the same IP subnet to be segmented beyond the normal layer 3 boundary (i.e. router) and still be able to find each other.

Now let me tell you how ARP works, ARP works to allow a source and destination VM in the same network to discover each others IP address by issuing a layer 2 broadcast to the all MAC addresses host i.e. FF.FF.FF.FF.FF.FF. Every VM on the broadcast segment will open and inspect the broadcast L2 frame and check the L3 header. When the VM discovers that the frame was not meant for them the frame is dropped. A switch will flood the ARP out of all ports if it doesn’t already know the MAC address or if it’s ARP table is full.

The receiving VM picks up the ARP request, recognizes the destination IP is itself and sends a unicast ARP back to the source of the ARP along with it’s MAC address . The destination host also may store the MAC and IP address of the sender in it’s own cache.

Well that is ARP, but hey stop, then what is proxy ARP?

With proxy ARP we essentially gets the benefits of ARP on a much larger scale to help to ‘blend’ disperate physical networks to create one logical network. Proxy ARP is all about ethernet segmentation. On a large network imagine a network of 1000 hosts, its a massive broadcast domain and as such will cause you a lot of hassle for things like ARP. Any layer 2 broadcast frame will be sent everywhere and that is bad. We’ll also get unnecessarily high traffic volumes across the network due to lack of segmentation. So what we may have done is to whip a router in there for segment the network

Now let us take this example below. Look at this Use Case sample design. Here we are talking about routing in between two different vApp but within the same Org. In this sample design, we have three NIC defined in out Edge device. vNIC0 is connected to the external interface. vNIC1 and vNIC2 are connected to Internal vAPP network. So what is Proxy ARP and how it is beneficial in this design.

SameOrgRouting

We have two vApps in the same Org with around 254 IP users per vApp making a total of little more that 500 IP addresses. We have chosen 2 x /24 network to provide us with 500+ usable hosts. The router is configured with IP as shown on the picture and also proxy ARP is enabled.

The VMs are given an IP address in the 192.168.0.0/24 range on one vApp and the 192.168.1.0/24 on the other. Now lets show you the mechanism of proxy ARP.

Lets say we have a VM, VM A in first vApp with IP address 192.168.0.50 and another VM, VM C in second vApp with IP Address 192.168.1.50. Now VM A trying to connect to VM C on IP address 192.168.1.50. VM A will send an ARP out for the IP address of VM C. The router, configured for Proxy ARP then ‘arps out’ via it’s locally connected network on 192.168.1.0/24 for VM A. VM C sees the ARP request coming from the router (Edge Device) and adds the routers MAC address as the source MAC for VM A then responds using a unicast ARP reply. The router (Edge Device) sees the reply, add that information to it’s own MAC table and proxies the response back to VM A. VM A then puts the MAC address of router into it’s table as the source for VM C.

This mechanism has a downside as well, on a large scale design you may need to consider the effects on the Proxy ARP on your Edge Device with regards to the MAC table. As the proxying grows, the table will grow and that consumes RAM and CPU cycles of your Edge Device! If you push your Edge that always polling out for ARP, you may hit performance issues. However with the gift of Large and X-Large Edge we can mitigate that too.

 

Now let me talk about the ICMP Redirect part. ICMP redirect messages are used by routers to notify the VMs on the data link that a better route is available for a particular destination.

The assumption on simple vApp Networks is that the VMs only need minimal routing information and can rely on Edge Devices having knowledge of the topology of the internetwork and all of the best routes. That’s why VMs are typically only configured with an IP address of a default router (in this case Edge Device NIC). Any remote traffic from the VM is forwarded to the Edge Gateway. While this makes it easier to configure the VMs, in IP internetworks where there are multiple Edge Device and it’s interface and subnet on a given network, the behavior of sending all remote traffic to the same router can produce non-optimal host routing. To prevent the perpetuation of non-optimal guest traffic routing, Edge Gateway can update the routing tables of hosts using an ICMP Redirect message.

Behavior of ICMP Redirect is different for Linux and Cisco.

Cisco’s behavior

  • The interface on which the packet comes into the router is the same interface on which the packet gets routed out
  • The subnet or network of the source IP address is on the same subnet or network of the next-hop IP address of the routed packet
  • The datagram is not source-routed
  • The kernel is configured to send redirects. (By default, Cisco routers send ICMP redirects.)

Linux behavior

  • The interface on which the packet comes into the router is the same interface on which the packet gets routed out, and the underlying device is shared media
  • For packet forwarding on same interface between different subnets, user can use the send-redirect control

In this example, the Edge Device is connected to two different Ethernet segment. The default gateway for VM A is configured to use vNIC1 IP. VM A sends a packet to vSE to reach the destination on VM C at 192.168.1.50. vSE, after it consults its routing table, finds that the next-hop to reach VM C at 192.168.1.50 is its own interface on vNIC2. Now vSE must forward the packet out the same interface (in this case vNIC1) on which it was received. vSE now forwards the packet to vNIC2 and also sends an ICMP redirect message to VM A. This informs VM A that the best route to reach VM C at 192.168.1.50 is by way of interface vNIC2. VM A then forwards all the subsequent packets destined for VM C at 192.168.1.50 to interface vNIC2.

 

About Prasenjit Sarkar

Prasenjit Sarkar is a Product Manager at Oracle for their Public Cloud with primary focus on Cloud Strategy, Oracle Openstack, PaaS, Cloud Native Applications and API Platform. His primary focus is driving Oracle’s Cloud Computing business with commercial and public sector customers; helping to shape and deliver on a strategy to build broad use of Oracle’s Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) offerings such as Compute, Storage, Java as a Service, and Database as a Service. He is also responsible for developing public/private cloud integration strategies, customer’s Cloud Computing architecture vision, future state architectures, and implementable architecture roadmaps in the context of the public, private, and hybrid cloud computing solutions Oracle can offer.

3 thoughts on “Proxy ARP & ICMP Redirect in vShield Edge NIC – Explained

  1. I have noticed that the ICMP redirect is not sticking in Windows and CentOS VMs (I have not tried others). I can see the redirect in a packet capture but the route is only respected for ~10 sec and is never added to the route table on the VM. I have verified that ICMP redirects are allowed. This behavior did not occur in VCD 1.5.1 with VSM 5. Any insight that can be provided would be extremely helpful.