Multipathing within a Multisite Cluster – A Design Constraint

HP P4000 Multi-Site SANs enable vSphere 4 clusters to be stretched across locations to provide multi-site VMotion, HA (High Availability), DRS (Distributed Resource Scheduler), and FT. Multi-Site SAN configurations use synchronous replication in the underlying SAN to create a single SAN that spans both location. This allows vSphere 4 clusters to act exactly the same way they do when physically located in a single location. When connecting ESX or ESXi hosts to a Multi-Site SAN each of the virtual IP addresses (VIPs) of the SAN from each site should be listed in the discovery list of the ESX or ESXi software initiator. Path selection policy for Multi-Site SAN volumes should be set to fixed (default).


Multi-pathing iSCSI for vSphere 4

Native iSCSI multi-pathing in vSphere 4 provides superior bandwidth performance by aggregating network ports. Configuring iSCSI multi-pathing requires at least two network ports on the virtual switch. The following steps must be performed on each ESX or ESXi server individually.


Create a second VMkernel port on the virtual switch for iSCSI. For each VMkernel port on the virtual switch assign a different physical network adapter as the active adapter. This ensures the multiple VMkernel ports use different network adapters for their I/O. Each VMkernel port should use a single physical network adapter and not have any standby adapters.


From the command line, bind both VMkernel ports to the software iSCSI adapter. The vmk# and vmhba## must match the correct numbers for the ESX or ESXi server and virtual switch you are configuring, for example:


> esxcli swiscsi nic add –n vmk0 –d vmhba36

> esxcli swiscsi nic add –n vmk1 –d vmhba36


Once configured correctly, perform a rescan of the iSCSI adapter. An iSCSI session should be connected for each VMkernel bound to the software iSCSI adapter. This gives each iSCSI LUN two iSCSI paths using two separate physical network adapters. As an example refer to Figure below for NIC teaming tab for VMkernel properties.

To achieve load balancing across the two paths, datastores should be configured with a path selection policy of round robin. This can be done manually for each datastore in the vSphere client or ESX can be configured to automatically choose round robin for all datastores. To make all new datastores automatically use round robin, configure ESX to use it as the default path selection policy from the command line:


> esxcli corestorage claiming unclaim –type location

> esxcli nmp satp setdefaultpsp –satp VMW_SATP_DEFAULT_AA –psp VMW_PSP_RR

> esxcli corestorage claimrule load

> esxcli corestorage claimrule run


It is important to note that native vSphere 4 multi-pathing cannot be used with HP P4000 Multi-Site SAN configurations that utilize more than one subnet and VIP (virtual IP). Multiple paths cannot be routed across those subnets by the ESX/ESXi 4 initiator.


Now the question is can we tweak this to get it to work (though still it will not be supported by vendor and us J). Yes of course we can. But please make sure that whatever I am saying right now is mainly for academic purpose.


Routing Setup with Vmware NMP

With iSCSI Multipathing via MPIO, the VMkernel routing table is bypassed in determining which outbound port to use from ESX. As a result of this VMware officially says that routing is not possible in iSCSI SANs using iSCSI Multipathing.   Further – routing iSCSI traffic via a gateway is generally a bad idea.   This will introduce unnecessary latency – so this is being noted only academically.  We all agree on this pointDO NOT ROUTE iSCSI TRAFFIC.

But, for academic thoroughness, you can provide minimal routing support in vSphere because a route look-up is done when selecting the vmknic for sending traffic. If your iSCSI storage network is on a different subnet AND you iSCSI Multipathing vmkNICs are on the same subnet as the gateway to that network, routing to the storage works. For example look at this configuration:

  • on the vSphere ESX/ESXi server:
    • vmk0 General purpose vmkNIC
    • vmk1 iSCSI vmkNIC
    • vmk2 iSCSI vmkNIC
    • Default route:
  • on the iSCSI array:
    • iSCSI Storage port 1:
    • iSCSI Storage port 2:

In this situation, vmk1 and vmk2 are unable to communicate with the two storage ports because the only route to the storage is accessible through vmk0, which is not set up for iSCSI use. If you add the route:

Destination: Gateway: (and have a router at the gateway address)

then vmk1 and vmk2 are able to communicate with the storage without interfering with other VMkernel routing setup.


About Prasenjit Sarkar

Prasenjit Sarkar is a Product Manager at Oracle for their Public Cloud with primary focus on Cloud Strategy, Cloud Native Applications and API Platform. His primary focus is driving Oracle’s Cloud Computing business with commercial and public sector customers; helping to shape and deliver on a strategy to build broad use of Oracle’s Infrastructure as a Service (IaaS) offerings such as Compute, Storage, Network & Database as a Service. He is also responsible for developing public/private cloud integration strategies, customer’s Cloud Computing architecture vision, future state architectures, and implementable architecture roadmaps in the context of the public, private, and hybrid cloud computing solutions Oracle can offer.

Leave a Reply