Data Domain Link Aggregation

Setting Up Link Aggregation

SUMMARY

Using the link aggregation feature in DDOS can result in an increase in throughput, however depending on several factors it could result in the opposite. The purpose of this article is to help determine and implement the appropriate configuration of Link Aggregation. As DD OS does not support many of the various dynamic port participation methods offered (such PAGp, MLT, etc.) it is important that the ports within the LAG are statically configured on the switch.

APPLIES TO

All DD OS versions that support the link aggregation feature

SOLUTION

Determine Appropriate Link Aggregation

The following is what should be considered when trying to decide on aggregation. If no aggregation is to be done then failover should be considered. Therefore, the last choice given is failover as an alternative to aggregation.

Direct Connect

1. mode roundrobin (if it is supported by the media servers)

2. separate NIC per media server (if there are enough NICs)

3. mode xor-L3L4

4. failover (if aggregation cannot be used)

Private Network

1. separate NIC per media server (if there are enough NICs)

2. mode xor-L3L4

3. mode xor-L2 (if there are a suitable number of clients)

4. failover (if aggregation cannot be used)

Local Network

1. separate NIC per media server (if there are enough NICs)

2. mode xor-L3L4

3. mode xor-L2 (if there are a suitable number of clients)

4. failover (if aggregation can not be used)

Remote Network

1. separate NIC per media server (if there are enough NICs)

2. mode xor-L3L4

3. failover (if aggregation can not be used)

Implementation of Data Domain Link Aggregation and Failover

There are two link aggregation methods supported by Data Domain:

Round Robin and

Balanced-xor (setup manually on both sides).

LACP(starting in 5.0 for 1 Gb and 5.1 for 10Gb)

The balanced-xor aggregation is selected by choosing the specific hash that is supported:

Layer 2 or

Layer 3+4

Layer 2+3(starting in 5.0)

There are four virtual interfaces that can be used to define the aggregation or failover:

veth0

veth1

veth2

veth3

Before aggregation can be utilized a virtual interface must be created. This can be done by running the following command

#net create virtual <veth#>

Example:

#net create virtual veth0

After a virtual interface is created any of the physical links that are available on the system can be included in the aggregation group: eth0, eth1, eth2, eth3, etc. The onboard links (eht0 and eth1) have only recently been allowed to be added. Older installations of the Data Domain software may not allow those two links to be aggregated.

To specify aggregation of eth2 and eth3 in the virtual interface veth0 one of the following commands would be used:

#net aggregate add veth0 mode roundrobin interfaces eth2 eth3

The first network packet sent to veth0 will be forwarded to one of the interfaces and the next packet would be forwarded to the other. Sending of packets will continue to alternate between the interfaces until there are no more packets or a link fails. If eth3 loses physical connection all packets are sent through eth2 until the eth3 link is brought back up. To make this effective the other side of the network will also need to be setup to do round robin. For direct connect (the only topology that is recommended for round robin) the media server will have to be able to setup and support round robin.

#net aggregate add veth0 mode xor-L2 interfaces eth2 eth3

The aggregation used would be balanced-xor. The packets are distributed across eth2 and eth3 based on XOR of the source and destination MAC addresses. Because there are only 2 links to be aggregated the lowest bit is used to determine the interface to use for the packet. If the result is 0 one interface will be chosen. If the result is 1 the other interface will be used. To get the packets to be spread across the two links requires that data is sent to more than one destination and the MAC addresses of the destination needs to be different in such a way that XOR results provide a different number. This means that one address needs to be odd and the other needs to be even. If there are three links that are aggregate, the XOR result is split 3 ways. There has to be at least two media servers there must be at least two media servers with odd and even MAC addresses to get any aggregation at all. In general, this aggregation should not be used with less than 4 media servers.

#net aggregate add veth0 mode xor-L3L4 interfaces eth2 eth3

The aggregation used with this command will also be balanced-xor. The packets are distributed across eth2 and eth3 based on the XOR of the source IP address, destination IP address, source port number, and the destination port number. The result gives a number in which the lowest bit is used to determine which link to use to send the packet. An even result will go over one and an odd result will go over the other. With three links the result is divided by 3 with the remainder determining which interface to use. This aggregation would be used when there are a lot of connections (there is one connection per stream) or a lot of media servers or both. This is the mode of choice for Data Domain, but some switches do not support this type of hashing.

#net aggregate add veth0 mode lacp hash xor-L3L4 interfaces eth4a eth4b

The aggregation used with this command will also be lacp-xor. The packets are distributed across eth2 and eth3 based on the XOR of the source IP address, destination IP address, source port number, and the destination port number. The data flow control follows the same mechanism used by balanced mode except it adds a control protocol to monitor the interfaces with a minimal amount automated administration of the interfaces including better sensing when a interface fails. The sensing goes beyond the sensing of carrier loss to the sensing of the ability to send and receive data. The heartbeat can be sent out every second or every 30 seconds. The default is every 30 seconds. The interval determines how fast the bonding will sense the link is no longer communicating and will stop using the interface. Once every 30 seconds is less invasive, but it will take longer to mark the link as down and there may be connection timeouts while it is waiting.

#net failover add veth0 interfaces eth2 eth3

This is not aggregation but the command will group together interfaces eth2 and eth3 for failover. There is only one failover type supported. If the active physical link goes away the data is sent to the second physical link. The active interface is determined by which link comes up first when it is setup. This is nondeterministic. It is dependent on several factors such as switch activity, network activity, and which interface is brought up first when they are enabled. The active one can be determined by specifying one of the links as primary. The primary interface will always be set as active if it is UP and RUNNING.

Note: For DDBOOST/OST backups, use ifgroup instead of aggregation for better throughput and performance.

Configure and enable the virtual interface

After the virtual interface has been created and the right aggregation method assigned, the interface will need to be configured with an IP address and subnet mask and then enabled. The following command can be used to do accomplish this:

#net config <veth#> <IP address> netmask <subnet address> up

Example:

#net config veth1 192.168.192.168 netmask 255.255.255.0 up

References

From <https://emc–c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004RuG>

Link aggregation increases network throughput, across a LAN

Link aggregation performance is impacted by:

Link and switch speed

The quantity of data the Data Domain system can process

Out-of-order packets

The number of clients

The number of streams (connections) per client

In Link Aggregation Please consider:

Links can be part of only one group.

Aggregation is only between two systems.

All links n a group must have the same speed.

All links n a group must be either half-duplex or full-duplex.

No changes to the network headers are allowed.

You must have a unique address across aggregation groups.

Frame distribution must be predictable and consistent.

Round robin: Transmits packets in sequential order from the first available link through the last in the aggregated group.

Balanced: Sends data over the interfaces as determined by the selected hash method. All associated interfaces on the switch must be grouped into an EtherChannel (trunk).

LACP: Is similar to Balanced, except for the control protocol that communicates with the other end and coordinates what links, within the bond, are available. It provides heartbeat failover.

When planning interface link aggregation, consider the following:

Changes to disabled Ethernet interfaces flush the routing table. It is recommended that you make interface changes only during scheduled
maintenance downtime. Afterwards, reconfigure the routing rules and gateways.

o Enable aggregation on an existing virtual interface by specifying the physical interfaces and mode and giving it an IP address.

o DD2500 on-board 10G Base-T interfaces ethMe and ethMf do not support link aggregation.

o On DD4200, DD4500, and DD7200 systems, the ethMA interface does not support link aggregation.

o Virtual interfaces must be created from identical physical interfaces. For example, all copper, all optical, all 1 Gb, or all 10 Gb. However, 1 Gb interfaces support bonding a mix of copper and optical interfaces. This applies to virtual interfaces across different cards with identical physical interfaces, except for Chelsio cards. For Chelsio cards, only failover is supported, and that is only across interfaces on the same card.

Advertisements

About Ahmad Sabry ElGendi

https://www.linkedin.com/pub/ahmad-elgendi/94/223/559
This entry was posted in Data Domain, E20-385, EMC. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s