Technical Tip

QUESTION

How can I set up and configure a network aggregation under Solaris 10 and Solaris 11?

ANSWER
An aggregation is similar to an IPMP link created for failover, but has other advantages:-
  • An aggregation is treated as one device for configuration and management, and uses only one IP address.
  • In-bound as well as outbound load spreading is provided, so extra bandwidth is available.
  • Automatic failover is provided, to the remaining working links in the aggregation.
  • Policies can be applied to the traffic flows.
  • Parallel aggregations between servers can be created, without going through a switch. 
Early versions of Solaris 10 do not have this facility - you would need to download the Sun Trunking software to get it.

Solaris 10 configuration:-
To show the available network interfaces:-
# dladmin  show-link
eri0            type: legacy    mtu: 1500       device: eri0
qfe0            type: legacy    mtu: 1500       device: qfe0
qfe1            type: legacy    mtu: 1500       device: qfe1
qfe2            type: legacy    mtu: 1500       device: qfe2
qfe3            type: legacy    mtu: 1500       device: qfe3

If the device shows up as type:legacy you cannot use it in an aggregation, therefore devices such as eri, hme and qfe are not supported in current Solaris production releases. In Opensolaris (and presumably later production releases) a new driver is included which does support the older devices.
Output from dladm will then show the following:-
# dladm    show-link
LINK  CLASS  MTU  STATE  OVER
eri0    phys   1500   up         --
hme0  phys   1500   up         --
qfe0   phys   1500  down     --

There are conditions to consider:-
  • If you need to run an aggregation through a network switch, that switch must support aggregation technology.
  • An interface that has been plumbed cannot become a member of an aggregation.
  • Currently, interfaces must be of the GLDv3 type: xge, e1000g, and bge.
  • NOTE: Later releases of Solaris 10 will support aggregations on what are now considered legacy devices such as eri. qfe and hme.
  • All interfaces in the aggregation must run at the same speed and in full duplex mode.
To set up an aggregation using dladm, the basics steps are:-
# dladm  create -aggr  -d  bge0  -d  bge1  1
Create an aggregation number 1 from interfaces bge0 and bge1.

# ifconfig  aggr1   plumb  192.168.200.98  up
Brings the aggregation interface up.

# dladm   show-aggr
key: 1 (0x0001) policy: L4      address: 0:3:ba:7:84:5e (auto)

device   address           speed         duplex  link    state
bge0     0:3:ba:7:84:5e    1000  Mbps    full    up      attached
bge1     0:3:ba:7:84:5e   1000  Mbps    full    up      attached
 
To add another interface to the aggregation:-
# dladm  add-aggr  -d  bge2  -d  bge3  1
# dladm  show-aggr  1
key: 1 (0x0001) policy: L4      address:0:3:ba:7:84:5e (auto)
           device       address                 speed           duplex  link    state
           bge0      0:14:4f:1:c8:b0   1000  Mbps    full    up      attached
           bge1      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached
           bge2      0:14:4f:1:c8:b1   1000  Mbps    full    up      attached
           bge3      0:14:4f:1:c8:b2   1000  Mbps    full    up      attached

To remove a device from the aggregation:-
# dladm  remove-aggr  -d  bge0  1
# dladm  show-aggr  1
key: 1 (0x0001) policy: L4      address: 0:3:ba:7:84:5e (auto)
           device       address                 speed           duplex  link    state
           bge1      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached
           bge2      0:14:4f:1:c8:b1   1000  Mbps    full    up      attached
           bge3      0:14:4f:1:c8:b2   1000  Mbps    full    up      attached
 
 
It is possible to set policies for aggregations, such as whether to use LACP protocol or not.
Here is an example which sets LACP mode to active, and the timer interval to short.
# dladm   modify-aggr  -l   active  -t  short  1

The default outbound load spreading is based on ports, but this can also be changed - see the dladm man page for full details.
Aggregations can be assigned to zones using IP instances (Update 4 onwards) - see zonecfg for details.
dladm can assign an aggregation temporarily to a zone, until the next reboot:-
dladm  set-linkprop   -t  -p  zone=apache  aggr1
The -t option must be used.
Use zonecfg to assign aggr1 permanently to the apache zone.

dladm also provides functions to display statistics, and not just for aggregations:-
# dladm  show-dev  -s  eri0
                ipackets  rbytes         ierrors opackets        obytes      oerrors
eri0            806693    122733634   0       808629    336565292   0


# dladm  show-aggr  -s
key: 1                   ipackets    rbytes      opackets   obytes          %ipkts %opkts
           Total        158030    31173663    12            504
           qfe1         132878    26014941    12            504                84.1    100.0
           qfe2          8818       1818890       0              0                    5.6     0.0
           qfe3          8174        1671697       0             0                    5.2     0.0
           qfe0          8160        1668135       0             0                    5.2     0.0

Solaris 11 configuration:-

There are two types of link aggregation supported in Solaris 11:-
Trunk (802.3ad) aggregations, which provide load spreading at MAC/IP/PORT level, but need 802.3ad network switch support.
DLMP (Data Link MultiPathing) where the network links can be connected to separate switches. DLMP load spreading is at link level,
and is intended primarily as the underlying link for VNICs. (Virtual Network Interfaces) These are used extensively with
virtual machines calless zones.

To show the available network interfaces:-
# dladm  show-phys
LINK     MEDIA        STATE        SPEED      DUPLEX       DEVICE
net1     Ethernet        unknown            0          unknown   e1000g1
net2     Ethernet        unknown            0          unknown   e1000g2
net0     Ethernet         up                 1000         full             e1000g0
net3     Ethernet        unknown            0          unknown   e1000g3

# dladm show-link
LINK    CLASS         MTU        STATE            OVER
net1      phys              1500       unknown          --
net2      phys              1500       unknown          --
net0      phys              1500       up               --
net3      phys              1500       unknown          --

Firstly, an 802.3ad compliant aggregation.
There are conditions to consider:-
If you need to run a trunk aggregation through a switch, that switch must support 802.3ad aggregation technology.
Interfaces must be of the GLDv3 type: xge, e1000g, and bge.
It is unlikely that other types would be supported in Solaris 11 in any case.
All interfaces in the aggregation must run at the same speed and in full duplex mode.
No IP instances must initially exist on the component interfaces.

To set up an aggregation using dladm, the basics steps are:-
# dladm  create-aggr  -L active -l  net1  -l  net2  1
Create an aggregation number 1 (aggr1) from interfaces net1 and net2.
-L active enables active LACP.

# ipadm  create-ip  aggr1
# ipadm  create-addr  -T  static  -a  130.100.1.5/16  aggr1/v4
Brings the aggregation interface aggr1 up with address 130.100.1.5
# dladm   show-aggr
LINK    MODE    POLICY    ADDRPOLICY        LACPACTIVITY         LACPTIMER
aggr1      trunk     L4           auto                         off                  short

Use dlstat to display link statistics.

To add another interface to the aggregation:-
# dladm  add-aggr  -l  net3  aggr1
# dladm  show-aggr  -x
LINK    PORT     SPEED     DUPLEX        STATE    ADDRESS      PORTSTATE
aggr1          --            1000Mb     full            up            0:14:4f:97:7d:f          --
               net1        1000Mb     full            up            0:14:4f:97:7d:f     attached
               net2      1000Mb     full            up            0:14:4f:97:7d:10   attached
               net3      1000Mb     full            up            0:14:4f:97:7d:11   attached

# dlstat  show-aggr  aggr1
    LINK    PORT    IPKTS    RBYTES        OPKTS    OBYTES
              aggr1      --        5.34M      435.78M            1.48M      190.05M
              aggr1     net1        2.02M      159.87M          570.29K       73.00M
              aggr1      net2        2.13M      181.61M          570.09K       72.97M
              aggr1     net3        1.19M       94.31M          344.38K       44.08M


To remove a device from the aggregation:-
# dladm  remove-aggr  -l  net0  1

To remove aggr1:-
# dladm  delete-aggr 1
or..
# dladm  delete-aggr  aggr1

Secondly, a DLMP aggregation.

To create a DLMP-mode aggregation:-
# dladm  create-aggr  -m  dlmp  -l  net1  -l  net2  -l  net3  aggr1
-m dlmp denotes that this is a DLMP-mode aggregation.

The links that form part of the aggregation can be connected to different switches which do not need 802.3ad support,
thus eliminating the switch as a point of failure, and the need for any switch configuration.

As VNICs are created they are associated with an aggregation port, and if the port fails, they will be switched to another (working) port.
If the number of VNICs exceeds the number of aggregation ports, then the ports are shared between them.

VNICs to be connected to the aggr1 interface are created as follows:-
# dladm  create-vnic  -l  aggr1  vnic0

We hope this has been useful, see below for additional notes
Mick Hosegood - First Alternative.

NOTES

  • All this is covered on the Solaris 10 and 11 Administration courses!

First Alternative course tutors can answer questions like this ... and are happy to do so. Look around our site for relevant courses in Linux -Unix - Solaris

close window