This page after years continues to get amazing traffic! 8/27/2020!
Currently working on the http://radiantcreators.com project. More about me on the “About” page there. For an example of Radiant Creators interviews checkout “Interview With Robert Allen – Bitcoin and Friends https://youtu.be/-i5rt_i7Gvc“
Now for what you are looking for below!
UCS FC uplinks can be bound to vHBA’s by VSAN or Pining. Each has pro’s and con’s.
- UCS balanced (round robin FLOGI’s to each uplink).
- If a path fails round robin kicks in and FLOGI’s to next uplink
- VSAN’s are “Best Practice” from Cisco
- Less configuration
- New uplinks can be added dynamically but FLOGI’s will NOT redistribute immediately. A reboot of Blades is needed to redistribute over new FC uplink
- Control Freak friendly.
- Servers per FC uplink can be managed
- If a path fails round robin does not happen. Link stays down until restored. This is an advantage in ensuring QoS bandwidth per vHBA/Blade
VSAN uplink example
Multiple FC uplinks are round robin load balanced with a common VSAN
Best practice for VSAN’s is a unique VSAN ID per uplink VSAN. Also the VSAN ID and the FCoE VLAN ID should not be the same. I prefer VSAN 1000 on FIA, and VSAN 2000 on FIB. If there are multiple fabrics being connected to I increment by 100. 1000, 1100, 1200,etc. Remember VSAN’s should Never be common between FI’s. Dual Fabric is the only way, if you want to keep failover and controller port login balance a certainty etc.
Bind each group of FC uplink to corresponding VSAN. If FC uplinks are added later they are simply added to VSAN. Easy peasy huh?
Bind each vHBA to corresponding VSAN.
Pining/VSAN uplink example
Pinning and unique VSAN’s can co-exist. A Brocade switch upstream has completely no idea what a VSAN is. So realize that the VSAN is strictly taking place on the FI. So for Brocade upstram VSAN 1 could be used. BUT best practice is unique VSAN’s (an no pining technically). The example below is only using VSAN 1. Needs to be fixed to proper VSANs 1000 & 2000, but other than that, example is 100%.
Basically steps are the same with added Pining. Create FC uplink ports (best to use the same on each FI if possible but not mandatory).
In vHBA template select proper VSAN and Pin group.
With most UCS builds I use the VSAN method. But the control freak in my likes Pining better. Following the “Rules of SAN” and my Brocade background Pining is a no-brainer. But the VSAN method fits UCS best practices and does allow adding and removing uplinks easy. Pining has made needed changes very difficult for me.
Unless there is a reason to hardcose pining, go VSAN. Even if you have some traditional SAN experience and it is hard to deal with.
Some useful CLI commands
To see what port vHBA’s are logged into fabric
connect nxos b
show npv external-interface-usage
NPV Traffic Usage Information:
See what VLAN a vHBA is in
connect nxos b
show npv status
npiv is enabled
disruptive load balancing is disabled
Interface: fc1/29, VSAN: 2100, FCID: 0x027e00, State: Up
Interface: fc1/30, VSAN: 2100, FCID: 0x02e600, State: Up
Interface: fc1/31, VSAN: 1100, FCID: 0x010600, State: Up
Interface: fc1/32, VSAN: 1100, FCID: 0x010700, State: Up
Number of External Interfaces: 4
Interface: vfc911, VSAN: 1100, State: Up
Interface: vfc919, VSAN: 1100, State: Up
Interface: vfc927, VSAN: 1100, State: Up
Interface: vfc935, VSAN: 1100, State: Up
Interface: vfc943, VSAN: 1100, State: Up
Interface: vfc951, VSAN: 1100, State: Up
Interface: vfc959, VSAN: 1100, State: Up
More related articles below.
UCSGuru has a Great Post on “Understanding UCS VIF Paths” that is a good next step in this study.