Skip to content

Cisco UCS FC uplinks to Brocade Fabric

This page after years continues to get amazing traffic! 8/27/2020!

Currently working on the http://radiantcreators.com project. More about me on the “About” page there. For an example of Radiant Creators interviews checkout “Interview With Robert Allen – Bitcoin and Friends https://youtu.be/-i5rt_i7Gvc

Now for what you are looking for below!

UCS FC uplinks can be bound to vHBA’s by VSAN or Pining. Each has pro’s and con’s.

Cisco UCS 6248 with 2208XP IOM's and Brocade Boot SAN
Cisco UCS 6248 with 2208XP IOM’s and Brocade Boot SAN

VSAN

  • UCS balanced (round robin FLOGI’s to each uplink).
  • If a path fails round robin kicks in and FLOGI’s to next uplink
  • VSAN’s are “Best Practice” from Cisco
  • Less configuration
  • New uplinks can be added dynamically but FLOGI’s will NOT redistribute immediately. A reboot of Blades is needed to redistribute over new FC uplink

Pining

  • Control Freak friendly.
  • Servers per FC uplink can be managed
  • If a path fails round robin does not happen. Link stays down until restored. This is an advantage in ensuring QoS bandwidth per vHBA/Blade

VSAN uplink example

Multiple FC uplinks are round robin load balanced with a common VSAN

UCS FC Uplinks
UCS FC Uplinks

Best practice for VSAN’s is a unique VSAN ID per uplink VSAN. Also the VSAN ID and the FCoE VLAN ID should not be the same. I prefer VSAN 1000 on FIA, and VSAN 2000 on FIB. If there are multiple fabrics being connected to I increment by 100. 1000, 1100, 1200,etc. Remember VSAN’s should Never be common between FI’s. Dual Fabric is the only way, if you want to keep failover and controller port login balance a certainty etc.

Creating FC Uplinks in VSAN
Creating FC Uplinks in VSAN

Bind each group of FC uplink to corresponding VSAN. If FC uplinks are added later they are simply added to VSAN. Easy peasy huh?

Bind FC Uplink to FIA VSAN 1000
Bind FC Uplink to FIA VSAN 1000
Bind FC Uplink to FIB VSAN 2000
Bind FC Uplink to FIB VSAN 2000

Bind each vHBA to corresponding VSAN.

Bind vHBA to VSAN FIA
Bind vHBA to VSAN FIA
Bind vHBA to VSAN FIB
Bind vHBA to VSAN FIB

 

Pining/VSAN uplink example

Pinning and unique VSAN’s can co-exist. A Brocade switch upstream has completely no idea what a VSAN is. So realize that the VSAN is strictly taking place on the FI. So for Brocade upstram VSAN 1 could be used. BUT best practice is unique VSAN’s (an no pining technically). The example below is only using VSAN 1. Needs to be fixed to proper VSANs 1000 & 2000, but other than that, example is 100%.

Basically steps are the same with added Pining. Create FC uplink ports (best to use the same on each FI if possible but not mandatory).

Pin Group FIA
Pin Group FIA
Pin Group FIB
Pin Group FIB

In vHBA template select proper VSAN and Pin group.

Pinned vHBA template FIA
Pinned vHBA template FIA
Pinned vHBA template FIB
Pinned vHBA template FIB

 

With most UCS builds I use the VSAN method. But the control freak in my likes Pining better. Following the “Rules of SAN” and my Brocade background Pining is a no-brainer. But the VSAN method fits UCS best practices and does allow adding and removing uplinks easy. Pining has made needed changes very difficult for me.

Unless there is a reason to hardcose pining, go VSAN. Even if you have some traditional SAN experience and it is hard to deal with.

Some useful CLI commands

To see what port vHBA’s are logged into fabric
connect nxos b
show npv external-interface-usage

NPV Traffic Usage Information:
—————————————-
Server-If External-If

—————————————-
vfc911 fc1/31
vfc919 fc1/31
vfc927 fc1/32
vfc935 fc1/32
vfc943 fc1/31
vfc951 fc1/32
vfc959 fc1/32
vfc967 fc1/31
vfc975 fc1/32
vfc983 fc1/32
vfc991 fc1/31
vfc999 fc1/32
vfc1007 fc1/31
vfc1015 fc1/31
vfc1023 fc1/32
vfc1031 fc1/32

See what VLAN a vHBA is in
connect nxos b
show npv status

npiv is enabled

disruptive load balancing is disabled

External Interfaces:
====================
Interface: fc1/29, VSAN: 2100, FCID: 0x027e00, State: Up
Interface: fc1/30, VSAN: 2100, FCID: 0x02e600, State: Up
Interface: fc1/31, VSAN: 1100, FCID: 0x010600, State: Up
Interface: fc1/32, VSAN: 1100, FCID: 0x010700, State: Up

Number of External Interfaces: 4

Server Interfaces:
==================
Interface: vfc911, VSAN: 1100, State: Up
Interface: vfc919, VSAN: 1100, State: Up
Interface: vfc927, VSAN: 1100, State: Up
Interface: vfc935, VSAN: 1100, State: Up
Interface: vfc943, VSAN: 1100, State: Up
Interface: vfc951, VSAN: 1100, State: Up
Interface: vfc959, VSAN: 1100, State: Up

More related articles below.

UCSGuru has a Great Post on “Understanding UCS VIF Paths” that is a good next step in this study.

Cisco UCS Boot From SAN HowTo

Published inBrocadeCisco UCS CentralPapersSANYouTube

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *