[ofw] Dual port HCA's in target and initiator, both ports hooked up, don't enumerate each drive twice in WInOF SRP

[Fab Tillier]

Hi Chris,

I'm going from memory here, so please excuse any inaccuracies - it's been a while since I touched that part of the code...

Chris Worley wrote on Fri, 15 Oct 2010 at 16:16:25

> Using the RC4 OFED WinOF code in W2K8R2.
> 
> With one dual-port IB HCA in the target and initiator, and both ports
> hooked up between target and initiator, the SRP initiator only
> enumerates one set of drives; the two paths between the initiator and
> the target should enumerate the drives twice, once as duplicates.
> 
> The target is setup s.t. all drives are in the default group, so there
> is no lun masking.
> 
> Both ports show up as active in ibstat on both target and initiator.
> 
> Remove one link, and it continues to enumerate the drives; replace
> that link and remove the other link and it continues to enumerate the
> drives; does SRP have some sort of multipathing built-in when two
> HCA's are connected back-to-back?  If so, what is the protocol for
> using both ports (i.e. fail-over only, round-robin, etc...)?

SRP keeps track of multiple paths over a single HCA, and hides them from the OS.  This dates to pre-built-in-MPIO support in Windows, and made it so that one would get rudimentary failover support.  The backup paths only get used when the active path fails.

>  If so,
> how is it disabled (I want to use Windows built-in multipathing)?

This would require code changes to the SRP driver, as well as to the bus driver to report an SRP device per local HCA port (rather than an SRP device per local HCA.)

> If I have two HCA's in the target, and connect each of the initiator
> ports to a different target HCA, then I see the duplicate drives.

That's because the paths aren't between the same devices.  Keep in mind that an HCA has a unique node GUID, that all its ports share - this is how the bus driver groups paths per device (the bus driver reports the paths available to the SRP driver via the IOC PnP logic.)

> Likewise, If I have two HCA's in the initiator, and connect each of
> the target ports to a different HCA, then I see the duplicate drives.
> 
> And, of course, if I have two HCA's and use one port each for the
> links, then I see the duplicate drives.
> 
> The only problem is using one HCA on each side, and connecting both
> ports; only one set of drives is enumerated, no duplicates.

This is "by design"...  Probably worth changing, or at least make it configurable so that those with MPIO can get the behavior they desire.

-Fab