[ofa-general] Service ID scope in IB Arch Spec A3.2.2 is incorrect, right?

[Scott Guthridge]

Sean,

I can see what you're saying -- the "IsCommunicationManagementSupported"
and "IsDeviceManagementSupported" capability flags are port attributes, and
despite the implication of the way the DM agent is drawn in A8.2.3 figure
309, different ports of the same TCA could be implemented to return
different sets of service ID's, and this would make it possible to tie
particular services to individual ports.  I think the IB arch. spec. could
be a little more clear on what the intent is here.


Let me tell you what I'm *really* asking...

I'm architecting an SRP driver.  In Fibrechannel FCP, SCSI ports coincide
1-1 with physical FC ports -- a FCP session doesn't span adapter ports or
migrate between ports.  SRP ports, in contrast, are provided by a service
on an IOC which is not necessarily tied to a physical IB port.  An IOU may
present the same IOC from all ports.  Further, a single SRP SCSI session
may be composed of several SRP channels which may span adapter ports and
migrate between ports.

I'm trying to decide if the TCA should provide a single SCSI port visible
from all physical ports, or if it should behave more like Fibrechannel
where there is a separate SCSI port for every IB port.  Comparing the two
approaches, I can see the following trade-offs:

SRP port for each TCA/IOU:

   Can support multiple SRP channels and transparent IB path migration
   without the need for a multipath driver at the SCSI or block device
   level.  *But* this only works within a single TCA -- if you have
   multiple paths where more than one TCA is visible from a given host, you
   still need a multipath capable driver above IB.

SRP port for each IB port:

   Behaves more like fibrechannel, which people are used to.  Multipath is
   handled by a single mechanism above the IB level.  Simpler multipath
   model may make it easier for management software to obtain and report
   the health of a given path.


Can anyone think of any other differences to consider between the two
approaches?  Any thoughts on which is more "correct"?


Scott




                                                                                                                       
                      Sean Hefty                                                                                       
                      <mshefty at ichips.i        To:       Scott Guthridge/Watson/IBM at IBMUS                              
                      ntel.com>                cc:       general at lists.openfabrics.org                                 
                                               Subject:  Re: [ofa-general] Service ID scope in IB Arch Spec A3.2.2 is  
                      10/30/07 03:01 PM         incorrect, right?                                                      
                                                                                                                       




Scott Guthridge wrote:
> IB Architecture Spec, r1.2 section A3.2.2 says [emphasis added]:
>
>    Each *port* on a CA may support a set of services. ... Since *not all
>    ports* support the same set of services...
>
> and later:
>
>    "it is the combination of the Port GID and Service ID that identifies
a
>    particular service provider"
>
>
> But this seems to contradict chapter 12 (communication management) and
> chapter A8 (device management) which consistently associate services with
> channel adapters, not ports.  See 12.6.5 table 99 (CA GUID), 12.6.8 table
> 103 (CA GUID), 12.9.9 connection state table (CA GUID), etc.  Similarly,
> figure 309 "I/O Components and Relationships" in section A8.2.3 that
shows
> the DM agent being a component of the I/O Unit, and because the I/O unit
is
> associated with a single TCA, it follows that the DMA belongs to the
> channel adapter, not to a particular port.
>
> The CM implementation in OFED 1.2 supports this notion that services are
> defined per CA, not per port in that ib_create_cm_id doesn't take a port
> number.

In short, I really don't know the answer here.

Automatic path migration allows a connection to migrate between ports on
the same HCA.  So, from at least that view, a service can be viewed as
being defined per CA, not per port.  However, service records are tied
to a specific port.

Also, the IB CM is not required to be implemented on each port; CM
support is a per port attribute.  Viewing the CM as a service is per
port, not per CA.

I'd need to verify this, but I don't think that a connection request
architecturally even has to be received on the port that the connection
will use.  I wouldn't interpret too much from the ib_create_cm_id API.
The use of the CA GUID in CM req/rep message helps detect
stale/duplicate connections, since QPs are per HCA, and not per port.

I'm not sure how this relates into section A8.

> So am I correct that A3.2.2 has it wrong?  Would it be right to say that
> with respect to provided services, all ports of a given CA are equal?

I don't believe you can say this.  The port attributes can be different.
  The ports could be on different subnets.  An SM could be running on
one port, but not another.  Etc.

- Sean