[openib-general] Re: A Couple of User CM Questions

[Rimmer, Todd]

> > > 2. Any idea on how device would be supported ? Would this 
> just be with
> > > an IB device number or a string like "mthca0" ?
> > 
> >   I was thinking that the destination GID in the req_event 
> path record
> > would be sufficient to identify the local device/port, but I had not
> > taken a look at sidr_req_event. I think this is the correct 
> GID to use
> > for creating the QP, but I have not taken a look at how 
> easy it is to
> > turn a GID into a suitable device/portin user verbs. Thoughts?
> 
> and source GID (in the REQ primary path) on the active side ? 
> 
> That seems like it would work but wouldn't you need to walk the GID
> tables on each port on each IB device until you find the 
> match ? Sounds
> a little expensive but would only be done once per connection.
> 
> The advantage of GID is this would be common across all IB 
> devices. Not
> sure what else could be other than a deterministic numbering of IB
> devices locally which I believe does not currently exist.
> 
> -- Hal

We found it useful to allow a flexible listener bind and matched REQ based on a comparison of the following fields:
//                                          Field is Used by models as follows:
// 				                 Listen     Peer Connect   Sidr Register
// SID                                     Y             Y              Y
// local GID                             option          Y         future option
// local LID                             option          Y         future option
// QPN                                  wildcard         Y           wildcard
// EECN                                 wildcard         Y           wildcard
// CaGUID                               wildcard         Y           wildcard
// remote GID                            option          Y         future option
// remote LID                            option          Y         future option
// private data discriminator length     option        option         option
// private data discriminator value      option        option         option

The search was done against listening CEPs, no search of port GIDs was necessary for the receiver.  The listener (if it chose to listen on a specific set of GIDs or ports), would need to query the CA to obtain the GID list.  The sender is already required to supply GIDs in order to formulate a proper REQ (GIDs should come from Path Record).

We found the private data discriminator to be necessary to handle multiple instances of the same app on the same node, such as MPI processes, in which case the listener rank was used as the private data which must be matched.

Fields marked as option could be omitted by the listener, in which case all values for that would match (hence a listener could listen on all ports very easily).  Fields marked as wildcard were always ignored for the given connection type.  These fields were required per IBTA in Peer Connect REQ handling.  We have found limited use for SIDR so did not add the LID/GID capabilities there.  LID is easily possible, GID is probably not worth the work.  Unlike REQ, SIDR_REQ does not include the GIDs, so for SIDR GID would only apply to packets with GRH.

Todd Rimmer