[openib-general] [PATCH] [CM] add private data comparison to match REQs with listens

[Fab Tillier]

> From: Caitlin Bestler [mailto:caitlinb at broadcom.com]
> Sent: Friday, December 02, 2005 12:28 PM
> 
> Fab Tillier wrote:
> >
> > Your focus is strictly on TCP socket semantics, but we're
> > talking about IB CM functionality - the IB CM does more than
> > just provide TCP socket semantics.
> >
> > Imagine a user-mode IB application (not virtualization mind
> > you, but just an
> > app) that wants to listen on a given SID (because the SID
> > defines the application), but wants to discriminate incoming
> > requests based on some content in the private data.  Multiple
> > instances of that application can only work properly if the
> > CM performs the private data comparison to properly dispatch
> > the incoming requests to the right user-mode process.
> >
> > If the CM doesn't provide the private data compare
> > functionality, then the app developer needs to create a
> > kernel agent to perform this functionality for the app.  The
> > functionality is simple enough, and has potential value to
> > multiple clients, that it makes sense to have the IB CM provide it.
> 
> You are proposing that the API be made more complex and
> you do not have any justification other that something
> some user-mode application *might* want to do.

In Windows, the Winsock Direct provider does exactly this, and would require a
kernel component if the IB CM wasn't providing this functionality.  WSD uses the
private data to carry the IP address of the client, but uses its own private
data format.

I believe some native-IB MPI implementations make use of similar functionality,
using the rank of the process in the private data.  This allows such
implementations to limit the size of their SID range to a single value or a
single value per job.

> Why are these different user-mode applications sharing
> a Service ID in the first place? On what basis do they
> trust each other? How do they co-ordinate their filtering?
> Couldn't they use CM redirection to share the Service ID?

The world is larger than just TCP-compatible applications.  I'm not talking
about two applications sharing a SID, but two instances of one application
sharing a SID.  Imagine processes in a larger MPI job - the SID can be used to
differentiate jobs, and the private data comparison can be used to differentiate
different processes within that job.  Alternatively, the SID could be constant,
and the job ID and rank could be expressed in the private data, with the
IB-level CM performing all the proper dispatching.

I don't think CM redirection would work since both apps are on the same system,
and share the same CM.  There can only be a single connection ID namespace per
HCA GUID or things quickly become ambiguous.

> The goal was supposed to be providing TCP-compatible
> connection setup, but this is describing something that
> is decidedly un-TCP-like. TCP applications differentiate
> within the daemon, or redirect connections. If they split
> connections based upon packet content it is only done by
> very sophisticated L7 load balancers that identify cookies
> or other HTTP content.

The goal of the CMA *is* to support TCP-compatible semantics, but that is not
the goal of the IB CM.  The IB CM already keeps track of listens and performs
lookups when a REQ comes in based on service ID.  Extending it to do some fairly
basic extra checking is far simpler than adding duplicate lookup functionality
to the CMA.  This allows the IB CM to do all the filtering at once as part of
REQ matching, and thus simplifies the CMA.  It also allows user-mode apps to use
similar functionality without requiring a kernel agent.

Anyhow, do you have an objection to the CM enabling simple comparisons on
private data?  If so, what are your objections (aside from it not being
TCP-like)?

Thanks,

- Fab