[openib-general] RDMA connection and address translation API

[Yaron Haviv]

> -----Original Message-----
> From: Roland Dreier [mailto:rolandd at cisco.com]
> Sent: Wednesday, August 24, 2005 7:29 PM
> To: Yaron Haviv
> Cc: James Lentini; Roland Dreier; openib-general at openib.org
> Subject: Re: [openib-general] RDMA connection and address translation
API
> 
> 
> Yaron, has anyone raised all this in the IBTA WG?
> 

I raised it about a year ago, but didn't really followed up on it 
At the time IBTA was also busy with other more urgent stuff (verb ext..)
We work with few key IBTA members to re-surface it with the need for an
abstract CM

See the following text that was proposed (a Year ago as is)
It is slightly different than your proposal but can be altered if needed

It basically uses SDP header and marks one of the fields with 01 (FlowC)
to indicate it's not SDP, this way even SDP can use it 
Also it covers some nice idea raised by MS & SUN to extend SDP to accept
PUT & GET operations for RDMA, so you can get a BSD like API with few
additional APIs rather than have a totally new API like DAPL


Establishing a TCP/iWarp like connections over InfiniBand
=========================================================

 In order to emulate an iWarp connection, it is required to open an 
 InfiniBand RC connection, associate it with IP addresses and TCP ports
 In addition protocols may transfer control/login packets before
 the migration to the RDMA mode; this requires exchanging receiver
buffer
 size and depth for initial usage (the ULP's will manage the flow
control
 for the duration of the connection).

 The mapping uses the same data structures already defined for
connection 
 establishment in SDP  (IBTA Socket Direct Protocol) which accomplish
the
 same goal of mapping TCP Sockets addressing to InfiniBand, the non 
 relevant SDP fields were Reserved. 

 iWarp emulation CM Request (Hello) Private Data header
  
        0                   1                   2                   3

        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 04    |      MID      |     Rsvd      |             bufs              |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 08    |                              len                              |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 12    |                           Reserved                            |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 16    |                           Reserved                            |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 20    | MajVer| MinVer| IPVer | FlowC |           Reserved            |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 24    |                          DesRemRcvSz                          |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 28    |                          LocalRcvSz                           |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 32    |         Local Port            |           Reserved            |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 36    |                       Src IP (127-96)                         |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 40    |                       Src IP ( 95-64)                         |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 44    |                       Src IP ( 63-32)                         |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 48    |                       Src IP ( 31-00)                         |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 52    |                       Dst IP (127-96)                         |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 56    |                       Dst IP ( 95-64)                         |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 60    |                       Dst IP ( 63-32)                         |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 64    |                       Dst IP ( 31-00)                         |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

             Figure 1 CM Hello private data structure   
  

 iWarp emulation CM Response (HelloReply) Private Data header

        0                   1                   2                   3

        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 04    |      MID      |     Rsvd      |             bufs              |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 08    |                              len                              |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 12    |                           Reserved                            |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 16    |                           Reserved                            |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 20    | MajVer| MinVer|    Reserved   |           Reserved            |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 24    |                           ActRcvSz                            |

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

             Figure 1 CM Reply private data structure   

 All the fields mentioned have the same functionality as defined in SDP
 The MID field can accept only the values 0 and 1, a new field called 
 FlowC is added that indicate that the ULP is responsible for the Flow
 Control and the Verb API usage (unlike SDP).

    MID - Message Identifier: 8 bits  
  
         0 - Hello message  
         1 - Reply message  

         Other values are reserved  

    FlowC - Flow Control and verb owner: 4 bits  
  
         0 - Transport owns flow control and may embed a 16 byte headers

             in RC Send (just like SDP does, and uses zero in that
field)  
         1 - ULP owns flow control and control the entire Send buffer
             (e.g. iSER, NFS use their own headers in Send operations)


         Other values are reserved