[openib-general] RFC on SDP checkin

[Libor Michalek]

On Fri, Feb 04, 2005 at 01:26:32PM -0800, Tom Duffy wrote:
> On Fri, 2005-02-04 at 11:42 -0800, Libor Michalek wrote:
> >   You mean a quick description of the code, like the primary contents of
> > each file and how to get it to do something ?
> 
> That would be great start.

  OK, I've gone with option number 3. Here is a one or two line
description of each file, and a TODO list. Both are checked in
along with the code in the infiniband/ulp/sdp directory.

-Libor


FILES
-----
sdp_advt.[ch]  - Data structure. 
                 RDMA advertisments representing remote IOCBs. Used to 
                 track RDMA read and write requests initiated in the data
                 path.
sdp_actv.c     - Control path.
                 Active connection establishment interface to Connection
                 Manager.
sdp_buff.[ch]  - Data structure. 
                 Global pool of buffers used for data and control messages.
sdp_conn.[ch]  - Data structure. 
                 Connection data, including all state and information. 
                 Global connection tables.
sdp_event.c    - Control and Data path. 
                 Event notification for Connection Manager state changes. 
                 Completion Queue notification.
sdp_inet.c     - Control path. 
                 Interface to Linux AF_INET/SOCK_STREAM sockets. 
sdp_iocb.[ch]  - Data structure.
                 RDMA IO control structure representing local buffers which
                 can be used as RDMA targets by the remote connection peer
                 for RDMA read and write requests.
sdp_kvec.c     - AIO IOCB cancellation code. (TODO)
sdp_link.c     - Control path.
                 Active connection establishment interface to Address
                 Resolution and Path Record lookups.
sdp_pass.c     - Control path.
                 Passive connection establishment interface to Connection
                 Manager.
sdp_post.c     - Control path.
                 Requests made to Connection Manager common for active
                 and passive connections.
sdp_proc.[ch]  - Misc.
                 Linux /proc filesystem interface for reporting internal
                 state to userspace.
sdp_queue.[ch] - Data structure.
                 Generic FIFO/LILO queue and element definition used for
                 BUFF, ADVT, and IOCB data structures.
sdp_rcvd.c     - Data path.
                 Process receive CQ notifications. Member of the receive
                 half of the data path.
sdp_read.c     - Data path.
                 Process send CQ RDMA Read notifications. Member of the
                 receive half of the data path.
sdp_recv.c     - Data path.
                 Member of the receive half of the data path. Processes data 
                 from upper OS interface to the posting of buffers to a QPs
                 receive or send WQ.
sdp_send.c     - Data path.
                 Member of the send half of the data path. Processes data from
                 upper OS interface to the posting of buffers to a QPs send WQ.
sdp_sent.c     - Data path.
                 Process send CQ notifications. Member of the send half of
                 the data path.
sdp_wall.c     - Control path.
                 Transition between Operating specific state and protocol
                 control state. (TODO)
sdp_write.c    - Data path.
                 Process send CQ RDMA Write notifications. Member of the
                 send half of the data path.

sdp_buff_p.h   - Data structure. 
                 Global pool of buffers.
sdp_dev.h      - Misc.
                 Global definitions for limits and constants
sdp_main.h     - Misc.
                 Top level SDP include.
sdp_msgs.h     - Data structure.
                 IBTA specification SDP messages and headers.
sdp_proto.h    - Misc.
                 Function prototype definitions.

TODO
----
SDP 

  - Cosmetic cleanup as needed.

  - Reduce connection state machine complexity. Specifically there exists
    redundant tracking of connection state which should be merged for
    improved readability and maintainability.

  - Add full connection cleanup once Connection Manager IDLE is in place.

  - Reduce code bloat.

  - Enable FMRs once mthca support is provided.

  - Provide real support for SO_{RCV,SND}BUF
    - internal posted buffers
    - QP WQ size
    - CQ size

  - Create slow start algorithm for posted receive buffers to improve
    memory consumption in large connection count applications, especially
    for connections that are predominately sending data.

  - sdp_link needs locking protection.

  - sdp_link request cancellation support.

  - SDP RDMA support for blocking socket operations.

  - IPv6 addressing

  - Alternate Path Migration (APM) support.

  - Keepalive (zero byte RDMA write)

  - Socket duplication (i.e. suspend)

Possible Issues

  - SDP address resolution requires looking at the IPoIB device's private
    data structure inorder to determine interface's SGID, Port, and PKey.

  - Memory locking for AIO requires a call to do_mlock() which is not a
    kernel exported function, the method for calling the function is not
    standard.

Other

  - Kernel level transparent socket switch for dynamic TCP/SDP selection.
    (Possible SDP port mapper protocol from RDMAC)

  - Extended Sockets API (ES-API). Initial Linux AIO based implemetation?

  - New SOCK_DGRAM protocol?

  - NFS over SDP.

  - QP striping?