[ofa-general] [PATCH] librdmacm/man: update man pages to clarify connection request params

[Sean Hefty]

Document connection requests parameters in rdma_connect(),
rdma_accept(), and rdma_get_cm_event(), specifically regarding
initiator_depth and responder_resources.

Signed-off-by: Sean Hefty <sean.hefty at intel.com>
---
Doug, these are the updates to the man pages that I've made to
better document some of the parameters based on your feedback.  I
will look at some code changes to try to trap errors setting
initiator_depth and responder_resources earlier in the
connection setup.  I will request that these changes go into
OFED 1.3.

The only other request I can recall was adding the ability to migrate
an id to another event channel.  I still need to look into this more,
but this would miss OFED 1.3.

 man/rdma_accept.3       |   22 +++++++++---
 man/rdma_ack_cm_event.3 |    3 +-
 man/rdma_connect.3      |   12 +++++--
 man/rdma_get_cm_event.3 |   83 ++++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 109 insertions(+), 11 deletions(-)

diff --git a/man/rdma_accept.3 b/man/rdma_accept.3
index e7c3788..c0c12d8 100644
--- a/man/rdma_accept.3
+++ b/man/rdma_accept.3
@@ -11,7 +11,8 @@ rdma_accept \- Called to accept a connection request.
 .IP "id" 12
 Connection identifier associated with the request.
 .IP "conn_param" 12
-Information needed to establish the connection.
+Information needed to establish the connection.  See CONNECTION PROPERTIES
+below for details.
 .SH "DESCRIPTION"
 Called from the listening side to accept a connection or datagram
 service lookup request.
@@ -25,13 +26,16 @@ rdma_accept is called on the new rdma_cm_id.
 .SH "CONNECTION PROPERTIES"
 The following properties are used to configure the communication and specified
 by the conn_param parameter when accepting a connection or datagram
-communication request.  Users should use the conn_param values reported in
-the connection request event to determine appropriate values for these fields
-when accepting.
+communication request.  Users should use the rdma_conn_param values reported
+in the connection request event to determine appropriate values for these
+fields when accepting.  Users may reference the rdma_conn_param structure in
+the connection event directly, or can reference their own structure.  If the
+rdma_conn_param structure from an event is referenced, the event must not be
+acked until after this call returns.
 .IP private_data
 References a user-controlled data buffer.  The contents of the buffer are
-transparently passed to the remote side as part of the communication request.
-May be NULL if private_data is not required.
+copied and transparently passed to the remote side as part of the
+communication request.  May be NULL if private_data is not required.
 .IP private_data_len
 Specifies the size of the user-controlled data buffer.  Note that the actual
 amount of data transferred to the remote side is transport dependent and may
@@ -39,9 +43,15 @@ be larger than that requested.
 .IP responder_resources
 The maximum number of outstanding RDMA read and atomic operations that the
 local side will accept from the remote side.  Applies only to RDMA_PS_TCP.
+This value must be less than or equal to the local RDMA device attribute
+max_qp_rd_atom and the responder_resources value reported in the connect
+request event.
 .IP initiator_depth
 The maximum number of outstanding RDMA read and atomic operations that the
 local side will have to the remote side.  Applies only to RDMA_PS_TCP.
+This value must be less than or equal to the local RDMA device attribute
+max_qp_init_rd_atom and the initiator_depth value reported in the connect
+request event.
 .IP flow_control
 Specifies if hardware flow control should be used.  Applies only to RDMA_PS_TCP.
 .IP retry_count
diff --git a/man/rdma_ack_cm_event.3 b/man/rdma_ack_cm_event.3
index 20ccd9c..3c24357 100644
--- a/man/rdma_ack_cm_event.3
+++ b/man/rdma_ack_cm_event.3
@@ -12,6 +12,7 @@ Event to be released.
 .SH "DESCRIPTION"
 All events which are allocated by rdma_get_cm_event must be released,
 there should be a one-to-one correspondence between successful gets
-and acks.
+and acks.  This call frees the event structure and any memory that it
+references.
 .SH "SEE ALSO"
 rdma_get_cm_event(3), rdma_destroy_id(3)
diff --git a/man/rdma_connect.3 b/man/rdma_connect.3
index a0a9095..71d5594 100644
--- a/man/rdma_connect.3
+++ b/man/rdma_connect.3
@@ -11,7 +11,7 @@ rdma_connect \- Initiate an active connection request.
 .IP "id" 12
 RDMA identifier.
 .IP "conn_param" 12
-connection parameters.
+connection parameters.  See CONNECTION PROPERTIES below for details.
 .SH "DESCRIPTION"
 For an rdma_cm_id of type RDMA_PS_TCP, this call initiates a connection request
 to a remote destination.  For an rdma_cm_id of type RDMA_PS_UDP, it initiates
@@ -25,8 +25,8 @@ by the conn_param parameter when connecting or establishing datagram
 communication.
 .IP private_data
 References a user-controlled data buffer.  The contents of the buffer are
-transparently passed to the remote side as part of the communication request.
-May be NULL if private_data is not required.
+copied and transparently passed to the remote side as part of the
+communication request.  May be NULL if private_data is not required.
 .IP private_data_len
 Specifies the size of the user-controlled data buffer.  Note that the actual
 amount of data transferred to the remote side is transport dependent and may
@@ -34,9 +34,15 @@ be larger than that requested.
 .IP responder_resources
 The maximum number of outstanding RDMA read and atomic operations that the
 local side will accept from the remote side.  Applies only to RDMA_PS_TCP.
+This value must be less than or equal to the local RDMA device attribute
+max_qp_rd_atom and remote RDMA device attribute max_qp_init_rd_atom.  The
+remote endpoint can adjust this value when accepting the connection.
 .IP initiator_depth
 The maximum number of outstanding RDMA read and atomic operations that the
 local side will have to the remote side.  Applies only to RDMA_PS_TCP.
+This value must be less than or equal to the local RDMA device attribute
+max_qp_init_rd_atom and remote RDMA device attribute max_qp_rd_atom.  The
+remote endpoint can adjust this value when accepting the connection.
 .IP flow_control
 Specifies if hardware flow control should be used.  Applies only to RDMA_PS_TCP.
 .IP retry_count
diff --git a/man/rdma_get_cm_event.3 b/man/rdma_get_cm_event.3
index 252a7ab..987ead5 100644
--- a/man/rdma_get_cm_event.3
+++ b/man/rdma_get_cm_event.3
@@ -21,7 +21,88 @@ modifying the file descriptor associated with the given channel.  All
 events that are reported must be acknowledged by calling rdma_ack_cm_event.
 Destruction of an rdma_cm_id will block until related events have been
 acknowledged.
-.SH "EVENTS"
+.SH "EVENT DATA"
+Communication event details are returned in the rdma_cm_event structure.
+This structure is allocated by the rdma_cm and released by the
+rdma_ack_cm_event routine.  Details of the rdma_cm_event structure are
+given below.
+.IP "id" 12
+The rdma_cm identifier associated with the event.  If the event type is
+RDMA_CM_EVENT_CONNECT_REQUEST, then this references a new id for that
+communication.
+.IP "listen_id" 12
+For RDMA_CM_EVENT_CONNECT_REQUEST event types, this references the
+corresponding listening request identifier.
+.IP "event" 12
+Specifies the type of communication event which occurred.  See EVENT TYPES
+below.
+.IP "status" 12
+Returns any asynchronous error information associated with an event.  The
+status is zero unless the corresponding operation failed.
+.IP "param" 12
+Provides additional details based on the type of event.  Users should
+select the conn or ud subfields based on the rdma_port_space of the
+rdma_cm_id associated with the event.  See UD EVENT DATA and CONN EVENT
+DATA below.
+.SH "UD EVENT DATA"
+Event parameters related to unreliable datagram (UD) services: RDMA_PS_UDP and
+RDMA_PS_IPOIB.  The UD event data is valid for RDMA_CM_EVENT_ESTABLISHED and
+RDMA_CM_EVENT_MULTICAST_JOIN events, unless stated otherwise.
+.IP "private_data" 12
+References any user-specified data associated with RDMA_CM_EVENT_CONNECT_REQUEST
+or RDMA_CM_EVENT_ESTABLISHED events.  The data referenced by this field matches
+that specified by the remote side when calling rdma_connect or rdma_accept.
+This field is NULL if the event does not include private data.  The buffer
+referenced by this pointer is deallocated when calling rdma_ack_cm_event.
+.IP "private_data_len" 12
+The size of the private data buffer.  Users should note that the size of
+the private data buffer may be larger than the amount of private data
+sent by the remote side.  Any additional space in the buffer will be
+zeroed out.
+.IP "ah_attr" 12
+Address information needed to send data to the remote endpoint(s).
+Users should use this structure when allocating their address handle.
+.IP "qp_num" 12
+QP number of the remote endpoint or multicast group.
+.IP "qkey" 12
+QKey needed to send data to the remote endpoint(s).
+.SH "CONN EVENT DATA"
+Event parameters related to connected QP services: RDMA_PS_TCP.  The
+connection related event data is valid for RDMA_CM_EVENT_CONNECT_REQUEST
+and RDMA_CM_EVENT_ESTABLISHED events, unless stated otherwise.
+.IP "private_data" 12
+References any user-specified data associated with the event.  The data
+referenced by this field matches that specified by the remote side when
+calling rdma_connect or rdma_accept.  This field is NULL if the event
+does not include private data.  The buffer referenced by this pointer is
+deallocated when calling rdma_ack_cm_event.
+.IP "private_data_len" 12
+The size of the private data buffer.  Users should note that the size of
+the private data buffer may be larger than the amount of private data
+sent by the remote side.  Any additional space in the buffer will be
+zeroed out.
+.IP "responder_resources" 12
+The number of responder resources requested of the recipient.
+This field matches the initiator depth specified by the remote node when
+calling rdma_connect and rdma_accept.
+.IP "initiator_depth" 12
+The maximum number of outstanding RDMA read/atomic operations
+that the recipient may have outstanding.  This field matches the responder
+resources specified by the remote node when calling rdma_connect and
+rdma_accept.
+.IP "flow_control" 12
+Indicates if hardware level flow control is provided.
+.IP "retry_count" 12
+For RDMA_CM_EVENT_CONNECT_REQUEST events only, indicates the number of times
+that the recipient should retry send operations.
+.IP "rnr_retry_count" 12
+The number of times that the recipient should retry receiver not ready (RNR)
+NACK errors.
+.IP "srq" 12
+Specifies if the sender is using a shared-receive queue.
+.IP "qp_num" 12
+Indicates the remote QP number for the connection.
+.SH "EVENT TYPES"
 The following types of communication events may be reported.
 .IP RDMA_CM_EVENT_ADDR_RESOLVED
 Address resolution (rdma_resolve_addr) completed successfully.