[openib-general] proposed MAD layer features

[Hefty, Sean]

I'd like to propose adding the following features to the MAD layer.
The purpose behind the request is to enable CM MRA support.

- Expand the retries field in the send_wr.wr.ud to be usable by
  all MADs expecting a response (i.e. timeout_ms > 0).  Currently,
  its use is restricted to RMPP.

- Add a mechanism to modify a posted MAD request.

Having the retries field is almost necessary for large RMPP messages,
but a linear timeout/retry mechanism is useful to the CM as well.
Clients that need an adjustable retry mechanism could set retries to
zero and perform manual retries, similar to what is required today.

The intent of modifying a posted MAD request is to allow the CM to
change the timeout value for a posted REQ or REP upon receiving an
MRA.  RMPP actually uses a similar mechanism to adjust MAD timeouts
after receiving an ACK, but the mechanism is not exposed outside of
the MAD layer.

A drawback is that I'm doubtful that other clients would find the
ability to modify a posted MAD request useful.  It could be
if clients were notified of retry attempts, which would let them
increase the timeout value.  (This would also close a minor hole
where a reply is received late, but in between retry attempts.)

Comments?

- Sean