[openib-general] InfiniBand EndPort Liveness and Responsiveness

[Hal Rosenstock]

Hi,

Here is a writeup on proposed IB endport liveness and resposiveness.
This "grew" out of discussions on ibping on this list some time ago.
Thanks in advance for any comments on this.

-- Hal

InfiniBand EndPort Liveness and Responsiveness

Motivation

  While there exist methods to obtain some degree of the liveness and
  responsiveness of an InfiniBand end port using existing mechanisms
  (e.g. SM Get attribute), these rely on a lack of protection as well
  as not allowing sufficient flexibility. In addition, less of the 
  end port stack may be exercised.

  IBA supports general service (GS) vendor classes. There are 2 ranges
  of these: the first range does not support RMPP whereas the second
  range does. OpenIB has a vendor OUI of 0x001405. A vendor class of 
  0x34 (in vendor class 2 range) would be utilized for this. The class
  version in the MAD header indicates the version of this. Currently,
  this is version 1.

Packet Format
  (Refer to IBA 1.2 p. 1006 for vendor class 2 MAD format)

  Bytes 40-255 are defined as vendor data as follows:

In MAD if not RMPP'd, or in first MAD if RMPP'd:
  Byte 40: Type (1 byte)
    1 - Echo request
    2 - Echo response
    3 - Timestamp request
    4 - Timestamp response
    5 - LID/GUID information request
    6 - LID/GUID information response
  
  Byte 41-43: Reserved

  Byte 44-45: Identifier
  Byte 46-47: Sequence Number

  Byte 48-255: Data for Type 

In subsequent (non first) RMPP MADs:
  Byte 40-255: Continued Data for Type


Echo

The data received in the echo request message must be returned
in the echo reply message.

The identifier and sequence number may be used by the echo sender to aid 
in matching the replies with the echo requests. For example, the identifier 
might be used like a port in TCP or UDP to identify a session, and the sequence
number might be incremented on each echo request sent. The echoer returns
these same values in the echo reply.

can be RMPP'd or not

In MAD if not RMPP'd, or in first MAD if RMPP'd:
Byte 48-255: Echo data

Identifier. 16 bits.
This field is used to help match echo requests to the associated reply.
It may be set to zero.

Sequence number. 16 bits.
This field is used to help match echo requests to the associated reply.
It may be zero.

Echo Data. Variable length.
Implementation specific data.


Timestamp

The data received (a timestamp) in the message is returned in the reply
together with an additional timestamp. The timestamp is 32 bits of
milliseconds since midnight UT.

The Originate Timestamp is the time the sender last touched the message
before sending it, the Receive Timestamp is the time the echoer first touched
it on receipt, and the Transmit Timestamp is the time the echoer last touched
the message on sending it.

If the time is not available in miliseconds or cannot be provided with respect
to midnight UT then any time can be inserted in a timestamp provided the high
order bit of the timestamp is also set to indicate this non-standard value.

The identifier and sequence number may be used by the echo sender to aid in
matching the replies with the requests. For example, the identifier might be
used like a port in TCP or UDP to identify a session, and the sequence number
might be incremented on each request sent. The destination returns these same
values in the reply.

can be RMPP'd or not

In MAD if not RMPP'd, or in first MAD if RMPP'd:
Byte 48-51: Originate timestamp
Byte 52-55: Receive timestamp
Byte 56-59: Transmit timestamp 

The preferred form for a timestamp value (the "standard value") is in units of
milliseconds since midnight Universal Time. However, it may be difficult to
provide this value with millisecond resolution. For example, many systems use
clocks that update only at line frequency, 50 or 60 times per second.
Therefore, some latitude is allowed in a "standard value":
(a) A "standard value" MUST be updated at least 15 times per second (i.e., at
most the six low-order bits of the value may be undefined).
(b) The accuracy of a "standard value" MUST approximate that of operator-set
CPU clocks, i.e., correct within a few minutes.


LID/GUID information

Obtain the LID and port GUID of the requested port.

can be RMPP'd or not

In MAD if not RMPP'd, or in first MAD if RMPP'd:
Byte 48-49: LID
Byte 50-57: Port GUID