[ofa-general] IPOIB CM (NOSRQ) patch -memory footprint

[Pradeep Satyanarayana]

Here are my thoughts about limiting the memory footprint for IPOIB CM
(NOSRQ) patch:

By default, cap the NOSRQ memory usage to 1GB. The default recvq_size
is set to 128. Therefore for 64KB packets this would imply a maximum of
128 endpoints.

-Make the maximum number of endpoints a module parameter with a default
value of 128.

-The NOSRQ limit of 1GB is also made a module parameter. However, 1GB is
the default limit and could be changed as needed (by the administrator)
depending on the system configuration, application needs and so on. The
server would return a "REJ" message upon receiving a "REQ", whenever one
of these limits (i.e. max number of endpoints or the max NOSRQ memory
usage) is reached. Currently, we only check for the maximum number of
endpoints -hard coded to 1024.

-The IPOIB CM (NOSRQ) patch is practically transparent to HCAs that
support SRQ like the Topspin HCA and, such HCAs should not be
impacted at all.

-Currently we allocate a default of 64KB for the ring buffer elements,
and this buffer size is not linked to the mtu. In the future, we could
allocate buffers based on the mtu and link that into the computation of
the memory cap. This way customers who might want to use a smaller mtu
could use a larger number of endpoints, or a larger recvq_size without
exceeding the memory cap.


Would this approach address the issues of scalability and enable IPOIB
CM to be turned as the default?


Pradeep