[openib-general] How do we prevent starvation say between TCP over IPOIB / and SRP traffic ?

[Bernard King-Smith]

Richard Frank <Richard.Frank at oracle.com> wrote:

Richard> Are there any mechanisms available to the client process to manage
the
Richard> QoS level for the various supported ULPs
(SDP,TCP,UDP,RDS,SRP,iSER,etc)
Richard> either at the ULP level or some combination of process and ULP -
or
Richard> perhaps even at the connection level ?

To manage QoS the question is who knows about all the traffic traversing a
specific adapter. For most kernel traversing protocols (IP, iSER, iSCSI,
etc) you can sometimes do this in the device driver, where you can examine
the headers as a packet is expedited and manage it there. Unfortunately you
are adding processing in the driver which can end up impacting bandwidth on
high speed adapters. You also introduce additional overhead hence higher
CPU utilization.

You can also introduce that in the adapter but I am not sure if the IB HCA
spec has this capability defined.

Richard> Using the same example, the cluster node monitors might set the
Richard> priority / QoS level of the heart beats to be more important than
normal
Richard> SRP/iSER traffic to ensure no starvation ?

When you add user space protocols such as SDP and MPI, then all bets are
off. The kernel has no knowledge of the traffic from the users so can't
properly manage QoS across all protocols. The user space traffic can mess
up the kernel traversing protocols QoS calculations. The only place to
handle QoS if user space protocols are used is in the adapter. Some
switches can handle QoS once the traffic gets there but can't help the
adapters in the nodes.


Bernie King-Smith
IBM Corporation
Server Group
Cluster System Performance
wombat2 at us.ibm.com    (845)433-8483
Tie. 293-8483 or wombat2 on NOTES

"We are not responsible for the world we are born into, only for the world
we leave when we die.
So we have to accept what has gone before us and work to change the only
thing we can,
-- The Future." William Shatner