[ofa-general] Can Mellanox cards handle scatter gather I/O?

[Kent Baxley]

Evidently, we're seeing performance problems when a card on one side
is capable of scatter gather IO but the mellanox driver is not able to
handle it. Then a whole lot of time is spent gathering the data into
one skbuff.

This is currently being observed on RHEL4.5 with OFED 1.2.

Is the Mellanox card capable of doing scatter gather IO and this just
a driver limitation or is the mellanox card just not able to do scatter
gather IO?

Here is the original description of the problem in case the original
synopsis was oversimplified: 

  A second issue arises in how current kernels and specifically NETDEV
  handle scatter/gather and the related SKBs. Kernel.org is currently
  in the middle of a massive re-write of this section but existing
  ports may incur an extra buffer copy. Of the existing backports
  there are two skbuff variations. One is simply a pointer to the SKB
  while the other is a list of SKBs. If a driver supports the list and
  forwards a packet to a driver which supports only a pointer then
  NETDEV is stuck copying the list of buffers into a new single SKB.
  At 10gbs this copy can sap a processor's capacity to move data. The
  identified case where this conflict arises is the Myri10GE driver
  which supports lists of SKBs and Mellanox IB (IPoIB) driver which
  does not.

Initial search through the web suggests that the IB verbs layer does
support scatter gather IO and that it has a limit of 60 segments.
However, we've as of yet been unable to determine if that is
implemented in the verbs layer or if it is done in hardware.

If anyone has any info that might help here, it will be appreciated.

Thanks.