[ofa-general] Re: [PATCH 0/10 REV5] Implement skb batching and support in IPoIB/E1000

[Or Gerlitz]

Hi Krishna,

Krishna Kumar2 wrote:
> What I will do today/tomorrow is to run the rev5 (which I didn't run
> for mthca) on both ehca and mthca and get statistics and send it out.
> Otherwise what you stated is correct as far as rev4 goes. After giving
> latest details, I will appreciate any help from Mellanox developers.

good, please test with rev5 and let us know.

> Correct, for every 1 retransmission in the regular code, I see two
> retransmissions in batching case (which I assume is due to overflow at the
> receiver side as I batch sometimes upto 4K skbs). I will post the exact
> numbers in the next post.

transmission of 4K batched packets sounds like a real problem for the 
receiver side, with 0.5K send/recv queue size, its 8 batches of 512 
packets each were for each RX there is completion (WC) to process, SKB 
to alloc and post to the QP where for the TX there's only posting to the 
QP, processes one (?) WC and free 512 SKBs.

If indeed the situation is so unsymmetrical, I am starting to think that 
the CPU utilization at the sender side might be much higher with 
batching then without batching, have you looked into that?

> I was using 2.6.23-rc1 on receiver (which also has NAPI, but uses the
> old API - the same fn ipoib_poll()).

I am not with you. Looking on 2.6.22 and 2.6.23-rc5, for both their 
ipoib-NAPI mechanism is implemented through the function ipoib_poll 
being the polling api for the network stack etc, so what is the old API 
and where does this difference exist?

> This is TCP (without No Delay), datagram mode, I didn't change mtu from
> the default (is it 2K?).  Command is iperf with various options for different
> test buffer-size/threads.

You might want to try something lighter such as iperf udp test, where a 
nice criteria would be to compare bandwidth AND packet loss between 
no-batching and batching. As for the MTU, the default is indeed 2K 
(2044) but its always to just know the facts, namely what was the mtu 
during the test.

> Regarding id/etc, this is what dmesg has:

if you have user space libraries installed, load ib_uverbs and run the 
command ibv_devinfo, you will see all the infiniband devices on your 
system and for each its device id and firmware version. If not, you 
should be looking on

/sys/class/infiniband/$device/hca_type
and
/sys/class/infiniband/$device/fw_ver

> Sep 16 22:49:26 elm3b39 kernel: eHCA Infiniband Device Driver (Rel.:
> SVNEHCA_0023)

> There are *fw* files for mthca0, but I don't see for ehca in /sys/class, so
> I am not sure (since these are pci-e cards, nothing shows up in lspci -v).
> What should I look for?

the above print seems to be from the ehca driver where you are talking 
on mthca0, which is quite confusing. If you want to be sure what hca is 
being used by the netdevice you are testing with (eg ib0) take a look on 
  the directory /sys/class/net/$netdevice/device/

If you have hca which is not reported in lspci and/or in 
/sys/class/infinidand it sounds like you have a problem or you found a bug.

Or.