[libfabric-users] ibv_post_send() execution time

[Ilango, Arun]

Hi Cheng,

Perhaps you could try asking in the following mailing lists for verbs related questions:

linux-rdma at vger.kernel.org
users at lists.openfabrics.org.

Thanks,
Arun.

From: Libfabric-users [mailto:libfabric-users-bounces at lists.openfabrics.org] On Behalf Of Dave Goodell (dgoodell)
Sent: Saturday, September 03, 2016 7:18 AM
To: Cheng Wang <wangchenghku at gmail.com>
Cc: libfabric-users at lists.openfabrics.org
Subject: Re: [libfabric-users] ibv_post_send() execution time

Hi Cheng,

Your question is about libibverbs but this list is for libfabric. Perhaps someone else on this list knows the appropriate venue for your question.

-Dave

On Sep 2, 2016, at 11:57 PM, Cheng Wang <wangchenghku at gmail.com<mailto:wangchenghku at gmail.com>> wrote:
Hi,

I am new to OFI and I am doing some microbenchmarks.

I measured the execution time of ibv_post_send() in both rping(ping-pong) program and ib_write_lat program. Below is my pseudo code:

clock_gettime(); // start
ibv_post_send();
clock_gettime(); // end

Specifically, I measured IBV_WR_RDMA_WRITE operation using the same data size in Reliable Connection (RC) mode.

And the result shows each ibv_post_send() in rping takes around 170~180 nanoseconds to return, but in ib_write_lat program it only takes around 60 nanoseconds to return in average.
I suspected that this is caused by memalign of the sent buffer in ib_write_lat. But after I changed the rdma sent buffer in rping to be also memory aligned, it still takes around 170 nanoseconds.

May I know what caused this difference?

Many thank in advance.
_______________________________________________
Libfabric-users mailing list
Libfabric-users at lists.openfabrics.org<mailto:Libfabric-users at lists.openfabrics.org>
http://lists.openfabrics.org/mailman/listinfo/libfabric-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20160906/7abaf9f9/attachment.html>