[Users] RDMA WRITE operation using kernel verbs.

[Yufei Ren]

On Fri, Dec 6, 2013 at 7:04 AM, jagadeesh kumar
<jagadeesh414.ece at gmail.com> wrote:
> Hi all,
>
>        I am learning RDMA_WRITE operation using OFED Verbs(Kernel space).
> As RDMA_WRITE operation will not generate completion entry on Responder
> side,
> I am following the technique like polling on the last byte of receive
> buffer.
>
> Steps i followed are:
> 1. kmalloc is used for memory allocation,
> 2. ib_reg_phys_mr is used for memory registration,
> 3. RDMA WRITE is requested on initiator side and the completion entry is
> success,
> 4. Responder is polling on last byte of receive buffer to be set,
> but Responder is not able to find the buffer updation and stuck in infinite
> loop.
>
> Please help me in solving the above mentioned problem. Let me know if
> something is missed.
>

The `last byte' in your responder is cached in a CPU cache line and
your responder is checking this CPU cached `last byte'. Because
RDMA_WRITE is CPU bypass, the CPU in your responder is not aware of
the modified memory status, thus it still keeps checking the outdated
CPU cache line even though this `last byte' in memory is actually
already updated. Therefore, your responder is busy-waiting there.

BTW, it seems your approach is wasting CPU cycles on checking the
`last byte'. Your might be interested in RDMA write with immediate
data.
https://oss.oracle.com/pipermail/rds-devel/2007-November/000208.html
quote:
"Generally, rdma operations do not provide a notification to the remote
host when they complete. It is possible to setup for notification -
using IB native "immediate" data (4 bytes) for example - but the data
size is limited."


Thanks
Yufei


>      The same approach i.e. polling on last byte of receive buffer works for
> me in
> user space(uverbs) RDMA_WRITE operation.
>
> My host machine is running Linux kernel 2.6.18.
>
> Any reference or web links will be useful.
>
> Thanks & Regards
> Jagadeesh.
>
>
> _______________________________________________
> Users mailing list
> Users at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>