[Users] RDMA WRITE operation using kernel verbs.

[jagadeesh kumar]

On Sat, Dec 7, 2013 at 5:32 AM, Yufei Ren <yufei.ren at stonybrook.edu> wrote:

> On Fri, Dec 6, 2013 at 7:04 AM, jagadeesh kumar
> <jagadeesh414.ece at gmail.com> wrote:
> > Hi all,
> >
> >        I am learning RDMA_WRITE operation using OFED Verbs(Kernel space).
> > As RDMA_WRITE operation will not generate completion entry on Responder
> > side,
> > I am following the technique like polling on the last byte of receive
> > buffer.
> >
> > Steps i followed are:
> > 1. kmalloc is used for memory allocation,
> > 2. ib_reg_phys_mr is used for memory registration,
> > 3. RDMA WRITE is requested on initiator side and the completion entry is
> > success,
> > 4. Responder is polling on last byte of receive buffer to be set,
> > but Responder is not able to find the buffer updation and stuck in
> infinite
> > loop.
> >
> > Please help me in solving the above mentioned problem. Let me know if
> > something is missed.
> >
>
> The `last byte' in your responder is cached in a CPU cache line and
> your responder is checking this CPU cached `last byte'. Because
> RDMA_WRITE is CPU bypass, the CPU in your responder is not aware of
> the modified memory status, thus it still keeps checking the outdated
> CPU cache line even though this `last byte' in memory is actually
> already updated. Therefore, your responder is busy-waiting there.
>

Thanks Yufei, for your reply.

     When  i used last byte polling approach using uverbs it works fine,
where as in using kernel verbs above mentioned problem is seen.
Can you please share some information on, How user space is not facing the
problem of CPU caching ?
and how it is managed ?

Thanks & Regards
Jagadeesh.


>
> BTW, it seems your approach is wasting CPU cycles on checking the
> `last byte'. Your might be interested in RDMA write with immediate
> data.
> https://oss.oracle.com/pipermail/rds-devel/2007-November/000208.html
> quote:
> "Generally, rdma operations do not provide a notification to the remote
> host when they complete. It is possible to setup for notification -
> using IB native "immediate" data (4 bytes) for example - but the data
> size is limited."
>
>
> Thanks
> Yufei
>
>
> >      The same approach i.e. polling on last byte of receive buffer works
> for
> > me in
> > user space(uverbs) RDMA_WRITE operation.
> >
> > My host machine is running Linux kernel 2.6.18.
> >
> > Any reference or web links will be useful.
> >
> > Thanks & Regards
> > Jagadeesh.
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users at lists.openfabrics.org
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20131209/31092266/attachment.html>