[Users] RDMA WRITE operation using kernel verbs.

[Yufei Ren]

On Mon, Dec 9, 2013 at 1:31 AM, jagadeesh kumar
<jagadeesh414.ece at gmail.com> wrote:
>
> On Sat, Dec 7, 2013 at 5:32 AM, Yufei Ren <yufei.ren at stonybrook.edu> wrote:
>>
>> On Fri, Dec 6, 2013 at 7:04 AM, jagadeesh kumar
>> <jagadeesh414.ece at gmail.com> wrote:
>> > Hi all,
>> >
>> >        I am learning RDMA_WRITE operation using OFED Verbs(Kernel
>> > space).
>> > As RDMA_WRITE operation will not generate completion entry on Responder
>> > side,
>> > I am following the technique like polling on the last byte of receive
>> > buffer.
>> >
>> > Steps i followed are:
>> > 1. kmalloc is used for memory allocation,
>> > 2. ib_reg_phys_mr is used for memory registration,
>> > 3. RDMA WRITE is requested on initiator side and the completion entry is
>> > success,
>> > 4. Responder is polling on last byte of receive buffer to be set,
>> > but Responder is not able to find the buffer updation and stuck in
>> > infinite
>> > loop.
>> >
>> > Please help me in solving the above mentioned problem. Let me know if
>> > something is missed.
>> >
>>
>> The `last byte' in your responder is cached in a CPU cache line and
>> your responder is checking this CPU cached `last byte'. Because
>> RDMA_WRITE is CPU bypass, the CPU in your responder is not aware of
>> the modified memory status, thus it still keeps checking the outdated
>> CPU cache line even though this `last byte' in memory is actually
>> already updated. Therefore, your responder is busy-waiting there.
>
>
> Thanks Yufei, for your reply.
>
>      When  i used last byte polling approach using uverbs it works fine,
> where as in using kernel verbs above mentioned problem is seen.
> Can you please share some information on, How user space is not facing the
> problem of CPU caching ?
> and how it is managed ?
>
> Thanks & Regards
> Jagadeesh.
>
Not clear about the detailed difference between your uverbs based
approach and kernel verbs one. It would be helpful if you could show
some sections of your code.

I have a question on your `last byte' approach. Will the `last byte'
of a memory block in an RDMA_WRITE operation must be transferred after
all the other bytes, and regard it as a completion event?

>>
>>
>> BTW, it seems your approach is wasting CPU cycles on checking the
>> `last byte'. Your might be interested in RDMA write with immediate
>> data.
>> https://oss.oracle.com/pipermail/rds-devel/2007-November/000208.html
>> quote:
>> "Generally, rdma operations do not provide a notification to the remote
>> host when they complete. It is possible to setup for notification -
>> using IB native "immediate" data (4 bytes) for example - but the data
>> size is limited."
>>
>>
>> Thanks
>> Yufei
>>
>>
>> >      The same approach i.e. polling on last byte of receive buffer works
>> > for
>> > me in
>> > user space(uverbs) RDMA_WRITE operation.
>> >
>> > My host machine is running Linux kernel 2.6.18.
>> >
>> > Any reference or web links will be useful.
>> >
>> > Thanks & Regards
>> > Jagadeesh.
>> >
>> >
>> > _______________________________________________
>> > Users mailing list
>> > Users at lists.openfabrics.org
>> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>> >
>
>