[Users] RDMA WRITE target buffer updation.

[jagadeesh kumar]

Hi all,

I am writing RDMA_WRITE latency(Round Trip Latency) application using OFED
user verbs and kernel verbs.

In both applications i am using technique like polling on the last byte of
rdma target buffer on the responder side
(even though it is not standard),to find RDMA_WRITE completion.

Steps followed in user space rdma_write
1. allocated memory using malloc,
2. registered memory using ibv_reg_mr, REMOTE access is enabled,
3. requested rdma_write from initiator, completion entry is success,
4. responder is polling(spinning) on the last byte of receive buffer to be
set, responder is able to see last byte set.

Steps followed in kernel space rdma_write
1. allocated memory using kmalloc,
2. registered memory using ib_reg_phys_mr, REMOTE access is enabled,
3. requested rdma_write from initiator, completion entry is success,
4. responder is polling(spinning) on the last byte of receive buffer to be
set. Responder is not able to find buffer updatation.

If i use ib_dma_sync_single_for_cpu() before accessing the buffer on
Responder side, Responder is able to see updated data(last byte set).

Please help me to understand following,
Is this buffer updation problem is because of CPU cache, or some other.
If it is CPU cache problem, then how user space is able to get updated data
without any explicit dma synchronization.

Test setup is
arch: x86_64
Linux kernel :2.6.18-194.el5 (RHEL 5.5)
OFED : 1.5.4

Thanks in advance
Jagadeesh.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20140110/1217eb06/attachment.html>