[ofa-general] puzzled by ATOMIC operation

[Dotan Barak]

zhouyonghao at ict.ac.cn wrote:
> Hi, all
>     these days I'm puzzled by a weird mistake in atomic operation, when I
> use FETCH_AND_ADD, the remote buffer only changed for 9 times at most,
> aka if the initial value is 0, when I execute 10 times FETCH_AND_ADD
> operations, the value is 9, and no more changes happens to the buffer.
>     Should I take care of the big-edian from/to litter-edian translation?
>   
No, you should write the value in local host endian.
> I tried this, but it seems not the point.
>     Is there some sample code for atomic operation just like "perftest" in
> the libibverbs package? Maybe I missed something? I paste some code
> below for reviewing, thanks in advance!
>   
You didn't miss it, not many people use the atomic operations and there
isn't any example that uses it.

>     struct ibv_send_wr wr;
>   
It will be wise to fill the following delaration here:
memset(&wr, 0, sizeof(wr));
>     wr.opcode     = IBV_WR_ATOMIC_FETCH_AND_ADD;
>     wr.wr.atomic.remote_addr = rem_dest->vaddr;
>     wr.wr.atomic.rkey = rem_dest->rkey;
>     wr.send_flags                         = IBV_SEND_SIGNALED |
> IBV_SEND_FENCE;
>     wr.wr.atomic.compare_add    = 1;
>     wr.wr.atomic.swap                 = 0;
>   
since you are using Fetch&Add the swap attribute is useless.
>     wr.wr_id      = id;
>     wr.sg_list    = &list;
>     wr.num_sge    = 1;
>     wr.next       = NULL;
>   
What about the value of the s/g entry?
> the qp and memory region is set with atomic flag :
>     mr = ibv_reg_mr(host->pd, buf, len,
>                 IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE
>                 | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC);
>
>    attr.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE
>         | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC;
>   
At least the QP/MR in the remote side should enable REMOTE_ATOMIC (if
you want to enable it locally, it is fine
but not really needed).
> some attibutes in my ib device about atomic operation are as below:
> max_qp_rd_atom is 4, max_qp_init_rd_atom is 128, max_res_rd_atom is 258048
>   
max_qp_rd_atom: The maximum number of outstanding atomic/RDMA read that
a QP can handle as target.
max_qp_init_rd_atom: The maximum number of outstanding atomic/RDMA read
that a QP can handle as initiator.

Obviously, you should use the minimum of the values (use the right value
in each side, since different HCAs can have
different attributes)

I think that max_res_rd_atom is the total outstanding number of
atomic/RDMA read that the HCA can handle,
but i'm not really sure.


If you think that you have a bug, please review the code in remote side
(maybe you have problem to read the data...).

I hope i helped you a little ...

Dotan