[ofa-general] puzzled by ATOMIC operation

[zhouyonghao at ict.ac.cn]

Hi, all
    these days I'm puzzled by a weird mistake in atomic operation, when I
use FETCH_AND_ADD, the remote buffer only changed for 9 times at most,
aka if the initial value is 0, when I execute 10 times FETCH_AND_ADD
operations, the value is 9, and no more changes happens to the buffer.
    Should I take care of the big-edian from/to litter-edian translation?
I tried this, but it seems not the point.
    Is there some sample code for atomic operation just like "perftest" in
the libibverbs package? Maybe I missed something? I paste some code
below for reviewing, thanks in advance!

    struct ibv_send_wr wr;
    wr.opcode     = IBV_WR_ATOMIC_FETCH_AND_ADD;
    wr.wr.atomic.remote_addr = rem_dest->vaddr;
    wr.wr.atomic.rkey = rem_dest->rkey;
    wr.send_flags                         = IBV_SEND_SIGNALED |
IBV_SEND_FENCE;
    wr.wr.atomic.compare_add    = 1;
    wr.wr.atomic.swap                 = 0;
    wr.wr_id      = id;
    wr.sg_list    = &list;
    wr.num_sge    = 1;
    wr.next       = NULL;

the qp and memory region is set with atomic flag :
    mr = ibv_reg_mr(host->pd, buf, len,
                IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE
                | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC);

   attr.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE
        | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC;

some attibutes in my ib device about atomic operation are as below:
max_qp_rd_atom is 4, max_qp_init_rd_atom is 128, max_res_rd_atom is 258048