[openib-general] amso userspace

[Steve Wise]

> Should I expect to be able to use the kernel directories in branches/iwarp
> directly with linux-2.6.17.1?  It looks like your branch may be out
> of date with respect to trunk for a few files.  I used it anyway and
> it does seem to build and run.

I haven't tried branches/iwarp with a 2.6.17 kernel.  It works fine with
2.6.16 though, and I expect it to work fine in 2.6.17.  The branch is a
snapshot of the main trunk and we only update it occasionally.

> 
> In the userspace source, amso_create_qp limits max_send_sge and
> max_recv_sge to 4.  Is this really the hardware limit?  It seems
> quite low.
> 

Yep.  That's a HW limit.  

> Should I expect the examples in
> branches/iwarp/src/userspace/libibverbs/examples to work?  I was
> hoping to use rc_pingpong.c as a way to understand what was going
> wrong with my code, but it exits when it finds that its local lid is
> zero (line 578).

Those examples only work for IB transports.  The examples in
librdma/examples will run over iwarp because they utilize the RDMA
CMA.  


> 
> One spot in my code where I'm trying to understand why libamso
> errors is this transition to RTS (not using rdmacm, just bringing
> it up by hand):
> 
>     /* transition qp to ready-to-send */
>     mask =
>        IBV_QP_STATE
>      | IBV_QP_SQ_PSN
>      | IBV_QP_MAX_QP_RD_ATOMIC
>      | IBV_QP_TIMEOUT
>      | IBV_QP_RETRY_CNT
>      | IBV_QP_RNR_RETRY;
>     memset(&attr, 0, sizeof(attr));
>     attr.qp_state = IBV_QPS_RTS;
>     attr.sq_psn = 0;
>     attr.max_rd_atomic = 1;
>     attr.timeout = 26;  /* 4.096us * 2^26 = 5 min */
>     attr.retry_cnt = 20;
>     attr.rnr_retry = 20;
>     ret = ibv_modify_qp(qp, &attr, mask);
>     if (ret)
>         error_xerrno(ret, "%s: ibv_modify_qp RTR -> RTS", __func__);
> 
> The return value is 11, EAGAIN.
> 
> With C2_DEBUG on, the kernel says to the console:
> 
>     c2: c2_qp_modify:145 qp=ffff81003e71f180, IB_QPS_RTR --> IB_QPS_RTS
>     c2: c2_qp_modify: c2_errno=-11
>     c2: c2_qp_modify:243 qp=ffff81003e71f180, cur_state=IB_QPS_RTR
> 
> I'm guessing one of those values must be off, but can't see where
> anything is enforced in the lib or kernel driver.  Some of these
> fields don't make sense for non-IB fabrics.  Just using a mask of
> IBV_QP_STATE caused the same return value.  Can you see the problem
> right off?  (This code does work fine on mthca.)
> 

You need to use librdmacm to setup iwarp connections. That's the only
way it will work for the amso device.  See librdma/examples.  I also
posted a patch to perftest/rdma_lat.c and rdma_bw.c that added a -c
option to utilize the RDMA CMA.   The patch didn't get pulled in,
however...


Steve.