[openib-general] amso userspace

[Pete Wyckoff]

Having seen your four patchsets recently, I thought I'd give amso in
openib another shot, r8187 is what I'm looking at now.  Here's a few
questions for you.  (I did ccflash2 to the fw in ogc kit 20060308
already, and use the boot_image in there too.)

Should I expect to be able to use the kernel directories in branches/iwarp
directly with linux-2.6.17.1?  It looks like your branch may be out
of date with respect to trunk for a few files.  I used it anyway and
it does seem to build and run.

In the userspace source, amso_create_qp limits max_send_sge and
max_recv_sge to 4.  Is this really the hardware limit?  It seems
quite low.

Should I expect the examples in
branches/iwarp/src/userspace/libibverbs/examples to work?  I was
hoping to use rc_pingpong.c as a way to understand what was going
wrong with my code, but it exits when it finds that its local lid is
zero (line 578).

One spot in my code where I'm trying to understand why libamso
errors is this transition to RTS (not using rdmacm, just bringing
it up by hand):

    /* transition qp to ready-to-send */
    mask =
       IBV_QP_STATE
     | IBV_QP_SQ_PSN
     | IBV_QP_MAX_QP_RD_ATOMIC
     | IBV_QP_TIMEOUT
     | IBV_QP_RETRY_CNT
     | IBV_QP_RNR_RETRY;
    memset(&attr, 0, sizeof(attr));
    attr.qp_state = IBV_QPS_RTS;
    attr.sq_psn = 0;
    attr.max_rd_atomic = 1;
    attr.timeout = 26;  /* 4.096us * 2^26 = 5 min */
    attr.retry_cnt = 20;
    attr.rnr_retry = 20;
    ret = ibv_modify_qp(qp, &attr, mask);
    if (ret)
        error_xerrno(ret, "%s: ibv_modify_qp RTR -> RTS", __func__);

The return value is 11, EAGAIN.

With C2_DEBUG on, the kernel says to the console:

    c2: c2_qp_modify:145 qp=ffff81003e71f180, IB_QPS_RTR --> IB_QPS_RTS
    c2: c2_qp_modify: c2_errno=-11
    c2: c2_qp_modify:243 qp=ffff81003e71f180, cur_state=IB_QPS_RTR

I'm guessing one of those values must be off, but can't see where
anything is enforced in the lib or kernel driver.  Some of these
fields don't make sense for non-IB fabrics.  Just using a mask of
IBV_QP_STATE caused the same return value.  Can you see the problem
right off?  (This code does work fine on mthca.)

Thanks,

		-- Pete