[openib-general] max_send_sge < max_sge

[Pete Wyckoff]

Using stock 2.6.17.1, with verbs 1.0.3-1.fc4 and mthca 1.0.2-1.fc4
with MT25204, this line:

    ret = ibv_query_device(ctx, &hca_cap);

tells me that hca_cap.max_sge = 30.

However, this code fails, with the last kernel write returning EINVAL:

    memset(&att, 0, sizeof(att));
    att.send_cq = 1024;
    att.recv_cq = 1024;
    att.cap.max_recv_wr = 512;
    att.cap.max_send_wr = 512;
    att.cap.max_recv_sge = 30;
    att.cap.max_send_sge = 30;
    att.qp_type = IBV_QPT_RC; 
    qp = ibv_create_qp(pd, &att);

But if I set:

    att.cap.max_recv_sge = 30;
    att.cap.max_send_sge = 29;  /* hca_cap.max_sge - 1 */

the QP create succeeds.

Is this a known issue?  Should I always subtract 1 from the reported
max on the send side?  Just for this hardware?

		-- Pete