[ofa-general] Receiving Unknown packets at regular interval
Sumit Gaur
Sumit.Gaur at Sun.COM
Mon May 19 09:08:56 PDT 2008
Hal Rosenstock wrote:
> On Mon, 2008-05-19 at 19:19 +0530, Sumit Gaur - Sun Microsystem wrote:
>
>> Hal Rosenstock wrote:
>>
>>> Hi Sumit,
>>>
>>> On Mon, 2008-05-19 at 17:20 +0530, Sumit Gaur - Sun Microsystem wrote:
>>>
>>>
>>>> Hi Hal,
>>>>
>>>>
>>>> Hal Rosenstock wrote:
>>>>
>>>>
>>>>> Sumit,
>>>>>
>>>>> On Mon, 2008-05-19 at 15:25 +0530, Sumit Gaur - Sun Microsystem wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Hi
>>>>>> I have an issue while my program interacting with OFED umad library.
>>>>>>
>>>>> Are you referring to libibumad ?
>>>>>
>>>> yes, I am using mad_receive(0, -1) function to get my response back.
>>>>
>>> OK.
>>>
>>>
>>>
>>>>>> I have two
>>>>>> separate threads one for sending SMP,GMP packets and another to receive
>>>>>> response. Things are working fine but during the whole process I keep receiving
>>>>>> packets with unknown tid apart from correct response.
>>>>>>
>>>>> What's the exact message ?
>>>>>
>>>> Response comes as proper mad packets but with "tid" that I have never send and
>>>> my logic to keep track of send/response pkts failed.
>>>>
>>>>
>>>>>> Is it a correct behavior.
>>>>>>
>>>>> It could be; there's not enough info as to what is going on. It could be
>>>>> some unsolicited message (e.g. from SM) comes in during your
>>>>> transactions. Can you see what MADs are incoming ? One way to do that
>>>>> would be to run madeye.
>>>>>
>>>> Yes I could see complete mad with madhdr as following fields
>>>>
>>>> Response TID2 = 0x000000006701869b , BaseVersion = 1, MgmtClass=129,
>>>> ClassVersion=1, R_Method=129, ClassSpecific=1, Status=128, AttributeID=435
>>>>
>>> Class 129 is a Subn directed route packet. Some of the other info (like
>>> attribute ID) doesn't look right to me but maybe that's something
>>> "special" to your environment.
>>>
>> Sorry missed last number AttributeID=4352
>>
>
> I don't know what that attribute ID is so there's something different
> about that.
>
> Out of curiousity, what SM are you using ?
>
>
>>>> If these are unsolicited packets. Is there anyway to filter them.
>>>>
>>> Yes. How do you register ?
>>>
>> For registration I am calling madrpc_init(ca, ca_port, mgmt_classes, 4)
>> function once before starting polling thread for SMI and GSI packet receive.
>> Once I received packet I am filtering them on the basis of madhdr->MgmtClass.
>>
>> int mgmt_classes[4] = {IB_SMI_CLASS,IB_SMI_DIRECT_CLASS, IB_SA_CLASS,
>> IB_PERFORMANCE_CLASS};
>>
>> for given ca and ca port of local node.
>>
>
> That looks like it would register with a NULL method mask which should
> filter unsolicited packets.
>
> I think I see the issue: the incoming packet appears to have a method of
> 129 (GetResp) which has the response bit on so it's not considered
> unsolicited. You need to see what exactly that packet is and where it's
> coming from and why.
>
> -- Hal
>
>
Hi Hal,
It is true that packets received are looks like proper response but as I
mentioned before they content TID that I have never send to OFED and
this cause the problem. Why OFED is sending these extra packets Is the
matter to investigate.
sumit
sumit
>>>
>>>> Any reference to madeye ?
>>>>
>>> There's only the code for this (kernel module) which is added by OFED
>>> (not upstream) in drivers/infiniband/util but it's pretty
>>> straightforward to use.
>>>
>>> -- Hal
>>>
>>>
>>>
>>>>>> If yes how I could avoid them ?
>>>>>>
>>>>> Not sure what you are seeing yet.
>>>>>
>>>>> -- Hal
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Thanks and Regards
>>>>>> sumit
>>>>>>
>>>>>> general-request at lists.openfabrics.org wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Send general mailing list submissions to
>>>>>>> general at lists.openfabrics.org
>>>>>>>
>>>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>>> or, via email, send a message with subject or body 'help' to
>>>>>>> general-request at lists.openfabrics.org
>>>>>>>
>>>>>>> You can reach the person managing the list at
>>>>>>> general-owner at lists.openfabrics.org
>>>>>>>
>>>>>>> When replying, please edit your Subject line so it is more specific
>>>>>>> than "Re: Contents of general digest..."
>>>>>>>
>>>>>>>
>>>>>>> Today's Topics:
>>>>>>>
>>>>>>> 1. Re: [PATCH] IB/core: handle race between elements in qork
>>>>>>> queues after event (Roland Dreier)
>>>>>>> 2. Re: RDS flow control (Steve Wise)
>>>>>>> 3. Re: RDS flow control (Olaf Kirch)
>>>>>>> 4. Re: RDS flow control (Steve Wise)
>>>>>>> 5. Re: RDS flow control (Olaf Kirch)
>>>>>>> 6. Re: [PATCH 3/3] IB/ipath - fix RDMA read response sequence
>>>>>>> checking (Roland Dreier)
>>>>>>> 7. Re: [PATCH][INFINIBAND]: Make ipath_portdata work with
>>>>>>> struct pid * not pid_t. (Roland Dreier)
>>>>>>> 8. Re: bitops take an unsigned long * (Roland Dreier)
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------------------
>>>>>>>
>>>>>>> Message: 1
>>>>>>> Date: Tue, 13 May 2008 10:41:39 -0700
>>>>>>> From: Roland Dreier <rdreier at cisco.com>
>>>>>>> Subject: Re: [ofa-general] [PATCH] IB/core: handle race between
>>>>>>> elements in qork queues after event
>>>>>>> To: Moni Shoua <monis at Voltaire.COM>
>>>>>>> Cc: Olga Stern <olgas at voltaire.com>, OpenFabrics General
>>>>>>> <general at lists.openfabrics.org>
>>>>>>> Message-ID: <adatzh2ksoc.fsf at cisco.com>
>>>>>>> Content-Type: text/plain; charset=us-ascii
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Can we please go on with this patch? We would like to see it in the next kernel.
>>>>>>>>
>>>>>>> I still don't get why this is important to you. Is there a concrete
>>>>>>> example of a situation where this actually makes a measurable difference?
>>>>>>>
>>>>>>> We need some justification for adding this locking complexity beyond "it
>>>>>>> doesn't hurt." (And also of course we need it fixed so there aren't races)
>>>>>>>
>>>>>>> - R.
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------
>>>>>>>
>>>>>>> Message: 2
>>>>>>> Date: Tue, 13 May 2008 12:58:11 -0500
>>>>>>> From: Steve Wise <swise at opengridcomputing.com>
>>>>>>> Subject: Re: [ofa-general] RDS flow control
>>>>>>> To: Richard Frank <richard.frank at oracle.com>
>>>>>>> Cc: rds-devel at oss.oracle.com, general at lists.openfabrics.org
>>>>>>> Message-ID: <4829D6B3.5080900 at opengridcomputing.com>
>>>>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>>>>
>>>>>>> Richard Frank wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Steve Wise wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Olaf Kirch wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Monday 12 May 2008 18:57:38 Jon Mason wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> As part of my effort to get RDS working for iWARP, I will be
>>>>>>>>>>> working on the RDS flow control. Flow control is needed for iWARP
>>>>>>>>>>> due to the fact that iWARP connections terminate if there is no
>>>>>>>>>>> posted recv for an incoming packet. IB connections do not have
>>>>>>>>>>> this limitation if setup in a certain way. In its current
>>>>>>>>>>> implementation, RDS sets the connection attribute rnr_retry to 7.
>>>>>>>>>>> This causes IB to retransmit until there is a posted recv buffer.
>>>>>>>>>>>
>>>>>>>>>> I think for the initial implementation, it is fine for iWARP to just
>>>>>>>>>> fail the connect when that happens, and re-establish the connection.
>>>>>>>>>>
>>>>>>>>>> If you use reasonable defaults for the send and recv queues, receiver
>>>>>>>>>> overruns should be relatively rare.
>>>>>>>>>>
>>>>>>>>>> Once everything else works, let's revisit the flow control part.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I _think_ you'll hit this quickly with one-way flows. Send
>>>>>>>>> completions for iWARP only mean the user's buffer can be reused. Not
>>>>>>>>> that its placed at the remote peer or in the remote user's buffer.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Let's see what happens - anyway - this could be solved in an IWARP
>>>>>>>> extension to RDS - right ?
>>>>>>>>
>>>>>>>
>>>>>>> Yes, by adding flow control. And it could be iwarp-specific if you
>>>>>>> want. I would not suggest relying on connection termination and
>>>>>>> re-establishment as the way to handle this :).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> But perhaps I'm wrong. Jon, maybe you should try to hit this with IB
>>>>>>>>> and rnr_retry == 0 using the rds perf tools?
>>>>>>>>> Also "the everything else" part depends on remove fmr usage. I'm
>>>>>>>>> working on the new RDMA memory verbs allowing fast registration of
>>>>>>>>> physical memory via a send WR. To support iWARP we need to remove
>>>>>>>>> the fmr usage from RDS. The idea was to replace fmrs with the new
>>>>>>>>> fastreg verbs. Thoughts?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> What does "fast" imply here - how does this compare to the performance
>>>>>>>> of FMRs ?
>>>>>>>>
>>>>>>>
>>>>>>> Don't know yet, but probably as fast.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Why would not push memory window creation into the RDS transport
>>>>>>>> specific implementations ?
>>>>>>>>
>>>>>>> Isn't it already transport-specific? IE you don't need FMRs for TCP.
>>>>>>> (I'm ignorant on the specifics of the implementation at this point, so
>>>>>>> please excuse any dumb statements :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Changing the API may be OK - if we retain the performance we have with
>>>>>>>> IB.
>>>>>>>>
>>>>>>>
>>>>>>> I assume nothing would fly that regresses IB performance. Worst case,
>>>>>>> you have an iwarp-specific RDS transport like you do for TCP, I guess.
>>>>>>> Hopefully though, IB + iWARP will be a common transport.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> Stay tuned for the new verbs API RFC...
>>>>>>>>>
>>>>>>>>> Steve.
>>>>>>>>> _______________________________________________
>>>>>>>>> general mailing list
>>>>>>>>> general at lists.openfabrics.org
>>>>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>>>>>
>>>>>>>>> To unsubscribe, please visit
>>>>>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------
>>>>>>>
>>>>>>> Message: 3
>>>>>>> Date: Tue, 13 May 2008 20:04:00 +0200
>>>>>>> From: Olaf Kirch <okir at lst.de>
>>>>>>> Subject: Re: [ofa-general] RDS flow control
>>>>>>> To: Steve Wise <swise at opengridcomputing.com>
>>>>>>> Cc: rds-devel at oss.oracle.com, general at lists.openfabrics.org
>>>>>>> Message-ID: <200805132004.01371.okir at lst.de>
>>>>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>>>>
>>>>>>> On Tuesday 13 May 2008 19:58:11 Steve Wise wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Yes, by adding flow control. And it could be iwarp-specific if you
>>>>>>>> want. I would not suggest relying on connection termination and
>>>>>>>> re-establishment as the way to handle this :).
>>>>>>>>
>>>>>>> No, not in the long term. But let's hold off on the flow control stuff
>>>>>>> for a little - I would first like to finish my patch set and hand it
>>>>>>> out for you folks to bang on it, rather than the other way round.
>>>>>>> Okay with you guys?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I assume nothing would fly that regresses IB performance. Worst case,
>>>>>>>> you have an iwarp-specific RDS transport like you do for TCP, I guess.
>>>>>>>> Hopefully though, IB + iWARP will be a common transport.
>>>>>>>>
>>>>>>> If it turns out that way, fine. If iWARP ands up sharing 80% of the
>>>>>>> code with IB except the RDMA specific functions, I think that's
>>>>>>> very much acceptable, too.
>>>>>>>
>>>>>>> Olaf
>>>>>>>
>>>>>> _______________________________________________
>>>>>> general mailing list
>>>>>> general at lists.openfabrics.org
>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>>
>>>>>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>>>>>>
>>>>>
>
>
More information about the general
mailing list