[openib-general] SDP Protocol Issue Was: Configuration of sdp
Eitan Zahavi
eitan at mellanox.co.il
Wed Nov 29 09:11:25 PST 2006
Sean Hubbell wrote:
> Ok, after futher inverstigation, here is what I have found:
>
> socket failed: Address family not supported by protocol
>
Do yo u have sdp loaded. Please try:
lsmod | grep ib_sdp
> I am assuming when sdp "exchanges" the AF_INET sockets to AF_SDP sockets
> that the current code that I am running has a issue or I don't know how
> to configure things...
>
> I have running libsdp v. 0.9.0 x86_64 as the rpm from downloaded from
> mirror.centos.org running kernel 2.6.9-42.0.3.plus.c4smp
>
>
I would recommend upgrading libsdp only to the one available from the
git tree at
git clone git://staging.openfabrics.org/git/~eitan/libsdp.git
or still fom SVN:
https://openib.org/svn/gen2/trunk/src/userspace/libsdp
> Also, my /etc/libsdp.conf file has the following:
>
> log min-level 1 destination syslog
> #use both server * *:*
> #use both client * *:*
> use sdp client * 10.10.0.0/16:*
> use sdp server * 10.10.0.0/16:*
>
> and seems to break all of my TCP connections, regardless if the
> interface is 10.10.* Is my configuration correct?
>
The "both" would have been better in preserving TCP based services as it
falls back to use TCP if SDP does not work.
> What would you recommend my next step to be? Upgrading my kernel as well
> as libsdp to the latest? Also, currently without using sdp I am getting
> 941 MBps, what do you get when you run using sdp, Karun mentioned that
> he is getting double the performance, which is where I am at; needing
> better performance from my apps?
>
Depends on your apps. If you have many connections of large messages SDP
is a clear win.
> Sean
>
> Eitan Zahavi wrote:
>
>> The error code is what the standard connect() returns.
>> Just "man connect"
>>
>> EZ
>>
>> Eitan Zahavi
>> Senior Engineering Director, Software Architect
>> Mellanox Technologies LTD
>> Tel:+972-4-9097208
>> Fax:+972-4-9593245
>> P.O. Box 586 Yokneam 20692 ISRAEL
>>
>>
>>
>>
>>> -----Original Message-----
>>> From: Sean Hubbell [mailto:shubbell at dbresearch.net]
>>> Sent: Wednesday, November 29, 2006 4:34 PM
>>> To: Eitan Zahavi
>>> Cc: Karun Sharma; openib-general at openib.org
>>> Subject: Re: [openib-general] Configuration of sdp
>>>
>>> This is what I got in /var/log/messages:
>>>
>>> Nov 29 08:32:14 neptune kernel: sdp_sock(32768:0): rdma_resolve_addr
>>> failed: -99
>>> Nov 29 08:32:14 neptune iperf: iperf[28484] libsdp Error connect:
>>>
>>>
>> failed with
>>
>>
>>> code <-1> for SDP fd:5
>>>
>>> Where can I find out about the error messages? Do I still need to go
>>>
>>>
>> through
>>
>>
>>> the code to find these?
>>>
>>> Sean
>>>
>>> Eitan Zahavi wrote:
>>>
>>>
>>>> Hi Sean
>>>>
>>>> Regarding libsdp.conf:
>>>>
>>>> The simplest config that would catch all applications/ports and will
>>>> NOT break your Eth based connections is:
>>>>
>>>> log min-level 7 destination syslog
>>>> use both server * *:*
>>>> use both client * *:*
>>>>
>>>> Not the level=7 on the "log" directive means that you will get a
>>>>
>>>>
>> line
>>
>>
>>>> in /var/log/messages for every listen/accept/connect with info that
>>>> should let you know if the connection was made through SDP or TCP.
>>>>
>>>> Sean Hubbell wrote:
>>>>
>>>>
>>>>> Yes, I tried that in our testing when I loaded the ib_sdp (I did
>>>>>
>>>>>
>> just
>>
>>
>>>>> use the defaults). I was wondering if the numbers that we are
>>>>>
>>>>>
>> seeing
>>
>>
>>>>> seem consistent with everyone elses... and at what point will the
>>>>>
>>>>>
>> sdp
>>
>>
>>>>> module increase my bandwidth when using a legacy / third party
>>>>> network library?
>>>>>
>>>>> Sean
>>>>>
>>>>> Karun Sharma wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Hi Sean:
>>>>>>
>>>>>> If you are using OFED 1.1 release, you will find a "README.txt"
>>>>>>
>>>>>>
>> file
>>
>>
>>>>>> (/usr/local/ofed/ directory). In this file, configuration steps of
>>>>>> SDP are mentioned.
>>>>>> 1. modprobe ib_sdp
>>>>>> 2. export LD_PRELOAD=/usr/local/ofed/lib/libsdp.so
>>>>>> 3. export LIBSDP_CONFIG_FILE=/usr/local/ofed/etc/libsdp.conf
>>>>>> <application name>.
>>>>>>
>>>>>> You may need to modify libsdp.conf before step 3.
>>>>>>
>>>>>> Thanks
>>>>>> Karun
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>> --------------------------------------------------------------------
>>
>>
>>>>>> ----
>>>>>>
>>>>>> *From:* openib-general-bounces at openib.org on behalf of Sean
>>>>>>
>>>>>>
>> Hubbell
>>
>>
>>>>>> *Sent:* Tue 11/28/2006 10:34 PM
>>>>>> *To:* openib-general at openib.org
>>>>>> *Subject:* [openib-general] Configuration of sdp
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have a question... we have tested our network on our cluster
>>>>>> using iperf v2.0.2.
>>>>>>
>>>>>> Our first test was using Gig-E network and the results were 941
>>>>>>
>>>>>>
>> Mbps.
>>
>>
>>>>>> (i.e. iperf -c <ipaddress> and iperf -s)
>>>>>>
>>>>>> Our second test was using Gig-E network and the results were 1.05
>>>>>> Mbps using UDP.
>>>>>> (i.e. iperf -c <ipaddress> -u and iperf -s -u)
>>>>>>
>>>>>> Our third test was using IBv4 network and the results was a max of
>>>>>> 3.8 Mbps using TCP.
>>>>>> (i.e. iperf -c <ipaddress> -P[1-16] and iperf -s)
>>>>>>
>>>>>> Loaded ib_sdp module and then tested:
>>>>>> Our fourth test was using IBv4 network and the results was a max
>>>>>>
>>>>>>
>> of
>>
>>
>>>>>> 4.16
>>>>>> Mbps using UDP.
>>>>>> (i.e. iperf -c <ipaddress> -b1000M -P[1-16] and iperf -s -u)
>>>>>>
>>>>>> UNLOADED ib_sdp module and then tested:
>>>>>> Our fourth test was using IBv4 network and the results was a max
>>>>>>
>>>>>>
>> of
>>
>>
>>>>>> 4.16
>>>>>> Mbps using UDP.
>>>>>> (i.e. iperf -c <ipaddress> -b1000M -P[1-16] and iperf -s -u)
>>>>>>
>>>>>> My question is what impact would it have on our network if
>>>>>> configured sdp and is there an example of how to configure sdp
>>>>>> somewhere on the wiki?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Sean
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> openib-general mailing list
>>>>>> openib-general at openib.org
>>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>>
>>>>>> To unsubscribe, please visit
>>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> openib-general mailing list
>>>>> openib-general at openib.org
>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>
>>>>> To unsubscribe, please visit
>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>>
>
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
More information about the general
mailing list