[ofa-general] Infiniband back-to-back without OpenSM?

[Talpey, Thomas]

At 08:39 AM 5/28/2008, Hal Rosenstock wrote:
>Tom,
>
>On Wed, 2008-05-28 at 08:06 -0400, Talpey, Thomas wrote:
>> Is it possible to manually configure two Infiniband ports to operate
>> with one another in back-to-back mode, without running OpenSM
>> on one of them?
>
>This is possible but something would need to do at least some subset of
>what the SM does depending on the precise requirements and the limits
>placed on the environment supported without a "full blown" SM.

Okay ... but IMO the only thing we need is a LID. Or at least, in my experience
all I've needed is a LID.

In a previous effort, we simply stole the low octet of an IP address, so we'd
"ifconfig ib0 1.2.3.X" and it would jam lid=X into the interface. Worked great.
If necessary, we would set a manual arp entry (using iproute) to avoid having
to broadcast.

>
>> We have done this on other IB implementations by manually assigning
>> LIDs, but I discover that the "lid" entry below 
>/sys/class/infiniband/<device>
>> is not writable, at least for mthca.
>
>This can be done via MADs so user_mad kernel module would be needed to
>do this.

Okay, all kernel modules can be assumed to be in place. How do we tell it
to manage the LID, with a shell command?

>
>> Also, I expect that the ipoib driver will
>> be unable to join the broadcast group, so will be unwilling to come up fully.
>
>Is IPoIB a requirement ?

I think so, for two reasons. One, principle of least surprise - the user will
expect to be able to ping, telnet etc if it has connectivity. Two, for NFS/RDMA
we require TCP and UDP connections in order to perform the mount and do
locking and recovery. We could do those over a parallel ethernet connection,
but that's kind of not the point.

>
>> With ethernet, and maybe iWARP, just a simple ifconfig can do this. So why
>> not IB?
>
>The simple answer is that it is the nature of IB management (being
>different than ethernet).

Which, IMO, we need to boil down to simplest-possible, for at least some
workable configuration.

Thanks for the ideas!

Tom.

>
>-- Hal
>
>> If you're wondering, my goal is give NFS/RDMA users a way to avoid having
>> to install the many userspace modules needed to do this, including 
>libibverbs,
>> opensm, etc. There's a lot to get wrong, and things go missing. Seeking an
>> "easy" way to get started with just the kernel and some shell commands.
>> 
>> Tom.
>> 
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>> 
>> To unsubscribe, please visit 
>http://openib.org/mailman/listinfo/openib-general