[openib-general] OpenIB Architecture, kernel mode driver examples

[Grant Grundler]

On Fri, Feb 04, 2005 at 05:50:59PM -0800, Nauzad Sadry wrote:
> The architecture available in the openib.org website
> OpenIB-Arch-122104.pdf is just a set of PPTs & does not explain
> anything in detail. It just shows me the System Architecture in
> general with no use-cases or theory of operations.

correct - it's just an overview document.
I gather you are looking for the "Openib Driver Programing Guide".

Just curious: have you ever worked on an open source project?

> It does NOT contain any reference to the APIs exposed by OpenIB Access Layer.

The only other docs I'm aware of specific to openib.org are in SVN:
grundler at gsyprf3:/usr/src/openib_gen2/src/linux-kernel/docs$ ls
ipoib.txt  ipoib_faq.txt  sysfs.txt  user_mad.txt

But I don't think those will cover what you want either.
You really need to look at the header files (source code) in SVN:
grundler at gsyprf3:/usr/src/openib_gen2/src/linux-kernel/infiniband/include$ ls
ib_cache.h     ib_mad.h   ib_sa.h   ib_user_mad.h
ib_fmr_pool.h  ib_pack.h  ib_smi.h  ib_verbs.h

Are the API's you need defined in ib-verbs.h:struct ib_device?
(Hint: there is a very long list of function pointers)


> Do the documents specifed in the http://infiniband.sourceforge.net
> hold any relevance with openib ??

For "theory of operation", I would think yes. Infiniband implementations
can vary but ultimately they have to do the same basic things because
the HW design (and architecture) drives that.

> The code involved in SRP, SDP & IPoIB is too complicated &
> undocumented to try & understand the use-cases & APIs.

If you were talking about the original implementation a bunch
of linux kernel folks would agree with you.

The entire IPoIB driver is < 4k lines of code.
There is no SRP or SDP driver (yet).

Modern NIC drivers are substantially more.
tg3 is > 8K lines.  e1000 is over 10K lines.
acenic and e100 drivers are ~3000 lines of code as well but
are much simpler implementations (e.g. no TSO).

I don't find any of them particularly complex for basic operation.
The initialization sequences are usually the ugliest bits that
are hard to sort out.

> If someone has
> a sample kernel mode driver using the AccessLayer APIs, then I would
> really appreciate it.

IPoIB *is* the sample code.  If something isn't clear about the
IPoIB driver, folks on this list can probably answer your questions.

But you might do well to use google to search the mailing list
archives a bit before asking alot of questions.

hth,
grant