[ofa-general][PATCH] mlx4_ib: Multi Protocol support

[Yevgeny Petrilin]

The mlx4_core driver doesn't allow the configuration you described.
The mlx4_ib module can always assume that if it has only one IB port,
It would always be port number 1. 


Yevgeny Petrilin
Mellanox Technologies
phone: +972-4-9097200 (ext. 7677)
cell:     +972-54-7839222
mailto: yevgenyp at mellanox.co.il

-----Original Message-----
From: Roland Dreier [mailto:rdreier at cisco.com] 
Sent: Wednesday, April 16, 2008 6:43 PM
To: Yevgeny Petrilin
Cc: general at lists.openfabrics.org
Subject: Re: [ofa-general][PATCH] mlx4_ib: Multi Protocol support

 > Main changes to mlx4_ib:
 >         1.  Mlx4_ib driver queries the low level driver for number of
IB ports.
 >         2.  Qps are being reserved prior to being allocated.
 >         3.  Cq allocation API change.

As I said before, these mlx4_ib changes should be rolled into the
mlx4_core patches that change these interfaces.

Also, I don't understand exactly how you're handling which ports are IB
and which aren't.  Have you tested this code in the case where port 1 is
non-IB and port 2 is IB?  It seems that you have a bitmap of which ports
are IB:

 > +	foreach_port(i, ibdev->ports_map)
 > +		ibdev->num_ports++;

(By the way, foreach_port() is too generic a name to expose, since it
could easily collide with some general API -- I would use
mlx4_foreach_port() instead)

But then you do stuff like:

 > -	for (p = 1; p <= dev->caps.num_ports; ++p)
 > +	for (p = 1; p <= ibdev->num_ports; ++p)
 >  		mlx4_CLOSE_PORT(dev, p);

which doesn't seem to work if you only have one IB port but it isn't
port 1.

I think there are two sane ways to handle non-IB ports in mlx4_ib:

 - Have mlx4_ib report the number of IB ports as phys_port_cnt and have
   an indirection table that maps from IB port # to physical HCA port #
   (to handle the case where only port 2 is IB, so you need to map IB
   port 1 to HCA physical port 2).  This leads to some confusion with
   the real-world labels on ports I guess, and also I guess you need
   some SMA trickery to report the right port # to the SM.

 - Report the number of physical HCA ports as phys_port_cnt and just
   have non-IB ports always say they're DOWN.  This makes changing
   config on the fly easier, since a port going from DOWN to INIT is a
   pretty normal thing.  I guess there is a little bit of hackery
   involved in handling requests to mlx4_ib that involve non-IB ports.

However your changes seem to take a third way and I don't understand how
it can work.  Perhaps you can clarify?

 - R.