[ofa-general] RE: [PATCH] core/mthca: Distinguish multiple IB cards in /proc/interrupts

[Roland Dreier]

 > > Not sure what you mean.  If we put msi-x info under /sys, then you can
 > > figure out which interrupts belong to a given HCA by following the
 > > device link from /sys/class/infiniband.  Similarly if /proc/interrupts
 > > gives the PCI device, then you have the same ability.  So either way
 > > works as far as I can tell.
 > Linux is supposed to move away from procfs to sysfs for this type of device
 > related info. However, /proc/interrupts is still present in the latest
 > distro
 > releases (for example, SLES11) and OFED needs to provide support
 > for this in procfs until the /proc/interrupts support is removed from
 > kernel.

I think we're talking past each other.  I agree that /proc/interrupts is
still needed.  However, there are two things I see that we can add, and
each one suffices to make everything unambiguous:

1) Add PCI device info ("mlx4-comp-1 at pci...") to the interrupt name.
   Then if userspace cares about the interrupts for device "foo", it can
   look at the /sys/class/infiniband/foo/device symlink to find the PCI
   device, and then look in /proc/interrupts for all interrupts related
   to that PCI device.

*OR*

2) Add /sys/devices/pci.../msix/vectorN files (or something like that)
   so userspace can similarly follow the
   /sys/class/infiniband/foo/device symlink to the PCI directory and
   read the MSI-X vector numbers for the device, and then get all info
   for that interrupt from /proc/interrupts, /proc/irq/NNN/smp_affinity,
   etc.

Either option by itself is completely sufficient.

 > I have not seen full sysfs support for Ethernet devices .
 > I have seen IRQ number info but no interrupt counters on a per CPU basis.
 > Do we know when the full support for ethernet devices will be available
 > in sysfs? We can enhance OFED at the same time ethernet support is made
 > available in the kernel.

Umm... for ethernet you can get per-CPU counters from /proc/interrupts,
if you know the IRQ number.  But if you have multiple MSI-X interrupt
then you have to get the IRQ number some other way.

 > 3) We can add dev_alloc_name() functionality to mlx4_core similar to
 > alloc_name()
 > present in ib_core. This is consistent with other ethernet device driver
 > implementations
 > using the function dev_alloc_name() present in the kernel. (Please see
 > .../net/core/dev.c)

Not sure how this could work.  If mlx4_core is allocating device
numbers, and I have 3 adapters, only 2 of which are IB HCAs and 1 of
which is an ethernet adapter, then how mlx4_core assign numbers that
match what the RDMA layer will use?

 - R.