[ewg] GPU direct support?

[Elken, Tom]

> Is there support in the standard OFED for GPU direct? Or is it only supported by
> the MLNX_OFED package Mellanox provides?

[Tom] 
It may depend on whether you are using a fairly recent Linux distro and CUDA 4.0 or newer.  In that case you may not need GPUDirect software from Mellanox to get GPUDirect v1 levels of performance.

I realize that you probably have Mellanox HCAs, but this answer for QLogic, now Intel, IB HCAs may also apply to Mellanox:

     The following applies to the QLogic OFED+ 7.1 (most recent) distribution :
If you are running on any Linux distribution supported by this QLogic
InfiniBand software release*, then you must use NVIDIA CUDA 4.0 or newer to
have a supported solution from NVIDIA.  To get optimal performance with the
qib InfiniBand driver in this software release in applications which use CUDA 4.x,
all you need to do is to set the environment variable:
   CUDA_NIC_INTEROP=1
and to propagate that variable to the compute nodes running the application.

The preceding instructions should also work with the qib driver in the recent OFED 1.5.4.1 release -- and probably for the Mellanox driver too.

If you need to use CUDA 3.x and/or older Linux releases than say RHEL 5.6 or SLES 11 SP1, then you probably need the closed-source GPUDirect software from Mellanox.   QLogic had a different way to provide the GPUDirect capability on these older CUDA/Linux releases.

-Tom Elken
Intel InfiniBand (formerly QLogic)
*  RHEL 5.6, 5.7 or  RHEL 6.0 or newer
    SLES 11 SP1 or SLES 10 SP4

> 
> I searched the mailing list and found some talk about GPU direct here:
> http://lists.openfabrics.org/pipermail/ewg/2011-February/016202.html
> but it indicated it might be a closed source thing from Mellanox only.
> 
> Are there any other sources of information about this that I should read?
> 
> Thanks,
> Rick
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg