[openib-general] Re: [mvapich-discuss] RE: [openfabrics-ewg] Current OFED kernel snapshot

[Vladimir Sokolovsky]

Sayantan Sur wrote:
> Hi,
>
> * On May,10 Roland Dreier<rdreier at cisco.com> wrote :
>   
>>     Abhinav> By referring to the PPC64 architecture, i was mentioning
>>     Abhinav> about the IBM HCAs(4x/12x) running on GX/GX+ Bus. To the
>>     Abhinav> best of my knowledge, these HCAs do not support the
>>     Abhinav> features mentioned above.
>>
>> Hmm, making this a compile-time thing seems like a problem then.  Some
>> ppc64 systems have IBM eHCAs and some have Mellanox and/or PathScale
>> HCAs.  Shouldn't the same MPI package work on all of these systems?
>>     
>
> We've just incorporated an autodetection utility as a part of our build
> script (make.mvapich.gen2). It essentially reads the type of HCA from
> the standard location /sys/class/infiniband/<hca>/hca_type. Using this
> script, we have decoupled the choice of architecture and the InfiniBand
> card in the system. We believe using this script, MVAPICH will be able
> to work on PPC64 systems connected with Mellanox/Pathscale cards.
>
> On our installation of the IBM PPC64 systems, there is no `hca_type'
> file present. If that file is there, then the above mentioned
> "decoupling" thing can be done easily by trivially modifying our script.
> Does anyone have a PPC64 installation which has that file?
>
> With OpenIB/Gen2 standardized method of exporting HCA types, we will
> soon have runtime detection and optimization.
>
> Thanks,
> Sayantan.
>
>   

Sayantan,
1. What if HCA driver is not loaded, it can happen when you are going to install OFED for the first time and HCA driver is not present?
I think, you should rely on "lspci" instead of /sys/... .

2. There is an option to build RPMs using OFED build.sh script on machine without HCAs. 
So, in this case HCA type should be passed as a parameter.

Regards,
Vladimir