[ofiwg] GASNet with OFI on PSM provider

Sayan Ghosh zsayanz at gmail.com
Tue Jun 19 22:44:41 PDT 2018 

I am trying to install GASNet 1.30.0 using OFI on an Intel TrueScale
network (PSM provider), and although OFI fi_info detects PSM provider (see
attached output of fi_info), when I mention it during GASNet configure
(config.log attached, please see line #10126), GASNet warns:

WARNING: Invalid or unsupported OFI provider specified via the
--with-ofi-provider flag.
Defaulting to runtime provider feature detection for ofi-conduit.

GASNet configuration:

CC=icc CXX=icpc MPI_CC=/home/sghosh/builds/mvapich222-intel-psm/bin/mpicc
MPI_CFLAGS="-I/home/sghosh/builds/mvapich222-intel-psm/include"
MPI_LIBS="-L/home/sghosh/builds/mvapich222-intel-psm/lib -lpciaccess -lnuma
-lxml2" ./configure --prefix=/home/sghosh/builds/gasnet-ofi --enable-ofi
--with-ofi-home=/home/sghosh/builds/libfabric --disable-psm --disable-ibv
--with-ofi-spawner=mpi --with-ofi-provider=psm

GASNet throws this warning during application execution:

WARNING: Using OFI provider (verbs;ofi_rxm), which has not been validated
to provide
WARNING: acceptable GASNet performance. You should consider using a more
WARNING: hardware-appropriate GASNet conduit. See ofi-conduit/README.
*** FATAL ERROR: The statically-determined value for
GASNETC_OFI_HAS_MR_SCALABLE=1 does
  not match the memory registration support that the (verbs;ofi_rxm)
provider reported.
  This could happen if a provider that previously only supported FI_MR_BASIC
  added support for FI_MR_SCALABLE, or if the wrong provider was selected
at runtime.
  Use configure option --disable-ofi-mr-scalable to correct this.

Instead of picking up psm, GASNet picks up verbs. I have reported this
issue on GASNet user list as well. Please let me know if I can try some
options in building libfabrics for PSM provider, that may fix this issue.

-- 
Thanks,
Sayan | https://sg0.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20180619/67f96788/attachment.html>
-------------- next part --------------
provider: psm
    fabric: psm
    domain: psm
    version: 1.6
    type: FI_EP_RDM
    protocol: FI_PROTO_PSMX
provider: verbs;ofi_rxm
    fabric: IB-0x18338657682652659712
    domain: mlx4_0
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: verbs;ofi_rxm
    fabric: IB-0x18338657682652659712
    domain: qib0
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: tcp;ofi_rxm
    fabric: TCP-IP
    domain: tcp
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: tcp;ofi_rxm
    fabric: TCP-IP
    domain: tcp
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: tcp;ofi_rxm
    fabric: TCP-IP
    domain: tcp
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: tcp;ofi_rxm
    fabric: TCP-IP
    domain: tcp
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: tcp;ofi_rxm
    fabric: TCP-IP
    domain: tcp
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: tcp;ofi_rxm
    fabric: TCP-IP
    domain: tcp
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: tcp;ofi_rxm
    fabric: TCP-IP
    domain: tcp
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: tcp;ofi_rxm
    fabric: TCP-IP
    domain: tcp
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_RXM
provider: verbs
    fabric: IB-0x18338657682652659712
    domain: mlx4_0
    version: 1.0
    type: FI_EP_MSG
    protocol: FI_PROTO_RDMA_CM_IB_RC
provider: verbs
    fabric: IB-0x18338657682652659712
    domain: mlx4_0-rdm
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_IB_RDM
provider: verbs
    fabric: IB-0x18338657682652659712
    domain: qib0
    version: 1.0
    type: FI_EP_MSG
    protocol: FI_PROTO_RDMA_CM_IB_RC
provider: verbs
    fabric: IB-0x18338657682652659712
    domain: qib0-rdm
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_IB_RDM
provider: verbs
    fabric: IB-0x18338657682652659712
    domain: qib0-dgram
    version: 1.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_IB_UD
provider: verbs
    fabric: IB-0x18338657682652659712
    domain: mlx4_0-dgram
    version: 1.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_IB_UD
provider: UDP
    fabric: UDP-IP
    domain: udp
    version: 1.1
    type: FI_EP_DGRAM
    protocol: FI_PROTO_UDP
provider: UDP
    fabric: UDP-IP
    domain: udp
    version: 1.1
    type: FI_EP_DGRAM
    protocol: FI_PROTO_UDP
provider: UDP
    fabric: UDP-IP
    domain: udp
    version: 1.1
    type: FI_EP_DGRAM
    protocol: FI_PROTO_UDP
provider: UDP
    fabric: UDP-IP
    domain: udp
    version: 1.1
    type: FI_EP_DGRAM
    protocol: FI_PROTO_UDP
provider: UDP
    fabric: UDP-IP
    domain: udp
    version: 1.1
    type: FI_EP_DGRAM
    protocol: FI_PROTO_UDP
provider: UDP
    fabric: UDP-IP
    domain: udp
    version: 1.1
    type: FI_EP_DGRAM
    protocol: FI_PROTO_UDP
provider: UDP
    fabric: UDP-IP
    domain: udp
    version: 1.1
    type: FI_EP_DGRAM
    protocol: FI_PROTO_UDP
provider: UDP
    fabric: UDP-IP
    domain: udp
    version: 1.1
    type: FI_EP_DGRAM
    protocol: FI_PROTO_UDP
provider: sockets
    fabric: 192.168.64.0/20
    domain: eth0
    version: 2.0
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 192.168.64.0/20
    domain: eth0
    version: 2.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 192.168.64.0/20
    domain: eth0
    version: 2.0
    type: FI_EP_RDM
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 140.221.74.0/23
    domain: eth2
    version: 2.0
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 140.221.74.0/23
    domain: eth2
    version: 2.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 140.221.74.0/23
    domain: eth2
    version: 2.0
    type: FI_EP_RDM
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 192.168.120.0/21
    domain: ib0
    version: 2.0
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 192.168.120.0/21
    domain: ib0
    version: 2.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 192.168.120.0/21
    domain: ib0
    version: 2.0
    type: FI_EP_RDM
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 127.0.0.0/8
    domain: lo
    version: 2.0
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 127.0.0.0/8
    domain: lo
    version: 2.0
    type: FI_EP_DGRAM
    protocol: FI_PROTO_SOCK_TCP
provider: sockets
    fabric: 127.0.0.0/8
    domain: lo
    version: 2.0
    type: FI_EP_RDM
    protocol: FI_PROTO_SOCK_TCP
provider: tcp
    fabric: TCP-IP
    domain: tcp
    version: 0.1
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: tcp
    fabric: TCP-IP
    domain: tcp
    version: 0.1
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: tcp
    fabric: TCP-IP
    domain: tcp
    version: 0.1
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: tcp
    fabric: TCP-IP
    domain: tcp
    version: 0.1
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: tcp
    fabric: TCP-IP
    domain: tcp
    version: 0.1
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: tcp
    fabric: TCP-IP
    domain: tcp
    version: 0.1
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: tcp
    fabric: TCP-IP
    domain: tcp
    version: 0.1
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
provider: tcp
    fabric: TCP-IP
    domain: tcp
    version: 0.1
    type: FI_EP_MSG
    protocol: FI_PROTO_SOCK_TCP
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: text/x-log
Size: 760041 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20180619/67f96788/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log.ofi
Type: application/octet-stream
Size: 92641 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20180619/67f96788/attachment.obj>

More information about the ofiwg mailing list