<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Since RDMA CM will be used, I would verify that IPOIB has been configured for the port being used and that the network interface has been brought up. Otherwise, as Sean suggested I would capture the Libfabric debug output and provide that.<div class=""><br class=""></div><div class="">Steve<br class="">
<div><br class=""><blockquote type="cite" class=""><div class="">On Jan 16, 2020, at 11:48 AM, Philip Davis <<a href="mailto:philip.e.davis@rutgers.edu" class="">philip.e.davis@rutgers.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Steve,
<div class=""><br class="">
</div>
<div class="">Here is the requested output:<br class="">
<div class=""><br class="">
</div>
<div class="">
<div style="caret-color: rgb(0, 0, 0);" class="">[]$ ibv_devinfo -v </div>
<span style="caret-color: rgb(0, 0, 0);" class="">hca_id: mlx4_0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> transport: InfiniBand (0)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> fw_ver: 2.36.5120</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> node_guid: 248a:0703:0077:9520</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> sys_image_guid: 248a:0703:0077:9523</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> vendor_id: 0x02c9</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> vendor_part_id: 4103</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> hw_ver: 0x0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> board_id: MT_1090111019</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> phys_port_cnt: 2</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_mr_size: 0xffffffffffffffff</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> page_size_cap: 0xfffffe00</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_qp: 131000</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_qp_wr: 16351</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> device_cap_flags: 0x057e9c76</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> BAD_PKEY_CNTR</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> BAD_QKEY_CNTR</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> AUTO_PATH_MIG</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> CHANGE_PHY_PORT</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> UD_AV_PORT_ENFORCE</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> PORT_ACTIVE_EVENT</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> SYS_IMAGE_GUID</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> RC_RNR_NAK_GEN</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> MEM_WINDOW</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> UD_IP_CSUM</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> XRC</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> MEM_MGT_EXTENSIONS</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> MEM_WINDOW_TYPE_2B</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> RAW_IP_CSUM</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> Unknown flags: 0x488000</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_sge: 32</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_sge_rd: 30</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_cq: 65408</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_cqe: 4194303</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_mr: 524032</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_pd: 32764</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_qp_rd_atom: 16</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_ee_rd_atom: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_res_rd_atom: 2096000</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_qp_init_rd_atom: 128</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_ee_init_rd_atom: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> atomic_cap: ATOMIC_HCA (1)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_ee: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_rdd: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_mw: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_raw_ipv6_qp: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_raw_ethy_qp: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_mcast_grp: 8192</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_mcast_qp_attach: 248</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_total_mcast_qp_attach: 2031616</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_ah: 2147483647</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_fmr: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_srq: 65472</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_srq_wr: 16383</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_srq_sge: 31</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_pkeys: 128</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> local_ca_ack_delay: 15</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> general_odp_caps:</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> rc_odp_caps:</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> NO SUPPORT</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> uc_odp_caps:</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> NO SUPPORT</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> ud_odp_caps:</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> NO SUPPORT</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> completion timestamp_mask: 0x0000ffffffffffff</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> hca_core_clock: 427000kHZ</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> device_cap_flags_ex: 0x57E9C76</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> tso_caps:</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_tso: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> rss_caps:</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_rwq_indirection_tables: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_rwq_indirection_table_size: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> rx_hash_function: 0x0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> rx_hash_fields_mask: 0x0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_wq_type_rq: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> packet_pacing_caps:</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> qp_rate_limit_min: 0kbps</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> qp_rate_limit_max: 0kbps</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> port: 1</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> state: PORT_ACTIVE (4)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_mtu: 4096 (5)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> active_mtu: 4096 (5)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> sm_lid: 1</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> port_lid: 1</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> port_lmc: 0x00</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> link_layer: InfiniBand</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_msg_sz: 0x40000000</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> port_cap_flags: 0x0259486a</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_vl_num: 8 (4)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> bad_pkey_cntr: 0x0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> qkey_viol_cntr: 0x0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> sm_sl: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> pkey_tbl_len: 128</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> gid_tbl_len: 128</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> subnet_timeout: 18</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> init_type_reply: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> active_width: 4X (2)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> active_speed: 14.0 Gbps (16)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> phys_state: LINK_UP (5)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> GID[ 0]: fe80:0000:0000:0000:248a:0703:0077:9521</span><br style="caret-color: rgb(0, 0, 0);" class="">
<br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> port: 2</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> state: PORT_DOWN (1)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_mtu: 4096 (5)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> active_mtu: 4096 (5)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> sm_lid: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> port_lid: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> port_lmc: 0x00</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> link_layer: InfiniBand</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_msg_sz: 0x40000000</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> port_cap_flags: 0x02594868</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> max_vl_num: 8 (4)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> bad_pkey_cntr: 0x0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> qkey_viol_cntr: 0x0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> sm_sl: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> pkey_tbl_len: 128</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> gid_tbl_len: 128</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> subnet_timeout: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> init_type_reply: 0</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> active_width: 4X (2)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> active_speed: 2.5 Gbps (1)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<span style="caret-color: rgb(0, 0, 0);" class=""> phys_state: POLLING (2)</span><br style="caret-color: rgb(0, 0, 0);" class="">
<div style="caret-color: rgb(0, 0, 0);" class=""> GID[ 0]: fe80:0000:0000:0000:248a:0703:0077:9522</div>
<div style="caret-color: rgb(0, 0, 0);" class=""><br class="">
</div>
<div style="caret-color: rgb(0, 0, 0);" class=""><br class="">
</div>
<div style="caret-color: rgb(0, 0, 0);" class="">[]$ lsmod | grep ib<br class="">
ib_isert 50770 0 <br class="">
iscsi_target_mod 302966 1 ib_isert<br class="">
ib_iser 47813 0 <br class="">
libiscsi 57233 1 ib_iser<br class="">
scsi_transport_iscsi 99909 2 ib_iser,libiscsi<br class="">
ib_srpt 48170 0 <br class="">
target_core_mod 367918 3 iscsi_target_mod,ib_srpt,ib_isert<br class="">
ib_srp 48454 0 <br class="">
scsi_transport_srp 20993 1 ib_srp<br class="">
ib_ipoib 110142 0 <br class="">
ib_ucm 22589 0 <br class="">
ib_uverbs 64636 2 ib_ucm,rdma_ucm<br class="">
ib_umad 22080 4 <br class="">
rdma_cm 54426 4 rpcrdma,ib_iser,rdma_ucm,ib_isert<br class="">
ib_cm 47287 5 rdma_cm,ib_srp,ib_ucm,ib_srpt,ib_ipoib<br class="">
mlx4_ib 159474 0 <br class="">
ib_core 211874 14 rdma_cm,ib_cm,iw_cm,rpcrdma,mlx4_ib,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert<br class="">
libcrc32c 12644 4 xfs,raid456,nf_nat,nf_conntrack<br class="">
mlx4_core 309354 2 mlx4_en,mlx4_ib<br class="">
libahci 31992 1 ahci<br class="">
libata 238896 2 ahci,libahci<br class="">
devlink 30193 3 mlx4_en,mlx4_ib,mlx4_core</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 16, 2020, at 12:28 PM, Steve Welch <<a href="mailto:swelch@systemfabricworks.com" class="">swelch@systemfabricworks.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 16, 2020, at 11:03 AM, Philip Davis <<a href="mailto:philip.e.davis@rutgers.edu" class="">philip.e.davis@rutgers.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Steve,
<div class=""><br class="">
</div>
<div class="">Thanks for the quick response.</div>
<div class=""><br class="">
</div>
<div class="">I am expecting to use the the rxm provider for verbs, but in fi_info I do not see an FI_EP_MSG-type verbs provider.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Could you provide the output for “ibv_devinfo -v” and “lsmod | grep ib”?</div>
<div class=""><br class="">
</div>
<div class="">Steve</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: tcp;ofi_rxm</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXM</div>
<div class="">provider: verbs;ofi_rxd</div>
<div class=""> fabric: IB-0xfe80000000000000</div>
<div class=""> domain: mlx4_0-dgram</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: UDP;ofi_rxd</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_RXD</div>
<div class="">provider: verbs</div>
<div class=""> fabric: IB-0xfe80000000000000</div>
<div class=""> domain: mlx4_0-dgram</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_IB_UD</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: UDP</div>
<div class=""> fabric: UDP-IP</div>
<div class=""> domain: udp</div>
<div class=""> version: 1.1</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_UDP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.1.0.0/16</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.1.0.0/16</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.1.0.0/16</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.157.14.0/24</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.157.14.0/24</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 10.157.14.0/24</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em1</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: fe80::/64</div>
<div class=""> domain: em2</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 127.0.0.0/8</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 127.0.0.0/8</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: 127.0.0.0/8</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: ::1/128</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: ::1/128</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_DGRAM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: sockets</div>
<div class=""> fabric: ::1/128</div>
<div class=""> domain: lo</div>
<div class=""> version: 2.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: tcp</div>
<div class=""> fabric: TCP-IP</div>
<div class=""> domain: tcp</div>
<div class=""> version: 0.1</div>
<div class=""> type: FI_EP_MSG</div>
<div class=""> protocol: FI_PROTO_SOCK_TCP</div>
<div class="">provider: shm</div>
<div class=""> fabric: shm</div>
<div class=""> domain: shm</div>
<div class=""> version: 1.0</div>
<div class=""> type: FI_EP_RDM</div>
<div class=""> protocol: FI_PROTO_SHM</div>
</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Philip</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 16, 2020, at 11:03 AM, Steve Welch <<a href="mailto:swelch@systemfabricworks.com" class="">swelch@systemfabricworks.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class="" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div class="">Hi Phillip,</div>
<div class=""><br class="">
</div>
<div class="">Since you are specifying an FI_EP_RDM in your hints I assume you want to utilize the RXM provider on top of the Verbs core provider (i.e. ofi_rxm;verbs). The Verbs provider does not offer native FI_RDM_EP support. To use either XRC (or FI_EP_RDM
endpoint) you will have to use RXM, but I am unaware of any IB provider that supported XRC that did not support RC.</div>
<div class=""><br class="">
</div>
<div class="">If you issue a 'fi_info -p verbs -v’ it will list all the verbs domains supported and the underlying protocol and you could verify if RC should be supported (via RXM for FI_EP_RDM). If you issue 'fi_info -p “ofi_rxm;verbs”', you should see multiple
domains for the “ofi_rxm;verbs” provider combination. XRC domains have the “-xrc” suffix.</div>
<div class=""><br class="">
</div>
<div class="">If you must use XRC and the RXM/Verbs combination then you will need to set the environment variable FI_OFI_RXM_USE_SRX=1 and RXM will handle the shared RX details.</div>
<div class=""><br class="">
</div>
<div class="">Steve</div>
<div class=""><br class="">
</div>
</div>
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">On Jan 16, 2020, at 8:56 AM, Philip Davis <<a href="mailto:philip.e.davis@rutgers.edu" class="">philip.e.davis@rutgers.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hello,<br class="">
<br class="">
I am working with a user that is running on an older Infiniband cluster. Using libfaric with the following hints:<br class="">
<br class="">
hints->caps = FI_MSG | FI_SEND | FI_RECV | FI_REMOTE_READ |<br class="">
FI_REMOTE_WRITE | FI_RMA | FI_READ | FI_WRITE;<br class="">
hints->mode = FI_CONTEXT | FI_LOCAL_MR | FI_CONTEXT2 | FI_MSG_PREFIX |<br class="">
FI_ASYNC_IOV | FI_RX_CQ_DATA;<br class="">
hints->domain_attr->mr_mode = FI_MR_BASIC;<br class="">
hints->domain_attr->control_progress = FI_PROGRESS_AUTO;<br class="">
hints->domain_attr->data_progress = FI_PROGRESS_AUTO;<br class="">
hints->ep_attr->type = FI_EP_RDM;<br class="">
<br class="">
<br class="">
No verbs providers are found. Looking through the debug output, I suspect this is the crucial line:<br class="">
<br class="">
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints<br class="">
<br class="">
I take it that the underlying hardware is only compatible with FI_PROTO_RDMA_CM_IB_XRC protocol for MSG endpoints, and it looks like I need to have FI_SHARED_CONTEXT enabled for these endpoints to be supported. I’m having some trouble understanding the implications
of using FI_SHARED_CONTEXT. If I only ever use one endpoint, is there any functional or performance impact to setting this? I’d rather not change to using shared contexts unconditionally, so is there a good way for me to detect this situation other than to
do a maximally permissive fi_getinfo and iterate through the verbs results?<br class="">
<br class="">
Thanks,<br class="">
Philip<br class="">
_______________________________________________<br class="">
Libfabric-users mailing list<br class="">
<a href="mailto:Libfabric-users@lists.openfabrics.org" class="">Libfabric-users@lists.openfabrics.org</a><br class="">
<a href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.openfabrics.org%2Fmailman%2Flistinfo%2Flibfabric-users&data=02%7C01%7Cphilip.e.davis%40rutgers.edu%7Cc0e0cc2c82fe4113b22d08d79aa97baa%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C1%7C637147925028517698&sdata=xS%2B0nVcPIbof8BC1cODtdUPjVFwtmxGMk7D1ot2bd4s%3D&reserved=0" originalsrc="https://lists.openfabrics.org/mailman/listinfo/libfabric-users" shash="qf99llk5cXl3JfOYFD4l0CtFdadLDDdgXaHr9QVTjPGDc9H+CJ4edAQSuQfR9K3O2LV3ySm62B4CbqjlaKMa5CNVgAAypxAwduVh+Y4VsMIXpoJmSW0wuwDRIePNLxJ8JPRL+NTLiNd/xbC+jeZIJUjBUj2HLRP7OC6VUjDxuG0=" class="">https://lists.openfabrics.org/mailman/listinfo/libfabric-users</a></div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div></blockquote></div><br class=""></div></body></html>