[ofa-general] Internal loopback for HCA (with no external cable required)
Phillip Wilson
phillipwils at gmail.com
Sun Feb 24 21:44:01 PST 2008
Internal loopback for HCA (with no external cable required)
<<Background Information>>
I would like to modify the existing "ibv_rc_pingpong or
(ivb_xx_pingpong)" program to perform internal loopback on a HCA with
no external cable required. My intention is to support internal DMA
loopback for packets transmitted and received between a QP that is
assigned to the same HCA port; the program will connect the QP send
queue to its receive queue.
I am new to Infiniband and I would like some advice on how to approach
this task. So far, I have been going through the management, verb
code, and the Infiniband Architecture specification volume 1 release
1.2. I have installed the OFED version 1.2.5 on Redhat (see uname –a
below for details).
I have two Mellanox dual port (VID=15B3: DID=5A46, VID=15B3: DID=5A44,
tavor) HCAs in the system. I have configured one HCA (mthca0) for
external loopback by connecting the HCA's ports together with a cable.
The other HCA has no cables connected to its ports. After running
the opensm program, the HCA (mthca0) cabled for external loopback had
a LID value assigned to each of its ports. The other HCA (mthca1)
ports were not assigned LIDs as expected. I was then able to run the
ibv_xx_pingpong programs as showed below.
Based on my limited understanding of the opensm program, I am not sure
if it possible to use this program to setup the HCA ports (mthca1)
when the port state is "PORT_DOWN". Can the guid2lid file be used to
force the opensm to assigned LIDs values to ports of and HCA that does
not have an active link? If not, I believe that I will need to
programmatically use subnet management packets to configure the HCA
for internal loopback. According to the Infiniband specification,
loopback can be done by Directed Routing. That is by setting the
HopCount=0, DrSLID=DrDLID and PacketLifeTime=0.
<<Questions>>
1.) Is it possible to modify the ibv_rc_pingpong program for internal loopback?
2.) If each mthca1 port is assigned a LID, can the ibv_rc_pingpong
program be used as is?
The ibv_rc_pingpong program uses the LID to route packets. Is it
necessary to assigned the sm_lid and port_lid values greater than zero
once the HCA is setup for internal loopback using Directed Routing as
shown in figure 186 on page 799 within the Infiniband specification?
If so, what other port parameters should be set? Is the port logical
(i.e. PORT_ACTIVE) and physical states (LINK_UP) important for
internal loopback?
<<System Information below>>
[root at hpdst165 src]# uname -a
Linux hpdst165 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:55 EDT 2007 ia64
ia64 ia64 GNU/Linux
External Loopback
[root at hpdst165 src]# ibv_rc_pingpong &
[1] 4359
[root at hpdst165 src]# local address: LID 0x0001, QPN 0x0a0406, PSN 0x46d737
[root at hpdst165 src]# ibv_rc_pingpong hpdst165 &
[2] 4361
[root at hpdst165 src]# local address: LID 0x0001, QPN 0x0a0407, PSN 0xb0a59e
remote address: LID 0x0001, QPN 0x0a0406, PSN 0x46d737
remote address: LID 0x0001, QPN 0x0a0407, PSN 0xb0a59e
8192000 bytes in 0.05 seconds = 1337.50 Mbit/sec
1000 iters in 0.05 seconds = 49.00 usec/iter
8192000 bytes in 0.05 seconds = 1330.63 Mbit/sec
1000 iters in 0.05 seconds = 49.25 usec/iter
[1]- Done ibv_rc_pingpong
[2]+ Done ibv_rc_pingpong hpdst165
[root at hpdst165 bin]# ibv_devinfo -v
hca_id: mthca0
fw_ver: 3.3.12
node_guid: 0016:35ff:ffbf:0944
sys_image_guid: 0016:35ff:ffbf:0947
vendor_id: 0x02c9
vendor_part_id: 23108
hw_ver: 0xA1
board_id: HP_0030000001
phys_port_cnt: 2
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffff000
max_qp: 64512
max_qp_wr: 65535
device_cap_flags: 0x00001c76
max_sge: 28
max_sge_rd: 0
max_cq: 65408
max_cqe: 131071
max_mr: 131056
max_pd: 32768
max_qp_rd_atom: 4
max_ee_rd_atom: 0
max_res_rd_atom: 258048
max_qp_init_rd_atom: 128
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
max_ee: 0
max_rdd: 0
max_mw: 0
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 8192
max_mcast_qp_attach: 8
max_total_mcast_qp_attach: 65536
max_ah: 0
max_fmr: 0
max_srq: 1008
max_srq_wr: 65535
max_srq_sge: 28
max_pkeys: 64
local_ca_ack_delay: 15
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 1
port_lid: 1
port_lmc: 0x00
max_msg_sz: 0x80000000
port_cap_flags: 0x02510a68
max_vl_num: 8 (4)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 64
gid_tbl_len: 32
subnet_timeout: 18
init_type_reply: 0
active_width: 4X (2)
active_speed: 2.5 Gbps (1)
phys_state: LINK_UP (5)
GID[ 0]:
fe80:0000:0000:0000:0016:35ff:ffbf:0945
port: 2
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 1
port_lid: 2
port_lmc: 0x00
max_msg_sz: 0x80000000
port_cap_flags: 0x02510a68
max_vl_num: 8 (4)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 64
gid_tbl_len: 32
subnet_timeout: 18
init_type_reply: 0
active_width: 4X (2)
active_speed: 2.5 Gbps (1)
phys_state: LINK_UP (5)
GID[ 0]:
fe80:0000:0000:0000:0016:35ff:ffbf:0946
hca_id: mthca1
fw_ver: 3.3.12
node_guid: 0016:35ff:ffbf:1954
sys_image_guid: 0016:35ff:ffbf:1957
vendor_id: 0x02c9
vendor_part_id: 23108
hw_ver: 0xA1
board_id: HP_0030000001
phys_port_cnt: 2
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffff000
max_qp: 64512
max_qp_wr: 65535
device_cap_flags: 0x00001c76
max_sge: 28
max_sge_rd: 0
max_cq: 65408
max_cqe: 131071
max_mr: 131056
max_pd: 32768
max_qp_rd_atom: 4
max_ee_rd_atom: 0
max_res_rd_atom: 258048
max_qp_init_rd_atom: 128
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
max_ee: 0
max_rdd: 0
max_mw: 0
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 8192
max_mcast_qp_attach: 8
max_total_mcast_qp_attach: 65536
max_ah: 0
max_fmr: 0
max_srq: 1008
max_srq_wr: 65535
max_srq_sge: 28
max_pkeys: 64
local_ca_ack_delay: 15
port: 1
state: PORT_DOWN (1)
max_mtu: 2048 (4)
active_mtu: 512 (2)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
max_msg_sz: 0x80000000
port_cap_flags: 0x02510a68
max_vl_num: 8 (4)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 64
gid_tbl_len: 32
subnet_timeout: 0
init_type_reply: 0
active_width: 1X (1)
active_speed: 2.5 Gbps (1)
phys_state: POLLING (2)
GID[ 0]:
fe80:0000:0000:0000:0016:35ff:ffbf:1955
port: 2
state: PORT_DOWN (1)
max_mtu: 2048 (4)
active_mtu: 512 (2)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
max_msg_sz: 0x80000000
port_cap_flags: 0x02510a68
max_vl_num: 8 (4)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 64
gid_tbl_len: 32
subnet_timeout: 0
init_type_reply: 0
active_width: 1X (1)
active_speed: 2.5 Gbps (1)
phys_state: POLLING (2)
GID[ 0]:
fe80:0000:0000:0000:0016:35ff:ffbf:1956
More information about the general
mailing list