<html><body><div style="color:#000; background-color:#fff; font-family:tahoma, new york, times, serif;font-size:12pt"><div><span>Hi Steve,</span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span><br></span></div><div><span>Thanks. 'rdma_bw --cma" works.</span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span><br></span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span>'rping' is based on RDMA CM + IB verbs, and works on IB and iWarp.</span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span>'ib_write_bw' is based on socket + IB verbs, and
only works on IB.</span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span>'rdma_bw' is based on RDMA CM + IB verbs and socket + IB verbs while '--cma'.<br></span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span><br></span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span>Is that correct? If so, I am confused about RDMA CM. What's the different between the RDMA CM(rdma_connect/rdma_accept/rdma_create_qp) and soc ket plus IB(connect/accept/ibv_create_qp)?<br></span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span>Does that mean the app
could only use RDMA CM to setup connection, then do RDMA operation in iWarp environment?</span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;"><span>Thank you very much.</span></div><div style="color: rgb(0, 0, 0); font-size: 20px; font-family: tahoma,new york,times,serif; background-color: transparent; font-style: normal;">Yufei<br><span></span></div><div><br></div><div style="font-family: tahoma,new york,times,serif; font-size: 12pt;"><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;"><font face="Arial" size="2"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Steve Wise <swise@opengridcomputing.com><br><b><span style="font-weight:
bold;">To:</span></b> ren yufei <renyufei83@yahoo.com.cn><br><b><span style="font-weight: bold;">Cc:</span></b> OpenFabrics EWG <ewg@lists.openfabrics.org><br><b><span style="font-weight: bold;">Sent:</span></b> Friday, April 22, 2011 11:21 AM<br><b><span style="font-weight: bold;">Subject:</span></b> Fw: [ewg] iWarp - Failed to modify RC QP to RTS<br></font><br> ib_write_bw is an Infiniband-only application. Try rdma_bw and rdma_lat and use the --cma flag.<br><br>Steve.<br><br>On 04/21/2011 05:39 PM, ren yufei wrote:<br>> Hi,<br>><br>> Two hosts connected through both IB and iWarp.<br>> 1 'ib_write_bw' success via IB connection, while failure via iWarp connection.<br>> 2 Both clinet and server export 'Failed to modify RC QP to RTS' failure.<br>> 2.1 client errno: 22(Invalid argument)<br>> 2.2 server errno: 22(Invalid argument)<br>> 3 'rping' works via both IB and iWarp.<br>><br>>
Any suggestion?<br>><br>> Connection<br>> ==<br>> client device <====> server device<br>> IB mlx4_1 mlx4_0 (addr IB)<br>> iWarp nes0 nes0 (addr iWarp)<br>><br>> iWarp failure<br>> ==<br>> server# ib_write_bw -d nes0<br>> client# ib_write_bw [addr iWarp] -d nes0<br>><br>> ------------------------------------------------------------------<br>> RDMA_Write BW Test<br>> Number of qp's running 1<br>> Connection type : RC<br>> Each Qp will post up to 100 messages each
time<br>> Inline data is used up to 0 bytes message<br>> Link type is ETH<br>> Using gid index 0 as source GID<br>> local address: LID 0x01 QPN 0x0062 PSN 0xf2d446 RKey 0x104e6416 VAddr 0x002b699d075000<br>> GID: 00:18:85:02:248:04:00:00:00:00:00:00:00:00:00:00<br>> remote address: LID 0x01 QPN 0x0277 PSN 0xbc7dd7 RKey 0x60f7b956 VAddr 0x002b6ff84f3000<br>> GID: 00:18:85:02:248:136:00:00:00:00:00:00:00:00:00:00<br>> Failed to modify RC QP to RTS<br>> Unable to Connect the HCA's through the link<br>><br>><br>> IB success<br>> ==<br>> 'ib_write_bw' via IB<br>> server# ib_write_bw -d mlx4_0<br>> client# ib_write_bw [addr IB] -d mlx4_1<br>><br>> ------------------------------------------------------------------<br>> RDMA_Write BW Test<br>> Number of qp's running
1<br>> Connection type : RC<br>> Each Qp will post up to 100 messages each time<br>> Inline data is used up to 0 bytes message<br>> Link type is ETH<br>> Using gid index 0 as source GID<br>> local address: LID 0000 QPN 0x30004e PSN 0x90ea46 RKey 0x10002a0f VAddr 0x002b3ceb0e9000<br>> GID: 254:128:00:00:00:00:00:00:02:02:201:255:254:08:254:218<br>> remote address: LID 0000 QPN 0x18004e PSN 0x29c2f2 RKey 0x58002af9 VAddr 0x002b841d5ce000<br>> GID: 254:128:00:00:00:00:00:00:02:02:201:255:254:08:254:212<br>> Mtu : 1024<br>> ------------------------------------------------------------------<br>> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<br>> Conflicting CPU frequency values detected: 2793.000000 != 1596.000000<br>> 65536 5000 0.00
0.00<br>> ------------------------------------------------------------------<br>><br>><br>> ibv_devinfo<br>> ==<br>> client:<br>> hca_id: mlx4_1<br>> transport: InfiniBand (0)<br>> fw_ver: 2.7.854<br>> node_guid: ffff:ffff:ffff:ffff<br>> sys_image_guid: ffff:ffff:ffff:ffff<br>> vendor_id: 0x02c9<br>> vendor_part_id:
26478<br>> hw_ver: 0xB0<br>> board_id: MT_0D80110017<br>> phys_port_cnt: 1<br>> port: 1<br>> state: PORT_ACTIVE (4)<br>> max_mtu: 2048 (4)<br>> active_mtu:
1024 (3)<br>> sm_lid: 0<br>> port_lid: 0<br>> port_lmc: 0x00<br>> link_layer: Ethernet<br>><br>> hca_id: nes0<br>> transport: iWARP (1)<br>> fw_ver: 3.16<br>>
node_guid: 0012:5502:f804:0000<br>> sys_image_guid: 0012:5502:f804:0000<br>> vendor_id: 0x1255<br>> vendor_part_id: 256<br>> hw_ver: 0x5<br>> board_id: NES020 Board ID<br>> phys_port_cnt: 1<br>> port:
1<br>> state: PORT_ACTIVE (4)<br>> max_mtu: 4096 (5)<br>> active_mtu: 1024 (3)<br>> sm_lid: 0<br>> port_lid: 1<br>> port_lmc:
0x00<br>> link_layer: Ethernet<br>><br>> server<br>> hca_id: mlx4_0<br>> transport: InfiniBand (0)<br>> fw_ver: 2.7.854<br>> node_guid: ffff:ffff:ffff:ffff<br>> sys_image_guid: ffff:ffff:ffff:ffff<br>> vendor_id: 0x02c9<br>> vendor_part_id:
26478<br>> hw_ver: 0xB0<br>> board_id: MT_0D80110017<br>> phys_port_cnt: 1<br>> port: 1<br>> state: PORT_ACTIVE (4)<br>> max_mtu: 2048 (4)<br>> active_mtu:
1024 (3)<br>> sm_lid: 0<br>> port_lid: 0<br>> port_lmc: 0x00<br>><br>> hca_id: nes0<br>> transport: iWARP (1)<br>> fw_ver: 3.16<br>> node_guid: 0012:5502:f888:0000<br>>
sys_image_guid: 0012:5502:f888:0000<br>> vendor_id: 0x1255<br>> vendor_part_id: 256<br>> hw_ver: 0x5<br>> board_id: NES020 Board ID<br>> phys_port_cnt: 1<br>> port: 1<br>> state:
PORT_ACTIVE (4)<br>> max_mtu: 4096 (5)<br>> active_mtu: 4096 (5)<br>> sm_lid: 0<br>> port_lid: 1<br>> port_lmc: 0x00<br>><br>> Thanks.<br>><br>> Yufei<br>><br>><br>> _______________________________________________<br>> ewg mailing list<br>> <a
ymailto="mailto:ewg@lists.openfabrics.org" href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><br>> <a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg</a><br><br><br><br></div></div></div></body></html>