<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div>Gaetano,<div>This is the right list for such a question - unfortunately I am just learning verbs programming myself, so cannot address your question.</div><div>There are others on the list that have much more experience - hopefully someone will respond.</div><div><br></div><div><br><div><div>On Aug 23, 2012, at 4:51 PM, Gaetano Mendola wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi all,<br>I'm sorry in advance if this is not the right mailing list for my question.<br><br>In my application I use an infiniband infrastructure to send a stream<br>of data from a server to<br>another one. I have used to easy the development ip over infiniband<br>because I'm more familiar<br>with socket programming. Until now the performance (max bw) was good<br>enough for me (I knew<br>I wasn't getting the maximum bandwith achievable), now I need to get<br>out from that infiniband<br>connection more bandwidth.<br><br>ib_write_bw claims that my max achievable bandwidth is around 1500<br>MB/s (I'm not getting<br>3000MB/s because my card is installed in a PCI 2.0 8x).<br><br>So far so good. I coded my communication channel using ibverbs and<br>rdma but I'm getting far<br>less than the bandwith I can get, I'm even getting a bit less<br>bandwidth than using socket but<br>at least my application doesn't use any CPU power:<br><br>ib_write_bw: 1500 MB/s<br><br>sockets: 700 MB/s <= One core of my system is at 100% during this test<br><br>ibvers+rdma: 600 MB/s <= No CPU is used at all during this test<br><br>It seems that the bottleneck is here:<br><br>ibv_sge sge;<br>sge.addr = (uintptr_t)memory_to_transfer;<br>sge.length = memory_to_transfer_size;<br>sge.lkey = memory_to_transfer_mr->lkey;<br><br>ibv_send_wr wr;<br>memset(&wr, 0, sizeof(wr));<br>wr.wr_id = 0;<br>wr.opcode = IBV_WR_RDMA_WRITE;<br>wr.sg_list = &sge;<br>wr.num_sge = 1;<br>wr.send_flags = IBV_SEND_SIGNALED;<br>wr.wr.rdma.remote_addr = (uintptr_t)thePeerMemoryRegion.addr;<br>wr.wr.rdma.rkey = thePeerMemoryRegion.rkey;<br><br>ibv_send_wr *bad_wr = NULL;<br>if (ibv_post_send(theCommunicationIdentifier->qp, &wr, &bad_wr) != 0) {<br> notifyError("Unable to ibv post receive");<br>}<br><br>at this point the next code waiting for completation that is:<br><br>//Wait for completation<br>ibv_cq *cq;<br>void* cq_context;<br>if (ibv_get_cq_event(theCompletionEventChannel, &cq, &cq_context) != 0) {<br> notifyError("Unable to get a ibv cq event");<br>}<br><br>ibv_ack_cq_events(cq, 1);<br><br>if (ibv_req_notify_cq(cq, 0) != 0) {<br> notifyError("Unable to get a req notify");<br>}<br><br>ibv_wc wc;<br>int myRet = ibv_poll_cq(cq, 1, &wc);<br>if (myRet > 1) {<br> LOG(WARNING) << "Got more than a single ibv_wc, expecting one";<br>}<br><br><br>The time from my ibv_post_send and when ibv_get_cq_event returns an<br>event is 13.3ms when transfering chuncks of 8 MB achieving then around 600 MB/s.<br><br>To specify more (in pseudocode what I do globally):<br><br>Active Side:<br><br>post a message receive<br>rdma connection<br>wait for rdma connection event<br><<at this point transfer tx flow starts>><br>start:<br>register memory containing bytes to transfer<br>wait remote memory region addr/key ( I wait for a ibv_wc)<br>send data with ibv_post_send<br>post a message receive<br>wait for ibv_post_send event ( I wait for a ibv_wc) (this lasts 13.3 ms)<br>send message "DONE"<br>unregister memory<br>goto start<br><br>Passive Side:<br><br>post a message receive<br>rdma accept<br>wait for rdma connection event<br><<at this point transfer rx flow starts>><br>start:<br>register memory that has to receive the bytes<br>send addr/key of memory registered<br>wait "DONE" message<br>unregister memory<br>post a message receive<br>goto start<br><br>Does anyone knows what I'm doing wrong? Or what I can improve? I'm not<br>affected by<br>"Not Invented Here" syndrome so I'm even open to throw away what I<br>have done until<br>now and adopting something else.<br><br>I only need a point to point contiguous transfer.<br><br><br>Regards<br>Gaetano Mendola<br><br><br>--<br><a href="http://cpp-today.blogspot.com">cpp-today.blogspot.com</a><br>_______________________________________________<br>Users mailing list<br>Users@lists.openfabrics.org<br>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>====================================</div><div><br></div><div>Susan Coulter<br>HPC-3 Network/Infrastructure<br>505-667-8425<br>Increase the Peace...<br>An eye for an eye leaves the whole world blind<br>====================================</div></div></span></div></span></span>
</div>
<br></div></body></html>