<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 11, 2015 at 10:03 AM, Amir Vadai <span dir="ltr"><<a href="mailto:amirv.mellanox@gmail.com" target="_blank">amirv.mellanox@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">On Tue, Jun 9, 2015 at 11:36 AM, Dinesh Kb <<a href="mailto:dinesh.kb@vvdntech.in">dinesh.kb@vvdntech.in</a>> wrote:<br>
><br>
> One more thing we have modified is in the file "drivers/infiniband/hw/rxe/rxe_net.c<br>
><br>
> we have commented the line 426<br>
><br>
> //struct net_device *ndev = arg;<br>
><br>
> and added the following<br>
><br>
> struct netdev_notifier_info *info =arg;<br>
> struct net_device *ndev = info->dev;<br>
><br>
> moreover we have copied "ib_pack.h" from the vanilla(3.0.0+) to latest kernel (4.0.4)<br>
><br>
> please help me... to achieve SoftRoCE in latest kernel<br>
><br>
><br>
><br>
> with warm regards<br>
> Dinesh.K.B<br>
> Software Engineer<br>
><br>
><br>
> Cell : +91 9944456867 | Skype : dinesh_kb93<br>
><br>
><br>
> On Tue, Jun 9, 2015 at 1:53 PM, Dinesh Kb <<a href="mailto:dinesh.kb@vvdntech.in">dinesh.kb@vvdntech.in</a>> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> "I am new to OFED"<br>
>><br>
>> I have downloaded the "Soft-RoCE" which includes the kernel (3.0.0+) and rxe support.<br>
>> I have Installed all the packages included in the OFED-1.5.2-rxe except the kernel-ib.<br>
>><br>
>> In kernel 3.0.0+, I am able to run the "rping" test successfully and able to perform the benchmark tests, I am comfort with benchmark output.<br>
>><br>
>> Now I am in need to perform the RoCE operation in the higher kernel version...say 4.0.4<br>
>><br>
>> I have downloaded source file for the kernel version 4.0.4 and i have copied "driver/infiniband/hw/rxe" directory from kernel 3.0.0+ and pasted in the 4.0.4 kernel source and I have applied some patch for rxe to work smoothly...<br>
>><br>
>> I am facing Kernel crash while running rping...<br>
>> and regarding the benchmark...<br>
>><br>
>> when running rping as server in the higher kernel version and client in the lower version is hold fine....<br>
>> but its worse(crash) when running vice-versa<br>
>><br>
>> Now i am facing problems, when kernel 4.04 acts as a client...<br>
>> while taking the wireshark output.. the server constantly sends the "read request" and its not getting any response from the client...<br>
>><br>
>> the virtual address and dma length fields are "zero".<br>
>><br>
>><br>
>> Kindly help me.....<br>
>><br>
>> Is there any possibilities/ perfect patch to make SoftRoCE work fine with higher kernel version (say kernel 3.19+)<br>
>><br>
>> I hereby attached the wireshark output for ib_read_bw (when higher version acts as a client and 3.0.0+ acts as a server) and the patch i have applied as an attachment<br>
>><br>
>> the crash in the server is as follows<br>
>><br>
>> root@ls2085aqds:/etc/libibverbs.d# rping -s<br>
>> ------------[ cut here ]------------<br>
>> WARNING: CPU: 0 PID: 810 at kernel/softirq.c:146 __local_bh_enable_ip+0x84/0xc0()<br>
>> Modules linked in: ib_rxe_net ib_rxe rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr ipv6<br>
>> CPU: 0 PID: 810 Comm: kworker/0:1 Not tainted 3.16.0-Layerscape2-SDK+g330dd4b #20<br>
>> Workqueue: ib_cm cm_work_handler [ib_cm]<br>
>> Call trace:<br>
>> [<ffffffc000088084>] dump_backtrace+0x0/0x12c<br>
>> [<ffffffc0000881c0>] show_stack+0x10/0x1c<br>
>> [<ffffffc000552fa4>] dump_stack+0x74/0xc4<br>
>> [<ffffffc0000a9b8c>] warn_slowpath_common+0x84/0xac<br>
>> [<ffffffc0000a9c78>] warn_slowpath_null+0x14/0x20<br>
>> [<ffffffc0000ae3d0>] __local_bh_enable_ip+0x80/0xc0<br>
>> [<ffffffc00049fccc>] __dev_queue_xmit+0x1cc/0x408<br>
>> [<ffffffc00049ff14>] dev_queue_xmit+0xc/0x18<br>
>> [<ffffffbffc11b620>] send_finish+0x34/0x40 [ib_rxe_net]<br>
>> [<ffffffbffc11b6b8>] send+0x8c/0xec [ib_rxe_net]<br>
>> [<ffffffbffc10933c>] $x+0x318/0x334 [ib_rxe]<br>
>> [<ffffffbffc109be0>] $x+0xa8/0x120 [ib_rxe]<br>
>> [<ffffffbffc109d54>] rxe_run_task+0x4c/0x90 [ib_rxe]<br>
>> [<ffffffbffc1093b4>] arbiter_skb_queue+0x5c/0x8c [ib_rxe]<br>
>> [<ffffffbffc0fffd4>] rxe_requester+0x83c/0xddc [ib_rxe]<br>
>> [<ffffffbffc109be0>] $x+0xa8/0x120 [ib_rxe]<br>
>> [<ffffffbffc109d54>] rxe_run_task+0x4c/0x90 [ib_rxe]<br>
>> [<ffffffbffc104e0c>] rxe_post_send+0x80/0x40c [ib_rxe]<br>
>> [<ffffffbffc097068>] ib_send_mad+0x288/0x454 [ib_mad]<br>
>> [<ffffffbffc09759c>] ib_post_send_mad+0x190/0x544 [ib_mad]<br>
>> [<ffffffbffc0b075c>] ib_send_cm_rej+0xd0/0x194 [ib_cm]<br>
>> [<ffffffbffc0b19b8>] cm_destroy_id+0x188/0x300 [ib_cm]<br>
>> [<ffffffbffc0b1e80>] cm_process_work+0x154/0x17c [ib_cm]<br>
>> [<ffffffbffc0b2600>] cm_req_handler+0x758/0x978 [ib_cm]<br>
>> [<ffffffbffc0b28ec>] cm_work_handler+0xcc/0x1584 [ib_cm]<br>
>> [<ffffffc0000c0918>] process_one_work+0x114/0x354<br>
>> [<ffffffc0000c12f8>] worker_thread+0x13c/0x500<br>
>> [<ffffffc0000c7034>] kthread+0xd0/0xe8<br>
>> ---[ end trace be7a1b95934c8f03 ]---<br>
>><br>
>><br>
>> with warm regards<br>
>> Dinesh.K.B<br>
>> Software Engineer<br>
>><br>
>><br>
>> Cell : +91 9944456867 | Skype : dinesh_kb93<br>
<br>
+ Kamal<br><br></blockquote><div><br></div><div><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi
Dinesh,</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I
think that you are using an old version of Soft-RoCE. We have a github project
[1] that include a newer version of Soft-RoCE that based on kernel 4.0.*</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">To
configure your environment please follow the wiki page under [2]. And to
validate that everything is working you can follow the wiki page under [3].</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Please
let me know if you had any issue.</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[1] -
<a href="https://github.com/SoftRoCE/" target="_blank">https://github.com/SoftRoCE/</a>
</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[2] -
<a href="https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home" target="_blank">https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home</a>
</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[3] -
<a href="https://github.com/SoftRoCE/rxe-dev/wiki/Validate-that-RXE-is-working" target="_blank">https://github.com/SoftRoCE/rxe-dev/wiki/Validate-that-RXE-is-working</a>
</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks,</span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Kamal</span></p></div><div> </div></div><br></div></div>