[ewg] Need urgent help : Soft-RoCE in Linux-3.19

[Amir Vadai]

On Tue, Jun 9, 2015 at 11:36 AM, Dinesh Kb <dinesh.kb at vvdntech.in> wrote:
>
> One more thing we have modified is in the file "drivers/infiniband/hw/rxe/rxe_net.c
>
> we have commented the line 426
>
> //struct net_device *ndev = arg;
>
> and added the following
>
> struct netdev_notifier_info *info =arg;
> struct net_device *ndev = info->dev;
>
> moreover we have copied "ib_pack.h" from the vanilla(3.0.0+) to latest kernel (4.0.4)
>
> please help me... to achieve SoftRoCE in latest kernel
>
>
>
> with warm regards
> Dinesh.K.B
> Software Engineer
>
>
> Cell : +91 9944456867 | Skype : dinesh_kb93
>
>
> On Tue, Jun 9, 2015 at 1:53 PM, Dinesh Kb <dinesh.kb at vvdntech.in> wrote:
>>
>> Hi,
>>
>>                                                          "I am new to OFED"
>>
>> I have downloaded the "Soft-RoCE" which includes the kernel (3.0.0+) and rxe support.
>> I have Installed all the packages included in the OFED-1.5.2-rxe except the kernel-ib.
>>
>> In kernel 3.0.0+, I am able to run the "rping" test successfully and able to perform the benchmark tests, I am comfort with benchmark output.
>>
>> Now I am in need to perform the RoCE operation in the higher kernel version...say 4.0.4
>>
>> I have downloaded source file for  the kernel version 4.0.4 and i have copied "driver/infiniband/hw/rxe" directory from kernel 3.0.0+ and pasted in the 4.0.4 kernel source and I have applied some patch for rxe to work smoothly...
>>
>> I am facing Kernel crash while running rping...
>> and regarding the benchmark...
>>
>> when running rping as server in the higher kernel version and client in the lower version is hold fine....
>> but its worse(crash) when running vice-versa
>>
>> Now i am facing problems, when kernel 4.04 acts as a client...
>> while taking the wireshark output.. the server constantly sends the "read request" and its not getting any response from the client...
>>
>> the virtual address and dma length fields are "zero".
>>
>>
>> Kindly help me.....
>>
>> Is there any possibilities/ perfect patch to make SoftRoCE work fine with higher kernel version (say kernel 3.19+)
>>
>> I hereby attached the wireshark output for ib_read_bw (when higher version acts as a client and 3.0.0+ acts as a server) and the patch i have applied as an attachment
>>
>> the crash in the server is as follows
>>
>> root at ls2085aqds:/etc/libibverbs.d# rping -s
>> ------------[ cut here ]------------
>> WARNING: CPU: 0 PID: 810 at kernel/softirq.c:146 __local_bh_enable_ip+0x84/0xc0()
>> Modules linked in: ib_rxe_net ib_rxe rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr ipv6
>> CPU: 0 PID: 810 Comm: kworker/0:1 Not tainted 3.16.0-Layerscape2-SDK+g330dd4b #20
>> Workqueue: ib_cm cm_work_handler [ib_cm]
>> Call trace:
>> [<ffffffc000088084>] dump_backtrace+0x0/0x12c
>> [<ffffffc0000881c0>] show_stack+0x10/0x1c
>> [<ffffffc000552fa4>] dump_stack+0x74/0xc4
>> [<ffffffc0000a9b8c>] warn_slowpath_common+0x84/0xac
>> [<ffffffc0000a9c78>] warn_slowpath_null+0x14/0x20
>> [<ffffffc0000ae3d0>] __local_bh_enable_ip+0x80/0xc0
>> [<ffffffc00049fccc>] __dev_queue_xmit+0x1cc/0x408
>> [<ffffffc00049ff14>] dev_queue_xmit+0xc/0x18
>> [<ffffffbffc11b620>] send_finish+0x34/0x40 [ib_rxe_net]
>> [<ffffffbffc11b6b8>] send+0x8c/0xec [ib_rxe_net]
>> [<ffffffbffc10933c>] $x+0x318/0x334 [ib_rxe]
>> [<ffffffbffc109be0>] $x+0xa8/0x120 [ib_rxe]
>> [<ffffffbffc109d54>] rxe_run_task+0x4c/0x90 [ib_rxe]
>> [<ffffffbffc1093b4>] arbiter_skb_queue+0x5c/0x8c [ib_rxe]
>> [<ffffffbffc0fffd4>] rxe_requester+0x83c/0xddc [ib_rxe]
>> [<ffffffbffc109be0>] $x+0xa8/0x120 [ib_rxe]
>> [<ffffffbffc109d54>] rxe_run_task+0x4c/0x90 [ib_rxe]
>> [<ffffffbffc104e0c>] rxe_post_send+0x80/0x40c [ib_rxe]
>> [<ffffffbffc097068>] ib_send_mad+0x288/0x454 [ib_mad]
>> [<ffffffbffc09759c>] ib_post_send_mad+0x190/0x544 [ib_mad]
>> [<ffffffbffc0b075c>] ib_send_cm_rej+0xd0/0x194 [ib_cm]
>> [<ffffffbffc0b19b8>] cm_destroy_id+0x188/0x300 [ib_cm]
>> [<ffffffbffc0b1e80>] cm_process_work+0x154/0x17c [ib_cm]
>> [<ffffffbffc0b2600>] cm_req_handler+0x758/0x978 [ib_cm]
>> [<ffffffbffc0b28ec>] cm_work_handler+0xcc/0x1584 [ib_cm]
>> [<ffffffc0000c0918>] process_one_work+0x114/0x354
>> [<ffffffc0000c12f8>] worker_thread+0x13c/0x500
>> [<ffffffc0000c7034>] kthread+0xd0/0xe8
>> ---[ end trace be7a1b95934c8f03 ]---
>>
>>
>> with warm regards
>> Dinesh.K.B
>> Software Engineer
>>
>>
>> Cell : +91 9944456867 | Skype : dinesh_kb93

+ Kamal