[Users] SR-IOV -- VF ports access to PF network? -- access to storage in vm

[Hal Rosenstock]

On Wed, Sep 13, 2017 at 2:06 PM, Simon Guilbault <
simon.guilbault at calculquebec.ca> wrote:

> Hi,
>
> SR-IOV can be used to pass a subset of a IB card directly to the VM. The
> VM will have RDMA access to the IB subnet without being able to start a
> subnet manager and access some of the Infiniband management features. I
> havent tested features in depth but a Lustre client will work with RDMA
> without any modification compared to a bare metal node. The VM will be also
> able to set any address for IPoIB, theres do not seem to be a filter like a
> virtualized Ethernet bridge.
>
> Check if SR-IOV is enabled on the host and the card, this might require to
> flash the card to activate theses virtual functions. You might need to add
> a kernel parameter and enable a feature inside the BIOS.
>
> Once SR-IOV is working, multiple virtual function will be listed for a card
> $ lspci | grep -i mellanox
> 01:00.0 Network controller: Mellanox Technologies MT27500 Family
> [ConnectX-3]
> 01:00.1 Network controller: Mellanox Technologies MT27500 Family
> [ConnectX-3 Virtual Function]
> 01:00.2 Network controller: Mellanox Technologies MT27500 Family
> [ConnectX-3 Virtual Function]
> 01:00.3 Network controller: Mellanox Technologies MT27500 Family
> [ConnectX-3 Virtual Function]
> [...]
>
> If using Virsh and KVM, modify a VM .xml with something like this:
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </hostdev>
>
> You can make it work with Openstack by adding theses line in nova.conf
> pci_alias={ "vendor_id":"15b3","product_id":"1004", "name": "ib"}
> pci_passthrough_whitelist =  [{ "vendor_id":"15b3","product_id":"1004"}]
>
> The compute flavor need a bit of metadata to bind a IB card to the VM.
> Nova will select a virtual interface on the hypervisor, this completely
> bypass any control done by Neutron.
> pci_passthrough:alias = ib:1
>
> The VM will see a standard infiniband interface
> # ip addr
> [...]
> 3: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state
> UP qlen 256
>     link/infiniband 80:[...] brd 00:[...]
>     inet 10.x.x.x/16 brd 10.225.255.255 scope global ib0
>
> # lspci | grep -i mellanox
> 00:06.0 Network controller: Mellanox Technologies MT27500/MT27520 Family
> [ConnectX-3/ConnectX-3 Pro Virtual Function]
>
> Some management commands will not work inside a VM, but will work in the
> hypervisor:
> [root at vm ~]# perfquery -H
> ibwarn: [865] mad_rpc: _do_madrpc failed; dport (Lid 1504)
> perfquery: iberror: failed: classportinfo query
>
>
I think perfquery with -G option may work in VM as it adds the needed GRH.

[root at hypervisor ~]# perfquery -H
> # Port counters: Lid 1504 port 1 (CapMask: 0x1600)
> PortSelect:......................1
> [...]
>
> On Wed, Sep 13, 2017 at 1:39 PM, Kevin Abbey <kevina at oarc.rutgers.edu>
> wrote:
>
>> Hi,
>>
>> I'm new to IB pass-through and SR-IOV.   We're looking to connect to
>> storage on a physical IB subnet via rdma from a virtual IB port in a vm.
>> I'm guessing that IB router may be possible or RoCE is needed?
>>
>> Is this possible?
>>
>> I haven't found a description of how to do this if it is possible.
>>
>> Thanks for any references or comments.
>> Kevin
>>
>> --
>> Kevin Abbey
>> Systems Administrator
>>
>> Office of Advanced Research Computing (OARC)
>> Rutgers, The State University of New Jersey
>> http://oarc.rutgers.edu/
>>
>> Telephone: (848) 445-5263
>> Email: kevina at oarc.rutgers.edu
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.openfabrics.org
>> http://lists.openfabrics.org/mailman/listinfo/users
>>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20170913/a0d58206/attachment.html>