[Users] SR-IOV -- VF ports access to PF network? -- access to storage in vm

[Simon Guilbault]

Hi,

SR-IOV can be used to pass a subset of a IB card directly to the VM. The VM
will have RDMA access to the IB subnet without being able to start a subnet
manager and access some of the Infiniband management features. I havent
tested features in depth but a Lustre client will work with RDMA without
any modification compared to a bare metal node. The VM will be also able to
set any address for IPoIB, theres do not seem to be a filter like a
virtualized Ethernet bridge.

Check if SR-IOV is enabled on the host and the card, this might require to
flash the card to activate theses virtual functions. You might need to add
a kernel parameter and enable a feature inside the BIOS.

Once SR-IOV is working, multiple virtual function will be listed for a card
$ lspci | grep -i mellanox
01:00.0 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3]
01:00.1 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3 Virtual Function]
01:00.2 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3 Virtual Function]
01:00.3 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3 Virtual Function]
[...]

If using Virsh and KVM, modify a VM .xml with something like this:
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </hostdev>

You can make it work with Openstack by adding theses line in nova.conf
pci_alias={ "vendor_id":"15b3","product_id":"1004", "name": "ib"}
pci_passthrough_whitelist =  [{ "vendor_id":"15b3","product_id":"1004"}]

The compute flavor need a bit of metadata to bind a IB card to the VM. Nova
will select a virtual interface on the hypervisor, this completely bypass
any control done by Neutron.
pci_passthrough:alias = ib:1

The VM will see a standard infiniband interface
# ip addr
[...]
3: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state
UP qlen 256
    link/infiniband 80:[...] brd 00:[...]
    inet 10.x.x.x/16 brd 10.225.255.255 scope global ib0

# lspci | grep -i mellanox
00:06.0 Network controller: Mellanox Technologies MT27500/MT27520 Family
[ConnectX-3/ConnectX-3 Pro Virtual Function]

Some management commands will not work inside a VM, but will work in the
hypervisor:
[root at vm ~]# perfquery -H
ibwarn: [865] mad_rpc: _do_madrpc failed; dport (Lid 1504)
perfquery: iberror: failed: classportinfo query

[root at hypervisor ~]# perfquery -H
# Port counters: Lid 1504 port 1 (CapMask: 0x1600)
PortSelect:......................1
[...]

On Wed, Sep 13, 2017 at 1:39 PM, Kevin Abbey <kevina at oarc.rutgers.edu>
wrote:

> Hi,
>
> I'm new to IB pass-through and SR-IOV.   We're looking to connect to
> storage on a physical IB subnet via rdma from a virtual IB port in a vm.
> I'm guessing that IB router may be possible or RoCE is needed?
>
> Is this possible?
>
> I haven't found a description of how to do this if it is possible.
>
> Thanks for any references or comments.
> Kevin
>
> --
> Kevin Abbey
> Systems Administrator
>
> Office of Advanced Research Computing (OARC)
> Rutgers, The State University of New Jersey
> http://oarc.rutgers.edu/
>
> Telephone: (848) 445-5263
> Email: kevina at oarc.rutgers.edu
>
> _______________________________________________
> Users mailing list
> Users at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20170913/5968c54f/attachment.html>