[ofw] Expose a vendor defined device in ibbus?
Leonid Keller
leonid at mellanox.co.il
Thu Jan 29 09:39:53 PST 2009
See inline
> -----Original Message-----
> From: Alex Estrin [mailto:alex.estrin at qlogic.com]
> Sent: Thursday, January 29, 2009 6:50 PM
> To: Leonid Keller; Deepak Gupta (Contractor - GS Labs)
> Cc: ofw at lists.openfabrics.org
> Subject: RE: Re: [ofw] Expose a vendor defined device in ibbus?
>
> Hi Leonid,
> Please see some notes inline.
>
> Thanks,
> Alex.
>
> > -----Original Message-----
> > From: ofw-bounces at lists.openfabrics.org
> > [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of
> Leonid Keller
> > Sent: Thursday, January 29, 2009 8:35 AM
> > To: Deepak Gupta (Contractor - GS Labs)
> > Cc: ofw at lists.openfabrics.org
> > Subject: RE: ***SPAM*** Re: [ofw] Expose a vendor defined device in
> > ibbus?
> >
> > Having an application, configuring IOC fabric means that you can't
> > reveal and work with remote disks at boot time.
>
> So far we could consider configuration only for netowrking
> IOCs (Qlogic VNIC).
>
> > Having a configuration file means additional maintance/deployment
> > work. Changing of the may require fixing of the node images.
>
> Not sure I'm following you. Would you please explain?
After full installation they used to make an image of the node as a
backup or for remote bootstrapping of the node.
With dynamic discovery you can use the same node image after changing
IOC fabric.
With static one (configuration file) you have to change the file on the
node and change the image.
Maybe one can solve by booting the image and then adding updated
configuration file to the node.
>
> > What "failover" are you expecting from virtual adapters,
> running over
> > a physical one ?
> > If the latter gets stuck, all of them are dead. :( [I'm
> sure missing
> > here something. Could you elaborate more on usage of these virtual
> > adapters ? TIA]
>
> QLogic EVIC gateway has 2 10G Eth ports mapped to Infiniband
> as 2 IOCs.
> EVIC allows to create and maintain hundreds of virtual ports
> mapped to any of two IOCs.
> On a host side ideally each interface( virtual adapter)
> should be able to maintain two configurable paths
> primary(active) and secondary(standby) to any EVIC IOC on a fabric.
> Each path can be mapped independently using any of path's
> keys available, like:
> -IOC location (chassis slot number, IOC number. Ideal for a
> hot swap scenarios).
> -IOC GUID
> -Local host HCA and port number can be specified in addition
> to previous two for further desirable path restrictions or
> failover scenarios.
> Besides, each path can be specified for any number of virtual
> interfaces created on a host, and total number of virtual
> adapters created/mapped is limited by number of EVICs on a
> fabric and it's capacities only.
> Paths configured should also persist across boot.
>
> Current IOC device creation implied limitations to the VNIC
> implementation:
> - one interface per IOC ( VNIC driver maintains two paths per
> interface internally
> and it is not configurable, both paths mapped to the
> same IOC through local HCA ports).
> - is tied to IOC GUID discovered, so all interfaces will
> disappear along with network configuration if IOU removed
> from a fabric.
> if same or other IOU is reinserted to the same chasis slot
> later, network config should be setup again.
> Not suitable for a hot swap or failover-failback scenarios.
>
> > BTW, if you are going to create IOC devices (from application
> > but) on base of ibiou.sys discovery, why wouldn't you do the same
> > without application ?
> > Let ibiou.sys creates IOC per HCA for every discovered device...
> >
> >
> > ________________________________
> >
> > From: mailmeatdkg at gmail.com
> > [mailto:mailmeatdkg at gmail.com] On Behalf Of Deepak Gupta
> > Sent: Thursday, January 29, 2009 2:47 PM
> > To: Leonid Keller
> > Cc: ofw at lists.openfabrics.org
> > Subject: Re: ***SPAM*** Re: [ofw] Expose a vendor
> defined device
> > in ibbus?
> >
> >
> > Leonid,
> >
> > I understand you might be busy with some other stuff too.
> >
> > As I already said, this was just a first draft patch
> and there
> > are no intentions to move it into the svn repository.
> > I am sorry, I guess I misled you by asking the
> question "whether
> > above patch is acceptable".
> >
> > This patch was just to show a use case of having
> multiple child
> > devices associated with a IOC instead of having a single
> > child PDO for each IOC.
> >
> > Plan is to provide a IOCTL interface in ibiou for creation of
> > child devices and all device information will be passed
> through device
> > IOCTLs (and not INF file, in this first draf patch I used INF
> > file for just POC). So no one has to know the IOC parameters at the
> > time of installation.
> >
> > It indeed raise the need of a user application which can scan
> > the fabric and provide list the of reachable IOCs
> (particularly Port
> > to IOC path, like ibsrpdm and ib_qlgc_vnic_query tools on linux do).
> > I am thinking of providing a IOCTL in ibiou to return
> the list
> > of reachable IOCs.
> > Having a list of reachable IOCs with the user, now user can
> > write a configuration file describing the child devices (or
> a GUI to
> > enter the child device descriptions). This user application
> will parse
> > the configuration file and will create the device interfaces.
> > Also it will record the all these descriptions into
> registry so
> > that from the next boot, ibiou will create the devices without the
> > intervention of user app.
> >
> > >> - having configured one node, one has to repeate
> > all this work with all the others or to prepare and clone the image;
> >
> > Once a node has been configured with a config file,
> that can be
> > used on other nodes too.
> >
> > >> - if a new IOC has been added, one has to go
> > through all the nodes and to change the Registries. It may also
> > require reimaging of all the nodes;
> >
> > If a new IOC has been added, then yes he would need to make
> > entries into config file and then user app will take care of rest.
> >
> > >> - what will happen during the WINOF package
> > upgrade ? We don't want user to repeat all this
> configuration work, do
> > we ?
> >
> > We will have to provide a versioning on above
> mentioned device
> > IOCTLs interface of ibiou. Minimal release changes will not
> affect the
> > config file structure and hence from a user perspective
> > reconfiguration will not be required.
> >
> > This User Application will help create multiple child devices
> > for ULP's (SRP/VNIC) and hence much more failover
> configurations for
> > the user.
> >
> > My basic question is if we agree to above then I can
> move ahead
> > for implmenting above mentioned device IOCTLs in ibiou
> > and also start doing work on the developing the user
> application
> > to do this.
> >
> > Please let me know if you have any further queries on it.
> >
> > Regards
> > Deepak
> >
> >
> >
> > On Thu, Jan 29, 2009 at 3:27 PM, Leonid Keller
> > <leonid at mellanox.co.il> wrote:
> >
> >
> > Hi Deepak,
> >
> > Sorry for that late reaction. Could not find time
> > beforehand to look into it... :(
> >
> > I'm not sure, whether the advantages of your
> patch are
> > worth the deployment problems that it can cause (IMO).
> > Here are several ones at the first look:
> > - one has to know all the IOC parameters
> at the time
> > of installation;
> >
> >
> >
> >
> > And what if they are yet unknown
> (IOCs still not
> > purchased) or unknown to the guy (he doesn't know what IOC
> at all is)
> > ? :(
> > - one has to enter IOC parameters during the
> > installation in some way;
> > Changing INF file is not a convenient
> way. One
> > has to provide some GUI to edit the parameters
> > during- and the Registry after the installation;
> > - having configured one node, one has to
> repeate all
> > this work with all the others or to prepare and clone the image;
> > - if a new IOC has been added, one has to
> go through
> > all the nodes and to change the Registries. It may also require
> > reimaging of all the nodes;
> > - what will happen during the WINOF
> package upgrade
> > ? We don't want user to repeat all this configuration work, do we ?
> >
> > What do you think ?
> >
> > ________________________________
> >
> > From: ofw-bounces at lists.openfabrics.org
> > [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Deepak Gupta
> > Sent: Tuesday, January 27, 2009 5:19 PM
> > To: ofw at lists.openfabrics.org
> > Subject: ***SPAM*** Re: [ofw] Expose a vendor
> > defined device in ibbus?
> >
> >
> >
> > Since I didn't receive any concrete response
> > from the list about the patch, I still have following questions
> > unanswered:--
> >
> > 1) I would like to know who is
> current owner of
> > ibiou and get a feedback on whether above patch is acceptable.
> > Or Do you suggest some modifications on it ?
> >
> > 2) If we are ok with this patch then
> what time
> > frame should we set for it. Should we target it for coming
> 2.1 release
> > or for post 2.1 releases?
> >
> > 3) Also, I want to move ahead for
> implementing
> > IOCTL layer in ibiou for creation of child devices and IOC
> listings on
> > user request. Is it acceptable to implement this IOCTL layer for
> > device creations and IOC listings in ibiou and you suggest
> to achieve
> > the same through some other means?
> >
> >
> > I want to reiterate the purpose of
> this patch:--
> >
> > This patch is for creating multiple
> PDO's for a
> > HCA to IOC path as configured by the user in INF file.
> > It will help in having multiple sessions with
> > the IOC and hence will give some user configurations for fail over.
> >
> > Behavior change from current ibiou
> > implementation:--
> >
> > Currently ibiou simply discovers the IOCs and
> > creates child devices for the IOCs discovered.
> > This patch will create multiple child devices
> > for HCA to IOC paths as configured by the user in INF file.
> > If there are no child device
> configurations for
> > a particular IOC then no devices will be created for that IOC and
> > hence will not appear in device manager's device tree.
> >
> > This is just a first draft patch to show the
> > usability and we have plans to implement IOCTLs in ibiou for child
> > device creations and for listing the reachable IOC PATHs so that a
> > user application can be written which can list the IOCs and
> local HCAs
> > to the user and user can send IOCTLs to create devices for the ULPs.
> >
> >
> > Regards
> > Deepak
> >
> >
> >
> >
> >
> > On Tue, Jan 27, 2009 at 7:09 PM, Deepak Gupta
> > <deepak.gupta at qlogic.com> wrote:
> >
> >
> > Forgot to include the list.
> >
> > Regards
> > Deepak
> >
> >
> > ---------- Forwarded message
> ----------
> > From: Deepak Gupta
> > <deepak.gupta at qlogic.com>
> > Date: Sat, Jan 24, 2009 at 5:07 PM
> > Subject: Re: [ofw] Expose a vendor
> > defined device in ibbus?
> >
> > To: Fab Tillier
> > <ftillier at windows.microsoft.com>
> >
> >
> > Fab, Nice to see your response.
> > Please see below.
> >
> >
> > On Fri, Jan 23, 2009 at 11:09 PM, Fab
> > Tillier <ftillier at windows.microsoft.com> wrote:
> >
> >
> > I don't have time to look at
> > this in depth, but I can tell you from past experience that
> having a
> > single PDO for an IOC leads to trouble if you ever have
> multiple HCAs
> > in the system. If you have a miniport driver (like NDIS or
> StorPort),
> > the port driver takes care of DMA mappings of user-provided buffers.
> > That DMA mapping goes down to the PCI driver for the
> particular HCA,
> > so you could potentially have a mapping that isn't valid for one of
> > the multiple HCAs in the system. It's best IMO to have a
> PDO per IOC
> > per HCA (this allows automatic path migration to work in multi-port
> > HCAs), and push failover between IOCs to a higher level (LBFO/MPIO).
> >
> >
> > I had the same concern while
> > implementing the patch.
> > So I tried to dig into the stack till
> > h/w drivers and it came to me that all DMA mappings are being
> > handled by PCI driver.
> > So this patch creates PDO's
> per IOC per
> > HCA.
> > User can specify the child device
> > descriptions in INF file (later on we can think of having a IOCTL
> > interface in ibiou) and HCA to IOC path.
> > ibiou will create child PDO's only if
> > HCA to IOC path is present.
> >
> > Any more suggestions are welcome!!!
> >
> > Regards
> > Deepak
> >
> >
> >
> >
> >
> >
> >
> > -Fab
> >
> >
> >
> > From:
> > mailmeatdkg at gmail.com [mailto:mailmeatdkg at gmail.com] On Behalf Of
> > Deepak Gupta
> > Sent: Friday, January
> 23, 2009
> > 6:41 AM
> > To: James Yang
> > Cc: Leonid Keller;
> Fab Tillier;
> > ofw at lists.openfabrics.org
> >
> > Subject: Re: [ofw] Expose a
> > vendor defined device in ibbus?
> >
> >
> >
> >
> >
> >
> >
> > Did any one get the chance to
> > have a look at the patch?
> >
> > I am really concerned
> with the
> > tight coupling of representing each single IOC as a single PDO.
> >
> > Regards
> > Deepak
> >
> >
> >
> >
> > On Wed, Jan 21, 2009 at
> > 5:55 PM, Deepak Gupta <deepak.gupta at qlogic.com> wrote:
> >
> > All,
> >
> > Based on James patch
> of creating
> > user defined devices tied to local HCA ports, I have
> created a first
> > draft patch attached herewith
> > to allow "ibiou" to
> create user
> > defined devices which will associate with IOC paths.
> >
> > This patch does the
> following:-
> >
> > Now ibiou will not
> create child
> > devices on it's own when it discovers a IOC.
> > Instead it will create child
> > devices only when it is instructed to do so.
> > For the sake of this patch,
> > currently it uses name/description of the devices hard
> coded into INF
> > file of ibiou driver.
> > Later on we can implement a
> > IOCTL to pass this information to the driver (more below)
> >
> > It will make a listing of
> > devices to be created in it's DriverEntry routine.
> > When it will get
> IOC_PNP_ADD pnp
> > events, it will check in device list prepare
> > earlier. If it finds
> > device(s) in it's list which corresponds to the new IOC
> > discovered then it
> will create
> > those child devices and a similar mechanism
> > will happen when
> ibiou will get
> > IOC_PNP_ADD event.
> >
> > This more of a
> initial patch for
> > giving an insight into what we want to achieve.
> > On a longer run we
> are thinking
> > of providing a IOCTL support in ibiou driver for creating child
> > devices (like vnic/srp/etc) and
> > differnet vendor specific
> > applications can create those devices and have their driver
> loaded on
> > top of that.
> >
> > Following is the advantage of
> > this functionality:-
> >
> > Earlier there was only one
> > device created per IOC on host side.
> > Now there can be multiple
> > devices on host side per IOC as configured in the registry.
> > It will be many to one
> > function i.e there can be many devices on host side which
> will target
> > to the same IOC.
> > It will be like having
> > different sessions with the same IOC.
> >
> > Please have a look at
> it and let
> > me know of your comments/suggestions/feeback.
> >
> > NOTE--> Device
> descriptions in
> > the INF file of this patch creates devices that corresponds
> to a IOC
> > PATH (CAGUID and IOCGUID).
> > Those values
> are local
> > to my machine and you should change it according to your fabric.
> >
> > Regards
> > Deepak
> >
> >
> >
> >
> > On Tue, Jan 20, 2009 at
> > 1:37 AM, James Yang <jyang at xsigo.com> wrote:
> >
> > The proposal
> is to use
> > registry key to define vendor devices, and the registry key
> is global
> > to the driver. The assumption is that multiple HCA cards
> will have the
> > same vendor defined devices.
> > It cannot support one HCA with vendor-A device, and the
> other HCA with
> > vendor-B device, in the same system. At default all hcas will have
> > IPoIB as child devices.
> >
> >
> >
> > Thanks,
> >
> > James
> >
> >
> >
> > ________________________________
> >
> > From: Leonid Keller
> > [mailto:leonid at mellanox.co.il]
> > Sent: Sunday, January
> > 18, 2009 5:57 AM
> > To: James
> Yang; Deepak
> > Gupta; Fab Tillier
> >
> >
> > Cc:
> > ofw at lists.openfabrics.org
> > Subject: RE:
> > [ofw] Expose a vendor defined device in ibbus?
> >
> >
> >
> > After first
> > look: why did you put the list of the created devices into Globals
> > (and not, say, FDO) ?
> >
> > How it will work for
> > multi-home machine ? (several HCA cards)
> >
> > Would anybode like to
> > check it for various partition keys ?
> >
> >
> >
> > ________________________________
> >
> > From:
> > ofw-bounces at lists.openfabrics.org
> > [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of James Yang
> > Sent:
> > Tuesday, January 06, 2009 10:11 PM
> > To:
> > Deepak Gupta; Fab Tillier
> > Cc:
> > ofw at lists.openfabrics.org
> >
> > Subject: RE: [ofw] Expose a vendor defined device in ibbus?
> >
> > Hi,
> >
> >
> >
> > Please review
> > the patch to create user defined devices by reading from
> registry. By
> > default there is only one IpoIB device enabled in
> mlx4_hca.inx file.
> > This patch will only work for ConnectX.
> >
> >
> >
> > The
> paritition
> > key if set to default to FFFF, I didn't test on other
> value. And the
> > Ioctl part to add partition key may also need to be verified.
> >
> >
> >
> > Thanks,
> >
> > James
> >
> >
> >
> >
> >
> > ________________________________
> >
> > From:
> > mailmeatdkg at gmail.com [mailto:mailmeatdkg at gmail.com] On Behalf Of
> > Deepak Gupta
> > Sent:
> > Monday, January 05, 2009 10:51 PM
> > To:
> Fab Tillier
> > Cc:
> > Tzachi Dar; James Yang; ofw at lists.openfabrics.org
> >
> > Subject: Re: [ofw] Expose a vendor defined device in ibbus?
> >
> >
> >
> > Have a
> > gr8 New Year to all the members!!!.
> >
> > Do you we any
> > updates on "vendor defined device in ibbus"?
> >
> > I wanted to
> > create multiple vnic interfaces irrespective of number of reachable
> > IOCs.
> >
> >
> > Currently I am creating vnic child devices on "root" bus.
> > Inside VNIC
> > driver, I am looking for arrival GUID_IB_AL_INTERFACE and hence
> > contacting the IBAL.
> > But
> since vnic
> > devices are root enumerated, VNIC driver gets loaded very
> earlier in
> > boot phase (Before "Extended Base"
> > group to which IB Stack drivers belong) and hence VNIC device
> > interfaces are not getting initialized properly.
> >
> > If we are
> > providing a vendor defined device functionality in ibbus in near
> > future, then it would be worth for me to wait for it.
> >
> > Can any one
> > please comment on this.
> >
> > Regards
> > Deepak
> >
> > On
> Mon, Dec 22,
> > 2008 at 1:12 PM, Deepak Gupta <deepak.gupta at qlogic.com> wrote:
> >
> > All,
> >
> > I came across
> > one more question in my mind which are I think is not clear to me
> > after reading the whole thread.
> >
> > In new design
> > being discussed:-
> > Are we making
> > sure that we can have more than one child devices
> configured for the
> > same IOC.
> >
> > Currently, there is one child device created per IOC discovered.
> >
> >
> Having more than
> > one child device configured for same IOC is a requirement if a user
> > wants two different ULP interfaces to be created on host side.
> >
> > Consider a case in which a host is connected to a single
> IOC and IOC
> > is connected to a ethernet network via switch.
> > If
> there are two
> > different IP subnets then there is a requirement of two different
> > Ethernet interfaces on the host side too.
> >
> > Please let me
> > know if you need more clarification of my question.
> >
> > Regards
> > Deepak
> >
> >
> >
> > On
> Sat, Dec 20,
> > 2008 at 2:19 PM, Deepak Gupta <deepak.gupta at qlogic.com> wrote:
> >
> > Please see
> > below.
> >
> > Regards
> > Deepak
> >
> > On
> Sat, Dec 20,
> > 2008 at 12:42 AM, Fab Tillier
> <ftillier at windows.microsoft.com> wrote:
> >
> > > On Wed, Dec
> > 17, 2008 at 11:43 PM, Fab Tillier
> > >
> > <ftillier at windows.microsoft.com> wrote:
> >
> > >> Are there
> > other properties that I have missed that are needed?
> > >
> > > We
> need a way
> > in which devices created should be configured for
> > > failovers (
> > ULPs like VNIC, SRP need more configurable failovers).
> > > Looking at
> > IBAL's code it create the devices based on the reachable
> > >
> IOC's and thus
> > failover's are possible across the HCA/ports and not
> > > across two
> > different IOCs.
> > >
> Users can have
> > a case in which two different IOCs connected to same
> > > physical
> > network/storage (redundancy is provided for high
> > availability)
> > > and want a
> > failover across the IOCs.
> >
> > This would be
> > done via LBFO for network devices, and MPIO for storage devices. I
> > think having the bus driver report a single IOC that really maps to
> > two IOCs on the fabric is asking for management problems. Further,
> > LBFO/MPIO can provide failover between different device
> types, so the
> > failover devices don't have to be identical HW.
> >
> >
> > I don't know
> > about how MPIO works. But for LBFO, BundleID param will have to be
> > included in extended params then so that user gets the freedom of
> > bundling different failover configurations.
> >
> >
> >
> >
> > -Fab
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > ofw mailing list
> >
> > ofw at lists.openfabrics.org
> >
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > ofw mailing list
> > ofw at lists.openfabrics.org
> >
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > ofw mailing list
> > ofw at lists.openfabrics.org
> >
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
> >
> >
> >
> >
>
More information about the ofw
mailing list