[Openib-windows] [RFC v2] Kernel Async Verbs

Tzachi Dar tzachid at mellanox.co.il
Mon Nov 7 00:58:51 PST 2005


Hi Fab,

As far as I remember we have agreed that the IP's shouldn't include IRPs,
but a different completion mechanism.
As a result I'm quite confused to see that all your API's do contain IRP's.

Please remember that using an IRP on each function will:
1) Make our code much more complicated.
2) Make a considerable performance hit to all our API's.
3) Will make our code very different from all other windows software that I
know of. As an example, look at the ZwWriteFile function. Although this
function can currently only be called from passive level, it's API is
prepared for assync transfer, but still doesn't use IRP's
4) Will make it impossible to share the same header files from user mode and
kernel mode (the way that IRP's look from user mode is by an overlapped
struct). By the way, taking into account the differences between user mode
and kernel mode, it is not clear if we want to do this in any way.
5) IRP's are harder to debug. Since IRP's are maintined in the operating
system opaque structures, finding where an IRP was lost, is somewhat harder.

I believe that since in any case, we will be using wrapper functions around
the IRPs based functions, we should probably write the wrapper function
first.

In any case, if what I wrote isn't enough, I believe that the best way to
continue this discussion is to try and take one of these functions,
implement it in some trivial way and see the results. Once we do this in
both ways, we can look at the code that is using the functions, measure the
performance and see how we want to continue from here.

Thoughts?

Thanks
Tzachi



Since working with 
>-----Original Message-----
>From: Fab Tillier [mailto:ftillier at silverstorm.com]
>Sent: Monday, November 07, 2005 10:20 AM
>To: openib-windows at openib.org
>Subject: [Openib-windows] [RFC v2] Kernel Async Verbs
>
>Folks,
>
>I spent a little time over the weekend fixing up the headers for the new
>kernel-level verbs API.  Code will now compile with both these and the
>existing
>headers included.
>
>As a refresher, the goals of this new API are:
>- Allow all verb calls to work at IRQL <= DISPATCH_LEVEL
>- Leverage capabilities of the OS
>	- Use IRPs for completion notifications
>	- Optimize memory registration APIs to use native structures
>- Look and feel like native Windows APIs
>
>This API does not change the access layer API.  For now, this API would be
>used
>solely between the access layer and the HCA driver, to be migrated to wider
>kernel use at a later point in time.  The files ibdef.h and ibstatus.h are
>intended to be sharable with user-mode at some later point in time.
>
>I haven't figured out what the SRQ API will look like yet, but will add
>that
>next, likely modeled after what's been done on the Linux side of things.
>
>Please let me know if you have comments or questions.
>
>Thanks,
>
>- Fab
>
>Index: inc/iba/ibstatus.h
>===================================================================
>/*
> * Copyright (c) 2005 SilverStorm Technologies.  All rights reserved.
> *
> * This software is available to you under the OpenIB.org BSD license
> * below:
> *
> *     Redistribution and use in source and binary forms, with or
> *     without modification, are permitted provided that the following
> *     conditions are met:
> *
> *      - Redistributions of source code must retain the above
> *        copyright notice, this list of conditions and the following
> *        disclaimer.
> *
> *      - Redistributions in binary form must reproduce the above
> *        copyright notice, this list of conditions and the following
> *        disclaimer in the documentation and/or other materials
> *        provided with the distribution.
> *
> * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> * SOFTWARE.
> *
> * $Id$
> */
>
>
>#pragma once
>
>
>#ifndef _IBSTATUS_
>#define _IBSTATUS_
>
>
>/* Defines IB-specific error values. */
>
>/* Matches FILE_DEVICE_INFINIBAND */
>#define FACILITY_INFINIBAND             0x3B
>
>#define STATUS_IB_INSUFFICIENT_WR       ((NTSTATUS)0xE03B0001L)
>#define STATUS_IB_INSUFFICIENT_SGE      ((NTSTATUS)0xE03B0002L)
>#define STATUS_IB_INVALID_WR_TYPE       ((NTSTATUS)0xE03B0003L)
>#define STATUS_IB_INVALID_STATE         ((NTSTATUS)0xE03B0004L)
>#define STATUS_IB_INVALID_PORT          ((NTSTATUS)0xE03B0005L)
>
>#endif  /* _IBSTATUS_ */
>
>
>Index: inc/iba/ibdef.h
>===================================================================
>/*
> * Copyright (c) 2005 SilverStorm Technologies.  All rights reserved.
> *
> * This software is available to you under the OpenIB.org BSD license
> * below:
> *
> *     Redistribution and use in source and binary forms, with or
> *     without modification, are permitted provided that the following
> *     conditions are met:
> *
> *      - Redistributions of source code must retain the above
> *        copyright notice, this list of conditions and the following
> *        disclaimer.
> *
> *      - Redistributions in binary form must reproduce the above
> *        copyright notice, this list of conditions and the following
> *        disclaimer in the documentation and/or other materials
> *        provided with the distribution.
> *
> * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> * SOFTWARE.
> *
> * $Id$
> */
>
>
>#pragma once
>
>
>#ifndef _IBDEFS_
>#define _IBDEFS_
>
>
>typedef struct _IB_CA   *IB_CA_HANDLE, * __ptr64 IB_CA_HANDLE64;
>typedef struct _IB_PD   *IB_PD_HANDLE, * __ptr64 IB_PD_HANDLE64;
>typedef struct _IB_CQ   *IB_CQ_HANDLE, * __ptr64 IB_CQ_HANDLE64;
>typedef struct _IB_SRQ  *IB_SRQ_HANDLE, * __ptr64 IB_SRQ_HANDLE64;
>typedef struct _IB_QP   *IB_QP_HANDLE, * __ptr64 IB_QP_HANDLE64;
>typedef struct _IB_AH   *IB_ADDR_HANDLE, * __ptr64 IB_ADDR_HANDLE64;
>typedef struct _IB_MR   *IB_MR_HANDLE, * __ptr64 IB_MR_HANDLE64;
>typedef struct _IB_MW   *IB_MW_HANDLE, * __ptr64 IB_MW_HANDLE64;
>
>
>#pragma pack(push, 1)
>typedef __declspec(align(8)) union _IB_GID
>{
>    UCHAR                   Raw[16];
>
>    struct _IB_GID_UNICAST
>    {
>        UINT64              Prefix;
>        UINT64              InterfaceId;
>
>    } Unicast;
>
>    struct _IB_GID_MULTICAST
>    {
>        UCHAR               Header[2];
>        UCHAR               RawGroupId[14];
>
>    } Multicast;
>
>}   IB_GID;
>
>
>typedef __declspec(align(8)) struct _IB_GRH
>{
>    ULONG           VerClassFlow;
>    USHORT          PayloadLength;
>    UCHAR           NextHdr;
>    UCHAR           HopLimit;
>    IB_GID          SGid;
>    IB_GID          DGid;
>
>}   IB_GRH;
>#pragma pack(pop)
>
>
>typedef enum _IB_MTU
>{
>    IB_MTU_256  = 1,
>    IB_MTU_512  = 2,
>    IB_MTU_1024 = 3,
>    IB_MTU_2048 = 4,
>    IB_MTU_4096 = 5
>
>}   IB_MTU;
>
>
>typedef enum _IB_LINK_STATE
>{
>    IB_LINK_NO_CHANGE   = 0,
>    IB_LINK_DOWN        = 1,
>    IB_LINK_INIT        = 2,
>    IB_LINK_ARMED       = 3,
>    IB_LINK_ACTIVE      = 4,
>    IB_LINK_ACT_DEFER   = 5
>
>}   IB_LINK_STATE;
>
>
>typedef enum _IB_LINK_WIDTH
>{
>    IB_LINK_WIDTH_1X    = 1,
>    IB_LINK_WIDTH_4X    = 2,
>    IB_LINK_WIDTH_8X    = 4,
>    IB_LINK_WIDTH_12X   = 8
>
>}   IB_LINK_WIDTH;
>
>
>typedef enum _IB_LINK_SPEED
>{
>    IB_LINK_SPEED_2_5_GBS   = 1,
>    IB_LINK_SPEED_5_GBS     = 2,
>    IB_LINK_SPEED_10_GBS    = 4
>
>}   IB_LINK_SPEED;
>
>
>enum IB_PORT_CAP_FLAGS
>{
>    IB_PORT_CAP_IS_SM                       = (1<<1),
>    IB_PORT_CAP_NOTICE_SUP                  = (1<<2),
>    IB_PORT_CAP_TRAP_SUP                    = (1<<3),
>    IB_PORT_CAP_OPT_IPD_SUP                 = (1<<4),
>    IB_PORT_CAP_APM_SUP                     = (1<<5),
>    IB_PORT_CAP_SL_MAP_SUP                  = (1<<6),
>    IB_PORT_CAP_MKEY_NVRAM                  = (1<<7),
>    IB_PORT_CAP_PKEY_NVRAM                  = (1<<8),
>    IB_PORT_CAP_LED_INFO_SUP                = (1<<9),
>    IB_PORT_CAP_IS_SM_DISABLED              = (1<<10),
>    IB_PORT_CAP_SYS_IMAGE_GUID_SUP          = (1<<11),
>    IB_PORT_CAP_PKEY_SW_EXT_PORT_TRAP_SUP   = (1<<12),
>    IB_PORT_CAP_CM_SUP                      = (1<<16),
>    IB_PORT_CAP_SNMP_TUNNEL_SUP             = (1<<17),
>    IB_PORT_CAP_REINIT_SUP                  = (1<<18),
>    IB_PORT_CAP_DEVICE_MGMT_SUP             = (1<<19),
>    IB_PORT_CAP_VENDOR_CLASS_SUP            = (1<<20),
>    IB_PORT_CAP_DR_NOTICE_SUP               = (1<<21),
>    IB_PORT_CAP_CAP_MASK_NOTICE_SUP         = (1<<22),
>    IB_PORT_CAP_BOOT_MGMT_SUP               = (1<<23),
>    IB_PORT_CAP_LINK_LATENCY_SUP            = (1<<24),
>    IB_PORT_CAP_CLIENT_REG_SUP              = (1<<25)
>};
>
>
>enum _IB_INIT_TYPE
>{
>    IB_INIT_TYPE_NO_LOAD                = 1,
>    IB_INIT_TYPE_PRESERVE_CONTENT       = (1<<1),
>    IB_INIT_TYPE_PRESERVE_PRESENCE      = (1<<2),
>    IB_INIT_TYPE_DO_NOT_RESUSCITATE     = (1<<3)
>};
>
>
>typedef __declspec(align(8)) struct _IB_PORT_ATTR
>{
>    ULONG                   Size;
>
>    /** Combination of IB_PORT_CAP_FLAGS values. */
>    ULONG                   CapabilitiesFlags;
>
>    /** GUID are in network byte order. */
>    UINT64                  PortGuid;
>
>    IB_LINK_STATE           LinkState;
>    IB_LINK_WIDTH           LinkWidth;
>    IB_LINK_SPEED           LinkSpeed;
>    ULONG                   LinkWidthSup;
>    ULONG                   LinkSpeedSup;
>    ULONG                   LinkWidthEnabled;
>    ULONG                   LinkSpeedEnabled;
>    IB_MTU                  Mtu;
>
>    UINT64                  MaxMsgSize;
>
>    USHORT                  NumGids;
>    USHORT                  NumPkeys;
>
>    /** Offsets are from beginning of structure. */
>    ULONG                   NextPortAttrOffset;
>    ULONG                   GidTblOffset;
>    ULONG                   PkeyTblOffset;
>
>    USHORT                  PkeyViolations;
>    USHORT                  QkeyViolations;
>
>    USHORT                  BaseLid;
>    USHORT                  SmLid;
>    USHORT                  MaxVls;
>    UCHAR                   Lmc;
>    UCHAR                   PortNum;
>    UCHAR                   InitTypeReply;
>
>    /** The maximum expected subnet propagation delay to reach any port on
>     * the subnet.  This value also determines the rate at which traps can
>     * be generated from this node.  The value is expressed as
>     *  timeout = 4.096 microseconds * 2^subnet_timeout
>     */
>    UCHAR                   SubnetTimeout;
>    UCHAR                   SmSvcLvl;
>
>}   IB_PORT_ATTR;
>
>
>enum IB_MODIFY_PORT_FLAGS
>{
>    IB_MOD_PORT_SHUTDOWN                = (1<<0),
>    IB_MOD_PORT_INIT_REPLY              = (1<<1),
>    IB_MOD_PORT_RESET_QKEY_VIOLATIONS   = (1<<2)
>};
>
>
>typedef __declspec(align(8)) struct _IB_MODIFY_PORT
>{
>    /** Combination of IB_PORT_CAP_FLAGS */
>    ULONG       SetCapabilites;
>    ULONG       ClearCapabilites;
>    UCHAR       InitTypeReply;
>
>}   IB_MODIFY_PORT;
>
>
>enum IB_CA_CAP_FLAGS
>{
>    IB_CA_CAP_RESIZE_MAX_WR     = (1<<0),
>    IB_CA_CAP_BAD_PKEY_CNTR     = (1<<1),
>    IB_CA_CAP_BAD_QKEY_CNTR     = (1<<2),
>    IB_CA_CAP_AUTO_PATH_MIG     = (1<<3),
>    IB_CA_CAP_CHANGE_PHY_PORT   = (1<<4),
>    IB_CA_CAP_AH_PORT_ENFORCE   = (1<<5),
>    IB_CA_CAP_CURR_QP_STATE_MOD = (1<<6),
>    IB_CA_CAP_SHUTDOWN_PORT     = (1<<7),
>    IB_CA_CAP_INIT_TYPE         = (1<<8),
>    IB_CA_CAP_PORT_ACTIVE_EVENT = (1<<9),
>    IB_CA_CAP_SYS_IMAGE_GUID    = (1<<10),
>    IB_CA_CAP_RC_RNR_NAK_GEN    = (1<<11),
>    IB_CA_CAP_N_NOTIFY_CQ       = (1<<12),
>    IB_CA_CAP_SRQ_RESIZE        = (1<<13)
>};
>
>
>typedef enum _IB_ATOMIC_CAP
>{
>    IbAtomicNone,
>    IbAtomicLocal,
>    IbAtomicGlobal
>
>}   IB_ATOMIC_CAP;
>
>
>typedef __declspec(align(8)) struct _IB_CA_ATTR
>{
>    ULONG                   Size;
>
>    UINT32                  VendorId;
>    UINT16                  DeviceId;
>    UINT16                  DeviceRev;
>    UINT32                  DriverVersion;
>    UINT64                  FwVersion;
>
>    /** GUIDs are in network byte order. */
>    UINT64                  CaGuid;
>    UINT64                  SystemImageGuid;
>
>    /** Byte offset from the beginning of this structure to the first
>    * IB_PORT_ATTR structure for the HCA.
>    */
>    ULONG                   PortAttrOffset;
>    UINT8                   NumPorts;
>
>    UINT8                   MaxQpReadAtomic;
>    UINT8                   MaxReadAtomic;
>    UINT8                   MaxQpInitReadAtomic;
>
>    /** Each bit in PageSizeSupport indicates a 2^n page size supported
>    * by the HCA, where bit locations are 1-based.  The minimum page
>    * size that can be supported is thus 2 bytes, and the largest 2^64.
>    */
>    UINT64                  PageSizeSupport;
>
>    /** Combination of IB_CA_CAP_FLAGS values. */
>    ULONG                   CapabilitiesFlags;
>
>    IB_ATOMIC_CAP           AtomicCap;
>
>    ULONG                   MaxQp;
>    ULONG                   MaxQpWr;
>    ULONG                   MaxQpSge;
>    ULONG                   MaxCq;
>    ULONG                   MaxCqe;
>    ULONG                   MaxPd;
>    ULONG                   MaxAh;
>    ULONG                   MaxMr;
>    UINT64                  MaxMrSize;
>    ULONG                   MaxMw;
>    ULONG                   MaxMcastGrp;
>    ULONG                   MaxMcastQp;
>    ULONG                   MaxMcastQpPerGrp;
>    ULONG                   MaxSrq;
>    ULONG                   MaxSrqWr;
>    ULONG                   MaxSrqSge;
>
>    UINT16                  MaxPKeys;
>
>    UINT8                   LocalAckDelay;
>
>}   IB_CA_ATTR;
>
>
>enum IB_MODIFY_CA_FLAGS
>{
>    IB_MODIFY_CA_SYS_IMAGE_GUID = (1<<0),
>    IB_MODIFY_CA_NODE_DESC = (1<<1)
>};
>
>
>typedef __declspec(align(8)) struct _IB_MODIFY_CA
>{
>    UINT64      SystemImageGuid;
>    CHAR        NodeDesc[64];
>
>}   IB_MODIFY_CA;
>
>
>/****d* Verbs/IB_ASYNC_EVENT
>* NAME
>*   Affiliated and unaffiliated asynchronous event types.
>*
>* DESCRIPTION
>*   This type indicates the reason an async event callback was invoked.
>*
>* SYNOPSIS
>*/
>typedef enum _IB_ASYNC_EVENT
>{
>    IbAeSqError = 1,
>    IbAeSqDrained,
>    IbAeRqError,
>    IbAeCqError,
>    IbAeQpFatal,
>    IbAeQpComm,
>    IbAeQpApm,
>    IbAeLocalFatal,
>    IbAePkeyTrap,
>    IbAeQkeyTrap,
>    IbAeMkeyTrap,
>    IbAePortTrap,
>    IbAeSysimgGuidTrap,
>    IbAeBufOverrun,
>    IbAeLinkIntegrity,
>    IbAeFlowCtrlError,
>    IbAeBkeyTrap,
>    IbAeQpApmError,
>    IbAeWqReqError,
>    IbAeWqAccessError,
>    IbAePortActive,
>    IbAePortDown,
>    IbAeClientReregister,
>    IbAeUnknown     /* ALWAYS LAST ENUM VALUE */
>
>}   IB_ASYNC_EVENT;
>/*
>* VALUES
>*   IbAeSqError
>*       An error occurred when accessing the send queue of the QP.
>*       This event is optional.
>*
>*   IbAeSqDrained
>*       The send queue of the specified QP has completed the outstanding
>*       messages in progress when the state change was requested and, if
>*       applicable, has received all acknowledgements for those messages.
>*
>*   IbAeRqError
>*       An error occurred when accessing the receive queue of the QP.
>*       This event is optional.
>*
>*   IbAeCqError
>*       An error occurred when writing an entry to the CQ.
>*
>*   IbAeQpFatal
>*       A catastrophic error occurred while accessing or processing the
>*       work queue that prevents reporting of completions.
>*
>*   IbAeQpComm
>*       The first packet has arrived for the receive work queue where the
>*       QP is still in the RTR state.
>*
>*   IbAeQpApm
>*       If alternate path migration is supported, this event indicates that
>*       the QP connection has migrated to the alternate path.
>*
>*   IbAeLocalFatal
>*       A catastrophic HCA error occurred which cannot be attributed to any
>*       resource; behavior is indeterminate.
>*
>*   IbAePkeyTrap
>*       A PKEY violation was detected.  This event is optional.
>*
>*   IbAeQkeyTrap
>*       A QKEY violation was detected.  This event is optional.
>*
>*   IbAeMkeyTrap
>*       An MKEY violation was detected.  This event is optional.
>*
>*   IbAePortTrap
>*       A port capability change was detected.  This event is optional.
>*
>*   IbAeSysimgGuidTrap
>*       If the system image GUID is supported, this event indicates that
>the
>*       system image GUID of this HCA has been changed.  This event is
>*       optional.
>*
>*   IbAeBufOverrun
>*       The number of consecutive flow control update periods with at least
>*       one overrun error in each period has exceeded the threshold
>specified
>*       in the port info attributes.  This event is optional.
>*
>*   IbAeLinkIntegrity
>*       The detection of excessively frequent local physical errors has
>*       exceeded the threshold specified in the port info attributes.  This
>*       event is optional.
>*
>*   IbAeFlowCtrlError
>*       An HCA watchdog timer monitoring the arrival of flow control
>updates
>*       has expired without receiving an update.  This event is optional.
>*
>*   IbAeBkeyTrap
>*       An BKEY violation was detected.  This event is optional.
>*
>*   IbAeQpApmError
>*       If alternate path migration is supported, this event indicates that
>*       an incoming path migration request to this QP was not accepted.
>*
>*   IbAeWqReqError
>*       An OpCode violation was detected at the responder.
>*
>*   IbAeWqAccessError
>*       An access violation was detected at the responder.
>*
>*   IbAePortActive
>*       If the port active event is supported, this event is generated
>*       when the link becomes active: IB_LINK_ACTIVE.
>*
>*   IbAePortDown
>*       The link is declared unavailable: IB_LINK_INIT, IB_LINK_ARMED,
>*       IB_LINK_DOWN.
>*
>*   IbAeClientReregister
>*       The SM idicate to client to reregister its SA records.
>*
>*   IbAeUnknown
>*       An unknown error occurred which cannot be attributed to any
>*       resource; behavior is indeterminate.
>*****/
>
>
>typedef enum _IB_WC_STATUS
>{
>    IbWcSuccess,
>    IbWcLocalLenErr,
>    IbWcLocalOpErr,
>    IbWcLocalProtectionErr,
>    IbWcWrFlushedErr,
>    IbWcMwBindErr,
>    IbWcRemAccessErr,
>    IbWcRemOpErr,
>    IbWcRnrRetryErr,
>    IbWcTimeoutRetryErr,
>    IbWcRemInvalidReqErr,
>    IbWcUnmatchedResponse,          /* InfiniBand Access Layer */
>    IbWcCanceled,                   /* InfiniBand Access Layer */
>    IbWcUnknown                     /* Must be last. */
>
>}   IB_WC_STATUS;
>
>
>typedef enum _IB_OP_TYPE
>{
>    IbSend,
>    IbRdmaWrite,
>    IbRdmaRead,
>    IbInterlockedCompareExchange,
>    IbInterlockedAdd,
>    IbBindMw,
>    /** Set value of IB_WC_RECV so consumers can test if a completion is a
>    * receive by testing (Type & IB_WC_RECV).
>    */
>    IbRecv          = (1 << 7),
>    IbRecvRdmaWrite,
>
>}   IB_OP_TYPE;
>
>
>enum IB_WC_FLAGS
>{
>    IB_WC_FLAG_IMMEDIATE    = (1<<0),
>    IB_WC_FLAG_GRH          = (1<<1),
>    /** Upper bits of WC flags are for vendor specific use. */
>    IB_WC_FLAG_VENDOR_MASK  = 0xFFFF0000
>};
>
>
>typedef __declspec(align(8)) struct _IB_WC
>{
>    IB_QP_HANDLE64      hQp;
>    UINT64              WrId;
>    IB_WC_STATUS        Status;
>    IB_OP_TYPE          Type;
>    ULONG               VendorError;
>    ULONG               Length;
>    /** OR'd IB_WC_FLAGS values. */
>    ULONG               Flags;
>    ULONG               ImmediateData;
>
>    struct _WC_UD
>    {
>        ULONG           RemoteQp;
>        USHORT          RemoteLid;
>        USHORT          PkeyIndex;
>        UCHAR           RemoteSl;
>        UCHAR           PathBits;
>
>    }   Ud;
>
>}   IB_WC;
>
>
>enum IB_ACCESS_MASK
>{
>    IB_AC_RDMA_READ     = (0<<1),
>    IB_AC_RDMA_WRITE    = (1<<1),
>    IB_AC_ATOMIC        = (1<<2),
>    IB_AC_LOCAL_WRITE   = (1<<3),
>    IB_AC_MW_BIND       = (1<<4)
>};
>
>
>enum IB_WR_FLAGS
>{
>    IB_WR_FLAG_IMMEDIATE    = (1<<0),
>    IB_WR_FLAG_FENCE        = (1<<1),
>    IB_WR_FLAG_SIGNALED     = (1<<2),
>    IB_WR_FLAG_SOLICITED    = (1<<3),
>    IB_WR_FLAG_INLINE       = (1<<4),
>    IB_WR_FLAG_LOCAL        = (1<<5),   /* Access Layer */
>    /** Upper bits of WC flags are for vendor specific use. */
>    IB_WR_FLAG_VENDOR_MASK  = 0xFFFF0000
>};
>
>
>typedef __declspec(align(8)) struct _IB_SGE
>{
>    VOID* __ptr64               Address;
>    ULONG                       Length;
>    ULONG                       LKey;
>
>}   IB_SGE;
>
>
>#pragma warning(disable:4200)
>typedef __declspec(align(8)) struct _IB_SEND_WR
>{
>    struct _IB_SEND_WR* __ptr64 pNext;
>    UINT64                      WrId;
>    /** Bind, Recv, RecvRdmaWrite not valid types for this struct. */
>    IB_OP_TYPE                  Type;
>    /** Combination of IB_WR_FLAG values. */
>    ULONG                       Flags;
>    ULONG                       ImmediateData;
>
>    union _WR
>    {
>        struct _SEND_UD
>        {
>            IB_ADDR_HANDLE      pAh;
>            ULONG               RemoteQpn;
>            ULONG               RemoteQkey;
>            USHORT              PKeyIndex;
>
>        }   Ud;
>
>        struct _SEND_RDMA
>        {
>            VOID* __ptr64       Address;
>            ULONG               RKey;
>
>        }   Rdma;
>
>        struct _SEND_COMP_XCHG
>        {
>            VOID* __ptr64       Address;
>            UINT64              Exchange;
>            UINT64              Comparand;
>            ULONG               RKey;
>
>        }   CompareExchange;
>
>        struct _SEND_XCHG_ADD
>        {
>            VOID* __ptr64       Address;
>            UINT64              Value;
>            ULONG               RKey;
>
>        }   ExchangeAdd;
>
>    } u;
>    ULONG                       SgeCount;
>    IB_SGE* __ptr64             Sge[];
>
>}   IB_SEND_WR;
>
>
>typedef __declspec(align(8)) struct _IB_RECV_WR
>{
>    struct _IB_RECV_WR* __ptr64 pNext;
>    UINT64                      WrId;
>    ULONG                       SgeCount;
>    IB_SGE* __ptr64             Sge[];
>
>}   IB_RECV_WR;
>#pragma warning(default:4200)
>
>
>typedef __declspec(align(8)) struct _IB_BIND_WR
>{
>    UINT64          WrId;
>    IB_MR_HANDLE    *pMr;
>    /** Combination of IB_WR_FLAG values. */
>    ULONG           Flags;
>    /** Combination of IB_ACCESS_MASK values. */
>    ULONG           AccessMask;
>    void* __ptr64   Addr;
>    ULONG           Length;
>
>}   IB_BIND_WR;
>
>
>enum IB_AH_FLAGS
>{
>    IB_AH_FLAG_GRH      = (1 << 0)
>};
>
>
>typedef __declspec(align(8)) struct _IB_AH_ATTR
>{
>    UCHAR           PortNumber;
>    UCHAR           SvcLvl;
>    USHORT          DLid;
>    /** Combination of IB_AH_FLAG values. */
>    ULONG           Flags;
>    IB_GRH          Grh;
>    UCHAR           StaticRate;
>    UCHAR           PathBits;
>
>}   IB_AH_ATTR;
>
>
>typedef enum _IB_QP_TYPE
>{
>    IbQpTypeReliableConn    = 0,        /* Matches CM REQ transport type */
>    IbQpTypeUnreliableConn  = 1,        /* Matches CM REQ transport type */
>    IbQpTypeUnreliableDgrm  = 3,        /* Purposefully skip RDD type. */
>    IbQpTypeQp0,
>    IbQpTypeQp1,
>    IbQpTypeMad,                        /* InfiniBand Access Layer */
>    IbQpTypeQp0Alias,                   /* InfiniBand Access Layer */
>    IbQpTypeQp1Alias                    /* InfiniBand Access Layer */
>
>}   IB_QP_TYPE;
>
>
>typedef __declspec(align(8)) struct _IB_QP_CREATE
>{
>    IB_QP_TYPE              Type;
>
>    ULONG                   SqDepth;
>    ULONG                   RqDepth;
>    ULONG                   SqSge;
>    ULONG                   RqSge;
>    ULONG                   SqMaxInline;
>
>    IB_CQ_HANDLE            pSendCq;
>    IB_CQ_HANDLE            pRecvCq;
>    IB_SRQ_HANDLE           pSrq;
>
>    BOOLEAN                 SqSignaled;
>
>}   IB_QP_CREATE;
>
>
>typedef enum _IB_QP_STATE
>{
>    IbQpsReset,
>    IbQpsInit,
>    IbQpsRtr,
>    IbQpsRts,
>    IbQpsSqd,
>    IbQpsSqdDraining,
>    IbQpsSqdDrained,
>    IbQpsSqErr,
>    IbQpsError
>
>}   IB_QP_STATE;
>
>
>typedef enum _IB_APM_STATE
>{
>    IbApmMigrated = 1,
>    IbApmRearm,
>    IbApmArmed
>
>}   IB_APM_STATE;
>
>
>enum IB_QP_MOD_FLAGS
>{
>    IB_MOD_QP_ALTERNATE_AV      = (1<<0),
>    IB_MOD_QP_PKEY              = (1<<1),
>    IB_MOD_QP_APM_STATE         = (1<<2),
>    IB_MOD_QP_PRIMARY_AV        = (1<<3),
>    IB_MOD_QP_RNR_NAK_TIMEOUT   = (1<<4),
>    IB_MOD_QP_RESP_RES          = (1<<5),
>    IB_MOD_QP_INIT_DEPTH        = (1<<6),
>    IB_MOD_QP_PRIMARY_PORT      = (1<<7),
>    IB_MOD_QP_ACCESS_CTRL       = (1<<8),
>    IB_MOD_QP_QKEY              = (1<<9),
>    IB_MOD_QP_SQ_DEPTH          = (1<<10),
>    IB_MOD_QP_RQ_DEPTH          = (1<<11),
>    IB_MOD_QP_CURRENT_STATE     = (1<<12),
>    IB_MOD_QP_RETRY_CNT         = (1<<13),
>    IB_MOD_QP_LOCAL_ACK_TIMEOUT = (1<<14),
>    IB_MOD_QP_RNR_RETRY_CNT     = (1<<15)
>};
>
>
>typedef __declspec(align(8)) struct _IB_QP_MODIFY
>{
>    IB_QP_STATE     ReqState;
>
>    union _QpState
>    {
>        struct _QpInit
>        {
>            UCHAR               PrimaryPort;
>            ULONG               Qkey;
>            USHORT              PkeyIndex;
>            ULONG               AccessMask;
>
>        }   Init;
>
>        struct _QpRtr
>        {
>            ULONG               RqPsn;
>            ULONG               DestQp;
>            IB_AH_ATTR          PrimaryAh;
>            UCHAR               RespRes;
>            UCHAR               RnrNakTimeout;
>
>            ULONG               Opts;
>            IB_AH_ATTR          AlternateAh;
>            ULONG               Qkey;
>            USHORT              PkeyIndex;
>            ULONG               AccessMask;
>            ULONG               SqDepth;
>            ULONG               RqDepth;
>
>        }   rtr;
>
>        struct _QpRts
>        {
>            ULONG               SqPsn;
>            UCHAR               RetryCnt;
>            UCHAR               RnrRetryCnt;
>            UCHAR               LocalAckTimeout;
>            UCHAR               InitDepth;
>
>            ULONG               Opts;
>            UCHAR               RnrNakTimeout;
>            IB_QP_STATE         CurrentState;
>            ULONG               Qkey;
>            ULONG               AccessMask;
>            UCHAR               RespRes;
>
>            IB_AH_ATTR          PrimaryAh;
>            IB_AH_ATTR          AlternateAh;
>
>            ULONG               SqDepth;
>            ULONG               RqDepth;
>
>            IB_APM_STATE        ApmState;
>            UCHAR               PrimaryPort;
>            USHORT              PkeyIndex;
>
>        }   rts;
>
>        struct _QpSqd
>        {
>            BOOLEAN             SqdEvent;
>
>        }   sqd;
>
>    }   state;
>
>}   IB_QP_MODIFY;
>
>
>typedef __declspec(align(8)) struct _IB_QP_ATTR
>{
>    ULONG                   AccessMask;
>    USHORT                  PkeyIndex;
>
>    UCHAR                   InitDepth;
>    UCHAR                   RespRes;
>
>    BOOLEAN                 SqSignaled;
>
>    IB_QP_STATE             State;
>    ULONG                   DestQpNum;
>    ULONG                   Qkey;
>
>    ULONG                   SqPsn;
>    ULONG                   RqPsn;
>
>    UCHAR                   primary_port;
>    UCHAR                   alternate_port;
>    IB_AH_ATTR              PrimaryAh;
>    IB_AH_ATTR              AlternateAh;
>    IB_APM_STATE            ApmState;
>
>}   IB_QP_ATTR;
>
>
>#endif /* _IBDEFS_ */
>
>Index: inc/kernel/iba/ibverbs.h
>===================================================================
>/*
> * Copyright (c) 2005 SilverStorm Technologies.  All rights reserved.
> *
> * This software is available to you under the OpenIB.org BSD license
> * below:
> *
> *     Redistribution and use in source and binary forms, with or
> *     without modification, are permitted provided that the following
> *     conditions are met:
> *
> *      - Redistributions of source code must retain the above
> *        copyright notice, this list of conditions and the following
> *        disclaimer.
> *
> *      - Redistributions in binary form must reproduce the above
> *        copyright notice, this list of conditions and the following
> *        disclaimer in the documentation and/or other materials
> *        provided with the distribution.
> *
> * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> * SOFTWARE.
> *
> * $Id$
> */
>
>
>#pragma once
>
>
>#ifndef _IBVERBS_
>#define _IBVERBS_
>
>
>#include <iba/ibdef.h>
>#include <iba/ibstatus.h>
>
>
>/****s* Verbs/CI_UDATA
>* NAME
>*   CI_UDATA
>*
>* DESCRIPTION
>*   Vendor specific structure to facilitate user mode IO
>*
>*   This structure is provided to assist the vendor specific user mode
>*   library to exchange information with its kernel mode driver. The
>*   user mode InfiniBand(tm) Verbs will call the vendor specific
>*   module before a call is made to the kernel mode driver. The kernel mode
>*   driver is expected to know the format and data in the input and output
>*   buffers, and copy any necessary data that must be handed to the user
>*   mode vendor library.
>*
>* PURPOSE
>*   pInputBuffer
>*       UVP input buffer.  May overlap the output buffer.
>*
>*   pOutputBuffer
>*       UVP output buffer.  NULL if no output buffer is provided.
>*       May overlap the input buffer.
>*
>*   InputSize
>*       Size of UVP provided buffer.  Zero if there is no input buffer.
>*
>*   OutputSize
>*       The size of the UVP provided output buffer.  Zero if no output
>*       buffer is provided.  The kernel mode driver sets this to the exact
>*       size that needs to be returned to its user mode counterpart.
>*
>* SOURCE
>*/
>typedef struct _CI_UDATA
>{
>    const VOID* __ptr64 const   pInputBuffer;
>    VOID* __ptr64 const         pOutputBuffer;
>    SIZE_T                      InputSize;
>    SIZE_T                      OutputSize;
>
>} CI_UDATA;
>/******/
>
>
>typedef struct _IB_CA
>{
>    VOID            *Context;
>    LONG volatile   nRef;
>
>}   IB_CA;
>
>
>typedef VOID
>(*PFN_IB_CA_ASYNC_EVENT)(
>    IN              IB_CA* const                pCa,
>    IN              VOID                        *CaContext,
>    IN      const   UCHAR                       PortNumber,
>    IN              IB_ASYNC_EVENT              Event );
>
>
>/****f* Verbs/POPEN_CA
>* DESCRIPTION
>*   Opens the HCA for client use.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*POPEN_CA)(
>    IN              VOID* const                 IfcContext,
>    IN              VOID* const                 Context,
>    IN              PFN_IB_CA_ASYNC_EVENT       CaAsyncEvent OPTIONAL,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   IfcContext
>*       [in] Interface context returned in INTERFACE structure retrieved
>*       using IRP_MN_QUERY_INTERFACE.
>*
>*   Context
>*       [in] A pointer to a caller-supplied context for the CA instance
>*       being created.  The HCA driver passes this value to the client's
>*       event callback functions.  If the client will not be using event
>*       callbacks, this value is unused.
>*
>*   CaAsyncEvent
>*       [in] A pointer to a client PFN_IB_CA_ASYNC_EVENT event callback
>*       function for the new CA instance.  This callback is optional, and
>*       events can be retrieved via calls to PGET_CA_FATAL_EVENT and
>*       PGET_CA_PORT_EVENT.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The operation completed successfully.  The IoStatus.Information
>*       field of the IRP contains a pointer to a CA instance structure,
>*       IB_CA, for the new CA instance.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PCLOSE_CA
>* DESCRIPTION
>*   Closes an open instance of an HCA.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PCLOSE_CA)(
>    IN              IB_CA* const                pCa,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCa
>*       [in] Pointer to an IB_CA structure previously created by a call to
>*       POPEN_CA.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CA instance was closed successfully.  The IRP will be completed
>*       with success status.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   There are no other status codes
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   This function cannot fail.  If a resource is in use, the IRP will not
>*   complete until all references to the resource have been released.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*
>*   Any pending event notification IRPs will be completed with
>*   STATUS_CANCELLED before the function returns.
>*********/
>
>
>/****f* Verbs/PGET_CA_FATAL_EVENT
>* DESCRIPTION
>*   Queues a request for HCA catastrophic error notification.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PGET_CA_FATAL_EVENT)(
>    IN              IB_CA* const                pCa,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCa
>*       [in] Pointer to an IB_CA structure previously created by a call to
>*       POPEN_CA.
>*
>*   pIrp
>*       [in] IRP that will be completed when a local catastrophic error
>*       occurs on the HCA.  The IRP major and minor functions as well as
>*       the I/O stack must be preserved when the IRP is completed.  The
>*       Irp.RequestorMode can be used to distinguish kernel and user
>clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CA instance experienced a fatal error and should be closed.
>*       The IRP will be completed with success status.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*
>*   The HCA driver must set a cancel routine to allow clients to cancel
>*   the notification request.
>*
>*   A client can queue up mutiple requests for notification.  Only one
>*   request is completed per event.  All such requests are completed with
>*   STATUS_CANCELLED if the IB_CA instance is closed.
>*********/
>
>
>/****f* Verbs/PGET_CA_PORT_EVENT
>* DESCRIPTION
>*   Queues a requests for port event notification.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PGET_CA_PORT_EVENT)(
>    IN              IB_CA* const                pCa,
>    IN      const   UCHAR                       PortNumber,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCa
>*       [in] Pointer to an IB_CA structure previously created by a call to
>*       POPEN_CA.
>*
>*   PortNumber
>*       [in] Port number to which the notification request is targetted.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CA instance experienced a port event.  The IoStatus.Information
>*       member of the IRP is updated with the IB_ASYNC_EVENT value.  The
>IRP
>*       will be completed with success status.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP and the IB_ASYNC_EVENT will be returned in the
>*       IoStatus.Information field of the IRP.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*
>*   The HCA driver must set a cancel routine to allow clients to cancel
>*   the notification request.
>*
>*   A client can queue up mutiple requests for notification.  Only one
>*   request is completed per event.  All such requests are completed with
>*   STATUS_CANCELLED if the IB_CA instance is closed.
>*********/
>
>
>/****f* Verbs/PQUERY_CA
>* DESCRIPTION
>*   Queries an HCA for its attributes.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PQUERY_CA)(
>    IN              VOID* const                 Context,
>    IN      const   SIZE_T                      AttrSize,
>        OUT         IB_CA_ATTR* const           pAttr,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   Context
>*       [in] Interface context returned in INTERFACE structure retrieved
>*       using IRP_MN_QUERY_INTERFACE.
>*
>*   pAttrSize
>*       [in] Size of the attributes buffer referenced by the pAttr
>parameter.
>*
>*   pAttr
>*       [out] CA attribute of this Host Channel adapter.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The full CA attributes, including port attributes and their GID
>*       and PKEY tables, were successfully copied to the attribute buffer.
>*       The IoStatus.Information field of the IRP is updated to indicate
>*       the number of bytes returned.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the number of bytes returned in the IoStatus.Information
>*       field of the IRP.
>*
>*   STATUS_BUFFER_OVERFLOW
>*       The operation completed but returned only partial attributes.  Full
>*       attributes were not returned due to insufficient space.  The size
>*       required to get the full attributes is returned in the Size field
>*       of the IB_CA_ATTR structure.  The IoStatus.Information field of the
>*       IRP is updated to indicate the number of bytes returned.
>*
>*   STATUS_BUFFER_TOO_SMALL
>*       The pAttr buffer was too small to return any information.  At a
>*       minimum, the buffer must be the 4 bytes to allow the required
>*       size to be returned.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PQUERY_PORT
>* DESCRIPTION
>*   Queries an HCA for one of its port attributes.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PQUERY_PORT)(
>    IN              VOID* const                 Context,
>    IN      const   UCHAR                       PortNumber,
>    IN      const   SIZE_T                      AttrSize,
>        OUT         IB_PORT_ATTR* const         pAttr,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   Context
>*       [in] Interface context returned in INTERFACE structure retrieved
>*       using IRP_MN_QUERY_INTERFACE.
>*
>*   PortNumber
>*       [in] 1-based port number targetted by this request.
>*
>*   pAttrSize
>*       [in] Size of the attributes buffer referenced by the pAttr
>parameter.
>*
>*   pAttr
>*       [out] CA attribute of this Host Channel adapter.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The full port attributes, including the GID and PKEY tables, were
>*       successfully copied to the attribute buffer.
>*       The IoStatus.Information field of the IRP is updated to indicate
>*       the number of bytes returned.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the number of bytes returned in the IoStatus.Information
>*       field of the IRP.
>*
>*   STATUS_MORE_ENTRIES
>*       The operation completed but returned only partial attributes.  Full
>*       attributes were not returned due to insufficient space.  The size
>*       required to get the full attributes is returned in the Size field
>*       of the IB_PORT_ATTR structure.  The IoStatus.Information field of
>*       the IRP is updated to indicate the number of bytes returned.
>*
>*   STATUS_BUFFER_TOO_SMALL
>*       The pAttr buffer was too small to return any information.  At a
>*       minimum, the buffer must be the 4 bytes to allow the required
>*       size to be returned.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PMODIFY_CA
>* DESCRIPTION
>*   Modify the attributes of a Channel Adapter.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PMODIFY_CA)(
>    IN              IB_CA* const                pCa,
>    IN      const   ULONG                       Flags,
>    IN      const   IB_MODIFY_CA* const         pModify,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCa
>*       [in] Pointer to an IB_CA structure previously created by a call to
>*       POPEN_CA.
>*
>*   Flags
>*       [in] Combination of IB_MODIFY_CA_FLAGS values indicating which
>*       fields are valid in the IB_MODIFY_CA structure.
>*
>*   pModify
>*       [in] Pointer to an IB_MODIFY_CA structure with the fields specified
>*       by the Flags parameter set to the desired values.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CA was modified successfully.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PMODIFY_PORT
>* DESCRIPTION
>*   Modify the attributes of a Channel Adapter's port.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PMODIFY_PORT)(
>    IN              IB_CA* const                pCa,
>    IN      const   UCHAR                       PortNumber,
>    IN      const   ULONG                       Flags,
>    IN      const   IB_MODIFY_PORT* const       pModify,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCa
>*       [in] Pointer to an IB_CA structure previously created by a call to
>*       POPEN_CA.
>*
>*   PortNumber
>*       [in] 1-based port number targetted by this request.
>*
>*   Flags
>*       [in] Combination of IB_MODIFY_CA_FLAGS values indicating which
>*       fields are valid in the IB_MODIFY_CA structure.
>*
>*   pModify
>*       [in] Pointer to an IB_MODIFY_CA structure with the fields specified
>*       by the Flags parameter set to the desired values.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CA's port was modified successfully.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   STATUS_FILE_LOCK_CONFLICT
>*       The requested capabilities conflict with existing capabilities.
>*       This can occur if multiple clients try to bits such as
>*       IB_PORT_CAP_IS_SM or IB_PORT_CAP_CM_SUP bits in the port
>*       capabilities.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>typedef struct _IB_CQ
>{
>    IB_CA           *pCa;
>    ULONG           NumCqes;
>    VOID            *Context;
>    LONG volatile   nRef;
>
>}   IB_CQ;
>
>
>typedef VOID
>(*PFN_IB_CQ_COMP_EVENT)(
>        IN              IB_CQ                       *pCq,
>        IN              VOID                        *CqContext );
>
>typedef VOID
>(*PFN_IB_CQ_ASYNC_EVENT)(
>        IN              IB_CQ                       *pCq,
>        IN              VOID                        *CqContext,
>        IN              IB_ASYNC_EVENT              Event );
>
>
>/****f* Verbs/PCREATE_CQ
>* DESCRIPTION
>*   Create a Completion Queue.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PCREATE_CQ)(
>    IN              IB_CA* const                pCa,
>    IN              ULONG const                 NumCqes,
>    IN              VOID* const                 Context,
>    IN              PFN_IB_CQ_COMP_EVENT        CqCompEvent OPTIONAL,
>    IN              PFN_IB_CQ_ASYNC_EVENT       CqAsyncEvent OPTIONAL,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCa
>*       [in] Pointer to an IB_CA structure previously created by a call to
>*       POPEN_CA.
>*
>*   NumCqes
>*       [in] Number of CQ entries requested by the consumer.
>*
>*   Context
>*       [in] A pointer to a caller-supplied context for the CQ being
>*       created.  The HCA driver passes this value to the client's
>*       event callback functions.  If the client will not be using event
>*       callbacks, this value is unused.
>*
>*   CqCompEvent
>*       [in] A pointer to a client PFN_IB_CQ_COMP_EVENT event callback
>*       function for the new CQ instance.  This parameter may be NULL and
>*       completion notification requests can be issued through a call to
>*       PGET_CQ_COMP_EVENT.
>*
>*   CqAsyncEvent
>*       [in] A pointer to a client PFN_IB_CQ_ASYNC_EVENT event callback
>*       function for the new CQ instance.  This parameter may be NULL and
>*       asynchonrous affiliated event notification requests can be issued
>*       through a call to PGET_CQ_ASYNC_EVENT.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CQ was created successfully.  The IB_CQ structure is returned
>*       in the IoStatus.Information field of the IRP.  The IB_CQ structure
>*       contains the actual number of CQ entries allocated for the CQ.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the IB_CQ structure in the IoStatus.Information field of
>*       the IRP.  The IB_CQ structure contains the actual number of CQ
>*       entries allocated for the CQ.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PDESTROY_CQ
>* DESCRIPTION
>*   Destroy a Completion Queue.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PDESTROY_CQ)(
>    IN              IB_CQ* const                pCq,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCq
>*       [in] Pointer to an IB_CQ structure previously created by a call to
>*       PCREATE_CQ.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CQ was destroyed successfully.  No futher operations can be
>*       performed on the CQ.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   There are no other status codes
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   This function cannot fail.  If a resource is in use, the IRP will not
>*   complete until all references to the resource have been released.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PRESIZE_CQ
>* DESCRIPTION
>*   Change the number of CQEs supported by a Completion Queue.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PRESIZE_CQ)(
>    IN              IB_CQ* const                pCq,
>    IN              ULONG const                 NumCqes,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCq
>*       [in] Pointer to an IB_CQ structure previously created by a call to
>*       PCREATE_CQ.
>*
>*   NumCqes
>*       [in] Number of CQ entries requested by the consumer.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CQ was resized successfully.  The IB_CQ structure is updated
>*       with the actual number of CQ entries allocated for the CQ.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.  The IB_CQ structure is updated with the actual
>*       number of CQ entries allocated for the CQ.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PREARM_CQ
>* DESCRIPTION
>*   Requests that the completion queue generate a notifification when
>*   the next completion of the requested type is added.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PREARM_CQ)(
>    IN              IB_CQ* const                pCq,
>    IN      const   BOOLEAN                     Solicited );
>/*
>* PARAMETERS
>*   pCq
>*       [in] Pointer to an IB_CQ structure previously created by a call to
>*       PCREATE_CQ.
>*
>*   Solicited
>*       [in] A flag indicating whether the request is to generate a
>*       notification on the next entry, if set to FALSE, or on the next
>*       solicited entry being added to the completion queue, if set to
>TRUE.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CQ was rearmed successfully.
>*
>*   Other status codes
>*       An error occurred.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*********/
>
>
>/****f* Verbs/PREARM_N_CQ
>* DESCRIPTION
>*   Request that the completion queue generate a notification when
>*   the next N completions have been added to this CQ.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PREARM_N_CQ)(
>    IN              IB_CQ* const                pCq,
>    IN      const   ULONG                       NumCqes );
>/*
>* PARAMETERS
>*   pCq
>*       [in] Pointer to an IB_CQ structure previously created by a call to
>*       PCREATE_CQ.
>*
>*   NumCqes
>*       [in] The number of completion queue entries to be added to the
>*       completion queue before notifying the client.  This value must
>*       greater than or equal to one and less than or equal to the size
>*       of the completion queue.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CQ was rearmed successfully.
>*
>*   STATUS_INVALID_PARAMETER
>*       The requested number of completion queue entries was invalid.
>*
>*   STATUS_NOT_SUPPORTED
>*       This operation is not supported by the channel adapter.
>*
>*   Other status codes
>*       An error occurred.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*********/
>
>
>/****f* Verbs/PPOLL_CQ
>* DESCRIPTION
>*   Poll a CQ for at most nCqe completions
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PPOLL_CQ)(
>    IN              IB_CQ* const                pCq,
>    IN  OUT         LONG* const                 pNumWc,
>        OUT         IB_WC*                      Wc[] );
>/*
>* PARAMETERS
>*   pCq
>*       [in] Pointer to an IB_CQ structure previously created by a call to
>*       PCREATE_CQ.
>*
>*   pNumWc
>*       [in/out] On input, number of IB_WC work completions in the Wc[]
>array
>*       provided by the consumer, on output number of work completions
>*       returned.  This pointer is resident and accessible from
>*       IRQL == DISPATCH_LEVEL.
>*
>*   Wc[]
>*       [in] Array of *pNumWc IB_WC structures used to retrive completions
>*       from the completion queue.  This buffer is resident and accessible
>*       from IQRL == DISPATCH_LEVEL.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The operation successfully polled the CQ.  The number of work
>*       completions returned is provided by the pNumWc parameter.
>*
>*   STATUS_NO_MORE_ENTRIES
>*       All entries have been reaped from the CQ.  The pNumWc parameter is
>*       returned as zero.
>*
>*   Other status codes
>*       An error occurred.  The pNumWc parameter is updated to reflect the
>*       number of successfully reaped work completions.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*********/
>
>
>/****f* Verbs/PPEEK_CQ
>* DESCRIPTION
>*   Peek a CQ and return the number of available work completions.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PPEEK_CQ)(
>    IN              IB_CQ* const                pCq,
>        OUT         LONG* const                 pNumWc );
>/*
>* PARAMETERS
>*   pCq
>*       [in] Pointer to an IB_CQ structure previously created by a call to
>*       PCREATE_CQ.
>*
>*   pNumWc
>*       [out] Number of work completions available on the CQ.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The operation completed successfully and the pNumWc parameter is
>*       updated with the number of available CQ entries.
>*
>*   Other status codes
>*       An error occurred.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*********/
>
>
>/****f* Verbs/PGET_CQ_COMP_EVENT
>* DESCRIPTION
>*   Queue a request for completion event notification.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PGET_CQ_COMP_EVENT)(
>    IN              IB_CQ* const                pCq,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCq
>*       [in] Pointer to an IB_CQ structure previously created by a call to
>*       PCREATE_CQ.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CQ experienced a completion event.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP upon the next completion event, at which
>point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*
>*   The HCA driver must set a cancel routine to allow clients to cancel
>*   the notification request.
>*
>*   Only a single requests for notification can be queued at a time.
>*   Any previously queued request is completed with STATUS_CANCELLED.
>*   All such requests are completed with STATUS_CANCELLED if the IB_CQ
>*   instance is destroyed.
>*********/
>
>
>/****f* Verbs/PGET_CQ_ASYNC_EVENT
>* DESCRIPTION
>*   Queue a request for asynchronous event notification.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PGET_CQ_ASYNC_EVENT)(
>    IN              IB_CQ* const                pCq,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCq
>*       [in] Pointer to an IB_CQ structure previously created by a call to
>*       PCREATE_CQ.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The CQ experienced an asynchronous event.  The IoStatus.Information
>*       field of the IRP is updated with the IB_ASYNC_EVENT code.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP upon the next completion event, at which
>point
>*       the status of the operation will be returned in the IoStatus.Status
>*       and the IB_ASYNC_EVENT code is returned in the IoStatus.Information
>*       fields of the IRP.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*
>*   The HCA driver must set a cancel routine to allow clients to cancel
>*   the notification request.
>*
>*   A client can queue up mutiple requests for notification.  Only one
>*   request is completed per event.  All such requests are completed with
>*   STATUS_CANCELLED if the IB_CQ instance is destroyed.
>*********/
>
>
>typedef struct _IB_PD
>{
>    IB_CA           *pCa;
>    LONG volatile   nRef;
>
>}   IB_PD;
>
>
>/****f* Verbs/PCREATE_PD
>* DESCRIPTION
>*   Create a Protection Domain.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PCREATE_PD)(
>    IN              IB_CA* const                pCa,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pCa
>*       [in] Pointer to an IB_CA structure previously created by a call to
>*       POPEN_CA.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The PD was created successfully.  The IB_PD structure is returned
>*       in the IoStatus.Information field of the IRP.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the IB_PD structure in the IoStatus.Information field of
>*       the IRP.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PDESTROY_PD
>* DESCRIPTION
>*   Destroy a Protection Domain.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PDESTROY_PD)(
>    IN              IB_PD* const                pPd,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The PD was destroyed successfully.  No futher operations can be
>*       performed on the PD.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   There are no other status codes
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   This function cannot fail.  If a resource is in use, the IRP will not
>*   complete until all references to the resource have been released.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>typedef struct _IB_AH
>{
>    IB_PD           *pPd;
>
>}   IB_AH;
>
>
>/****f* Verbs/PCREATE_AH
>* DESCRIPTION
>*   Create an unreliable datagram address handle.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PCREATE_AH)(
>    IN              IB_PD* const                pPd,
>    IN              IB_AH_ATTR* const           pAhAttr,
>        OUT         IB_AH** const               ppAh );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   pAhAttr
>*       [in] Pointer to an IB_AH_CREATE structure containing the desired
>*       attributes for the newly created queue pair.
>*
>*   ppAh
>*       [out] Pointer to user's AH poiner, set on output if the function
>*       succeeds.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The AH was created successfully.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The client must release all
>*       resources and close the CA as soon as possible.
>*
>*   STATUS_INVALID_PARAMETER_1
>*       Invalid port number.
>*
>*   STATUS_INVALID_PARAMETER_2
>*       Invalid source GID.  Only possible if IB_AH_GRH flag is specified.
>*
>*   Other status codes
>*       An error occurred.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*********/
>
>
>/****f* Verbs/PDESTROY_AH
>* DESCRIPTION
>*   Destroy an Address Handle.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PDESTROY_AH)(
>    IN              IB_AH* const                pAh );
>/*
>* PARAMETERS
>*   pAh
>*       [in] Pointer to an IB_AH structure previously created by a call to
>*       PCREATE_AH.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The AH was destroyed successfully.  No futher operations can be
>*       performed referencing the AH.
>*
>*   There are no other status codes
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   This function cannot fail.
>*********/
>
>
>/****f* Verbs/PMODIFY_AH
>* DESCRIPTION
>*   Modifies the attributes of an Address Handle.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PMODIFY_AH)(
>    IN              IB_AH* const                pAh,
>    IN              IB_AH_ATTR* const           pModify );
>/*
>* PARAMETERS
>*   pAh
>*       [in] Pointer to an IB_AH structure previously created by a call to
>*       PCREATE_AH.
>*
>*   pModify
>*       [in] Specifies the new attributes for the address handle.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The AH was modified successfully.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The client must release all
>*       resources and close the CA as soon as possible.
>*
>*   STATUS_INVALID_PARAMETER_1
>*       Invalid port number.
>*
>*   STATUS_INVALID_PARAMETER_2
>*       Invalid source GID.  Only possible if IB_AH_GRH flag is specified.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*********/
>
>
>/****f* Verbs/PQUERY_AH
>* DESCRIPTION
>*   Modifies the attributes of a Queue Pair.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PQUERY_AH)(
>    IN              IB_AH* const                pQp,
>        OUT         IB_AH_ATTR* const           pAttr );
>/*
>* PARAMETERS
>*   pQp
>*       [in] Pointer to an IB_AH structure previously created by a call to
>*       PCREATE_AH.
>*
>*   pAttr
>*       [out] Pointer to an IB_AH_ATTR structure to update with the AH's
>*       attributes.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The AH attributes were successfully retrieved.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The client must release all
>*       resources and close the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>typedef struct _IB_SRQ
>{
>    IB_PD           *pPd;
>    VOID            *Context;
>
>}   IB_SRQ;
>
>
>typedef NTSTATUS
>(*PPOST_SRQ_RECV)(
>    IN              IB_SRQ* const               pSrq,
>    IN      const   IB_RECV_WR*                 pRecvWr,
>    IN              IB_RECV_WR** const          ppFailedRecv );
>
>
>typedef struct _IB_QP
>{
>    IB_PD           *pPd;
>    VOID            *Context;
>
>    IB_CQ           *pSendCq;
>    IB_CQ           *pRecvCq;
>    IB_SRQ          *pSrq;
>
>    IB_QP_TYPE      Type;
>    ULONG           Qpn;
>
>    ULONG           SqDepth;
>    ULONG           RqDepth;
>    ULONG           SqSge;
>    ULONG           RqSge;
>    ULONG           SqMaxInline;
>
>}   IB_QP;
>
>
>typedef VOID
>(*PFN_IB_QP_ASYNC_EVENT)(
>        IN              IB_QP                       *pQp,
>        IN              VOID                        *QpContext,
>        IN              IB_ASYNC_EVENT              Event );
>
>
>/****f* Verbs/PCREATE_QP
>* DESCRIPTION
>*   Create a Queue Pair.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PCREATE_QP)(
>    IN              IB_PD* const                pPd,
>    IN              IB_QP_CREATE* const         pCreateQp,
>    IN              VOID* const                 Context,
>    IN              PFN_IB_QP_ASYNC_EVENT       QpAsyncEvent OPTIONAL,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   pCreateAttr
>*       [in] Pointer to an IB_QP_CREATE structure containing the desired
>*       attributes for the newly created queue pair.
>*
>*   Context
>*       [in] A pointer to a caller-supplied context for the QP being
>*       created.  The HCA driver passes this value to the client's
>*       event callback functions.  If the client will not be using event
>*       callbacks, this value is unused.
>*
>*   QpAsyncEvent
>*       [in] A pointer to a client PFN_IB_QP_ASYNC_EVENT event callback
>*       function for the new QP instance.  This parameter may be NULL and
>*       asynchonrous affiliated event notification requests can be issued
>*       through a call to IbGetQpAsyncEvent.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The QP was created successfully.  The IB_QP structure is returned
>*       in the IoStatus.Information field of the IRP.  The IB_QP structure
>*       contains the actual QP attributes allocated for the QP.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the IB_QP structure in the IoStatus.Information field of
>*       the IRP.  The IB_QP structure contains the actual QP attributes.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PCREATE_QP
>* DESCRIPTION
>*   Create a Queue Pair.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PCREATE_SPECIAL_QP)(
>    IN              IB_PD* const                pPd,
>    IN              UCHAR                       PortNum,
>    IN              IB_QP_CREATE* const         pCreateQp,
>    IN              VOID* const                 Context,
>    IN              PFN_IB_QP_ASYNC_EVENT       QpAsyncEvent OPTIONAL,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   PortNum
>*       [in] One-based port number for which this special queue pair
>*       is created.
>*
>*   pCreateAttr
>*       [in] Pointer to an IB_QP_CREATE structure containing the desired
>*       attributes for the newly created queue pair.
>*
>*   Context
>*       [in] A pointer to a caller-supplied context for the QP being
>*       created.  The HCA driver passes this value to the client's
>*       event callback functions.  If the client will not be using event
>*       callbacks, this value is unused.
>*
>*   QpAsyncEvent
>*       [in] A pointer to a client PFN_IB_QP_ASYNC_EVENT event callback
>*       function for the new QP instance.  This parameter may be NULL and
>*       asynchonrous affiliated event notification requests can be issued
>*       through a call to IbGetQpAsyncEvent.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The QP was created successfully.  The IB_QP structure is returned
>*       in the IoStatus.Information field of the IRP.  The IB_QP structure
>*       contains the actual QP attributes allocated for the QP.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the IB_QP structure in the IoStatus.Information field of
>*       the IRP.  The IB_QP structure contains the actual QP attributes.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PDESTROY_QP
>* DESCRIPTION
>*   Destroy a Queue Pair.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PDESTROY_QP)(
>    IN              IB_QP* const                pQp,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pQp
>*       [in] Pointer to an IB_QP structure previously created by a call to
>*       PCREATE_QP.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The QP was destroyed successfully.  No futher operations can be
>*       performed on the QP.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   There are no other status codes
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   This function cannot fail.  If a resource is in use, the IRP will not
>*   complete until all references to the resource have been released.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PMODIFY_QP
>* DESCRIPTION
>*   Modifies the attributes of a Queue Pair.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PMODIFY_QP)(
>    IN              IB_QP* const                pQp,
>    IN              IB_QP_MODIFY* const         pModify,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pQp
>*       [in] Pointer to an IB_QP structure previously created by a call to
>*       PCREATE_QP.
>*
>*   pModify
>*       [in] Specifies the attributes to modify and their requested values.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The QP was modified successfully.  The IB_QP structure is updated
>*       with the actual attributes.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.  The IB_QP structure is updated with the actual
>*       attribute.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PQUERY_QP
>* DESCRIPTION
>*   Modifies the attributes of a Queue Pair.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PQUERY_QP)(
>    IN              IB_QP* const                pQp,
>        OUT         IB_QP_ATTR* const           pAttr,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pQp
>*       [in] Pointer to an IB_QP structure previously created by a call to
>*       PCREATE_QP.
>*
>*   pAttr
>*       [out] Pointer to an IB_QP_ATTR structure to update with the QP's
>*       attributes.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The QP attributes were successfully retrieved.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*********/
>
>
>/****f* Verbs/PPOST_SEND
>* DESCRIPTION
>*   Posts a work request to the send queue of a queue pair.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PPOST_SEND)(
>    IN              IB_QP* const                pQp,
>    IN      const   IB_SEND_WR*                 pSendWr,
>        OUT         IB_SEND_WR** const          ppFailedSend );
>/*
>* PARAMETERS
>*   pQp
>*       [in] Pointer to an IB_QP structure previously created by a call to
>*       PCREATE_QP.
>*
>*   pSendWr
>*       [in] A reference to the head of the send work request list.  The
>list
>*       is resident and accessible from IRQL == DISPATCH_LEVEL.
>*
>*   ppFailedSend
>*       [out] If the post send operation failed, this references the work
>*       request in the pSendWr list where the first failure occurred.
>*       This parameter may be NULL only if a single work request is being
>*       posted to the QP.  The pointer is resident and accessible from
>*       IRQL == DISPATCH_LEVEL.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       All work requests were successfully posted.
>*
>*   STATUS_INVALID_PARAMETER
>*       A reference to the send work request list was not provided.
>*
>*   STATUS_IB_INSUFFICIENT_WR
>*       The number of posted work requests exceed the current depth
>available
>*       on the send queue.
>*
>*   STATUS_IB_INSUFFICIENT_SGE
>*       The number of scatter-gather entries referenced by the work request
>*       exceeded the send queue configuration.
>*
>*   STATUS_IB_INVALID_WR_TYPE
>*       The work request type was invalid.
>*
>*   STATUS_IB_INVALID_STATE
>*       The current queue pair state does not allow posting sends.
>*
>*   STATUS_INVALID_OPERATION
>*       The requested operation is not supported by the queue pair.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*****/
>
>
>/****f* Verbs/PPOST_RECV
>* DESCRIPTION
>*   Posts a work request to the receive queue of a queue pair.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PPOST_RECV)(
>    IN              IB_QP* const                pQp,
>    IN      const   IB_RECV_WR*                 pRecvWr,
>    IN              IB_RECV_WR** const          ppFailedRecv );
>/*
>* PARAMETERS
>*   pQp
>*       [in] Pointer to an IB_QP structure previously created by a call to
>*       PCREATE_QP.
>*
>*   pRecvWr
>*       [in] A reference to the head of the work request list.
>*
>*   ppFailedRecv
>*       [out] If the post receive operation failed, this references the
>work
>*       request in the pRecvWr list where the first failure occurred.
>*       This parameter may be NULL only if a single work request is being
>*       posted to the QP.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       All work requests were successfully posted.
>*
>*   STATUS_INVALID_PARAMETER
>*       A reference to the send work request list was not provided.
>*
>*   STATUS_IB_INSUFFICIENT_WR
>*       The number of posted work requests exceed the current depth
>available
>*       on the send queue.
>*
>*   STATUS_IB_INSUFFICIENT_SGE
>*       The number of scatter-gather entries referenced by the work request
>*       exceeded the send queue configuration.
>*
>*   STATUS_IB_INVALID_WR_TYPE
>*       The work request type was invalid.
>*
>*   STATUS_IB_INVALID_STATE
>*       The current queue pair state does not allow posting sends.
>*
>*   STATUS_INVALID_OPERATION
>*       The requested operation is not supported by the queue pair.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*****/
>
>
>/****f* Verbs/PGET_QP_ASYNC_EVENT
>* DESCRIPTION
>*   Queue a request for asynchronous event notification.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PGET_QP_ASYNC_EVENT)(
>    IN              IB_QP* const                pQp,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pQp
>*       [in] Pointer to an IB_QP structure previously created by a call to
>*       PCREATE_QP.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The QP experienced an asynchronous event.  The IoStatus.Information
>*       field of the IRP is updated with the IB_ASYNC_EVENT code.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP upon the next completion event, at which
>point
>*       the status of the operation will be returned in the IoStatus.Status
>*       and the IB_ASYNC_EVENT code is returned in the IoStatus.Information
>*       fields of the IRP.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*
>*   The HCA driver must set a cancel routine to allow clients to cancel
>*   the notification request.
>*
>*   A client can queue up mutiple requests for notification.  Only one
>*   request is completed per event.  All such requests are completed with
>*   STATUS_CANCELLED if the IB_QP instance is destroyed.
>*********/
>
>
>#pragma warning(disable:4201)
>typedef struct _IB_MR
>{
>    IB_PD           *pPd;
>    UINT8* __ptr64  pLocalStart;
>    UINT8* __ptr64  pLocalEnd;
>    UINT8* __ptr64  pRemoteStart;
>    UINT8* __ptr64  pRemoteEnd;
>    ULONG           AccessMask;
>    UINT32          LKey;
>    UINT32          RKey;
>    union _u
>    {
>        /** For virtual registrations */
>        MDL                 *pMdlList;
>        /** For non-virtual registrations */
>        PHYSICAL_ADDRESS    IoVa;
>    };
>    LONG volatile   nRef;
>
>}   IB_MR;
>#pragma warning(default:4201)
>
>
>/****f* Verbs/PREG_MDL
>* DESCRIPTION
>*   Registers a virtual memory region with a channel adapter.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PREG_MDL)(
>    IN              IB_PD* const                pPd,
>    IN              MDL* const                  pMdlList,
>    IN      const   ULONG                       AccessMask,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   pMdlList
>*       [in] A list of virtually contiguous MDLs describing the memory
>*       region.  The MDL page tables are populated and the memory
>referenced
>*       properly pinned.  The only partial pages referenced by the MDLs
>*       are the first page of the first MDL in the list and last page of
>*       the last MDL in the list.  All other pages are full pages.  The
>page
>*       addresses are CPU addresses, and should be properly mapped for DMA
>*       access by the HCA driver.
>*
>*   AccessMask
>*       [in] The local access rights requested forthis registered memory
>*       region.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The memory region was successfully registered.  The IB_MR structure
>*       is returned in the IoStatus.Information field of the IRP.  The
>IB_MR
>*       structure contains the actual attributes of the MR.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the IB_MR structure in the IoStatus.Information field of
>*       the IRP.  The IB_MR structure contains the actual MR attributes.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*****/
>
>
>/****f* Verbs/PREG_SCATTER_GATHER
>* DESCRIPTION
>*   Registers a virtual memory region with a channel adapter.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PREG_SCATTER_GATHER)(
>    IN              IB_PD* const                pPd,
>    IN              SCATTER_GATHER_LIST*        pScatterGather,
>    IN      const   ULONG                       AccessMask,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   pScatterGather
>*       [in] Scatter/gather list to register.  Only the first and last
>*       elements in the list can reference partial pages.  All other
>elements
>*       must reference an intergral number of pages.  If the list has more
>*       than one element, the first and last elements in the list must end
>*       and start on a page boundary, respectively.  The physical addresses
>*       referenced by the scatter/gather list are bus-relative addresses
>and
>*       have already been properly mapped for DMA access.
>*
>*   AccessMask
>*       [in] The local access rights requested forthis registered memory
>*       region.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The memory region was successfully registered.  The IB_MR structure
>*       is returned in the IoStatus.Information field of the IRP.  The
>IB_MR
>*       structure contains the actual attributes of the MR.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the IB_MR structure in the IoStatus.Information field of
>*       the IRP.  The IB_MR structure contains the actual MR attributes.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   The I/O virtual address for the first element's lower bound is zero.
>*   Clients are responsible for any offset within the region.
>*
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*****/
>
>
>/****f* Verbs/PGET_DMA_MR
>* DESCRIPTION
>*   Get a memory region representing all of physical memory.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PGET_DMA_MR)(
>    IN              IB_PD* const                pPd,
>    IN      const   ULONG                       AccessMask,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   AccessMask
>*       [in] The local access rights requested forthis registered memory
>*       region.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The memory region was successfully registered.  The IB_MR structure
>*       is returned in the IoStatus.Information field of the IRP.  The
>IB_MR
>*       structure contains the actual attributes of the MR.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the IB_MR structure in the IoStatus.Information field of
>*       the IRP.  The IB_MR structure contains the actual MR attributes.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*****/
>
>
>/****f* Verbs/PDEREG_MR
>* DESCRIPTION
>*   Deregister (MDL/SGL) or release (DMA) a memory region.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PDEREG_MR)(
>    IN              IB_MR* const                pMr,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_MR structure previously created by a call to
>*       PREG_MDL, PREG_SCATTER_GATHER, or PGET_DMA_MR.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The MR was successfully deregistered.  No futher operations can be
>*       performed using the MR.  The MDL list associated with the MR is
>*       returned in the IoStatus.Information field of the IRP.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   There are no other status codes
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   This function cannot fail.  If a resource is in use, the IRP will not
>*   complete until all references to the resource have been released.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*****/
>
>
>typedef struct _IB_MW
>{
>    IB_PD           *pPd;
>    UINT32          RKey;
>
>}   IB_MW;
>
>
>/****f* Verbs/PCREATE_MW
>* DESCRIPTION
>*   Get a memory region representing all of physical memory.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PCREATE_MW)(
>    IN              IB_PD* const                pPd,
>    IN  OUT         CI_UDATA* const             pUserData,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   pUserData
>*       [in] Descriptor for any data exchanged with the user-mode verb
>*       provider.  The input and output buffers are shadowed by the
>*       access layer in non-paged pool.  The HCA driver can only access
>*       further embedded pointers from the context of this call.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The memory window was successfully created.  The IB_MW structure
>*       is returned in the IoStatus.Information field of the IRP.  The
>IB_MW
>*       structure contains the unbound RKey value for the memory window.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field and the IB_MW structure in the IoStatus.Information field of
>*       the IRP.  The IB_MW structure contains the unbound RKey value.
>*
>*   STATUS_FILE_FORCED_CLOSED
>*       The CA is no longer functional.  The IRP will be completed with
>*       failure status.  The client must release all resources and close
>*       the CA as soon as possible.
>*
>*   Other status codes
>*       An error occurred.  The IRP will be completed with failure status.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*****/
>
>
>/****f* Verbs/PDESTROY_MW
>* DESCRIPTION
>*   Get a memory region representing all of physical memory.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PDESTROY_MW)(
>    IN              IB_MR* const                pMr,
>    IN              IRP* const                  pIrp );
>/*
>* PARAMETERS
>*   pPd
>*       [in] Pointer to an IB_PD structure previously created by a call to
>*       PCREATE_PD.
>*
>*   pIrp
>*       [in] IRP that will be completed when the request completes.
>*       The IRP major and minor functions as well as the I/O stack must
>*       be preserved when the IRP is completed.  The Irp.RequestorMode
>*       can be used to distinguish kernel and user clients.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       The MR was successfully deregistered.  No futher operations can be
>*       performed using the MR.  The MDL list associated with the MR is
>*       returned in the IoStatus.Information field of the IRP.
>*
>*   STATUS_PENDING
>*       The operation could not be completed immediately.  The HCA driver
>*       will complete the IRP when the operation completes, at which point
>*       the status of the operation will be returned in the IoStatus.Status
>*       field of the IRP.
>*
>*   There are no other status codes
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*
>*   This function cannot fail.  If a resource is in use, the IRP will not
>*   complete until all references to the resource have been released.
>*
>*   The HCA driver must call IoCallDriver for the input IRP so that it
>*   can call I/O complete request.  The target of IoCallDriver must be
>*   the FDO of the HCA since the IRP will have only a single I/O stack
>*   location for the HCA to use.  This implies that the IRP cannot be
>*   passed down a chain of layered drivers.
>*****/
>
>
>/****f* Verbs/PBIND_MW
>* DESCRIPTION
>*   Binds a memory window.
>*
>* SYNOPSIS
>*/
>typedef NTSTATUS
>(*PBIND_MW)(
>    IN              IB_QP* const                pQp,
>    IN              IB_MW* const                pMw,
>    IN      const   IB_BIND_WR* const           pBindWr );
>/*
>* PARAMETERS
>*   pQp
>*       [in] Pointer to an IB_QP structure previously created by a call to
>*       PCREATE_QP.
>*
>*   pSendWr
>*       [in] A reference to the head of the send work request list.  The
>list
>*       is resident and accessible from IRQL == DISPATCH_LEVEL.
>*
>*   ppFailedSend
>*       [out] If the post send operation failed, this references the work
>*       request in the pSendWr list where the first failure occurred.
>*       This parameter may be NULL only if a single work request is being
>*       posted to the QP.  The pointer is resident and accessible from
>*       IRQL == DISPATCH_LEVEL.
>*
>* RETURN VALUES
>*   STATUS_SUCCESS
>*       All work requests were successfully posted.
>*
>*   STATUS_INVALID_PARAMETER
>*       A reference to the send work request list was not provided.
>*
>*   STATUS_IB_INSUFFICIENT_WR
>*       The number of posted work requests exceed the current depth
>available
>*       on the send queue.
>*
>*   STATUS_IB_INSUFFICIENT_SGE
>*       The number of scatter-gather entries referenced by the work request
>*       exceeded the send queue configuration.
>*
>*   STATUS_IB_INVALID_WR_TYPE
>*       The work request type was invalid.
>*
>*   STATUS_IB_INVALID_STATE
>*       The current queue pair state does not allow posting sends.
>*
>*   STATUS_INVALID_OPERATION
>*       The requested operation is not supported by the queue pair.
>*
>* NOTES
>*   This function can be invoked at IRQL <= DISPATCH_LEVEL.
>*****/
>
>
>#define IB_VERBS_INTERFACE_VERSION  (1)
>
>typedef struct _IB_VERBS_INTERFACE_STANDARD
>{
>    /** Generic interface header */
>    USHORT                  Size;
>    USHORT                  Version;
>
>    /** The context is used in the OpenCa, QueryCa, QueryPort,
>     * ModifyCa, and ModifyPort calls
>     */
>    PVOID                   Context;
>    PINTERFACE_REFERENCE    InterfaceReference;
>    PINTERFACE_DEREFERENCE  InterfaceDereference;
>
>    /** HCA information */
>    UINT64                  NodeGuid;
>    UINT32                  VendorId;
>    UINT16                  DeviceId;
>    UINT16                  DeviceRev;
>    UINT32                  FwVersion;
>    CHAR                    UvpName[32];
>
>    /** HCA Management */
>    POPEN_CA                OpenCa;
>    PCLOSE_CA               CloseCa;
>    PGET_CA_FATAL_EVENT     GetCaFatalEvent;
>    PGET_CA_PORT_EVENT      GetCaPortEvent;
>    PQUERY_CA               QueryCa;
>    PQUERY_PORT             QueryPort;
>    PMODIFY_CA              ModifyCa;
>    PMODIFY_PORT            ModifyPort;
>
>    /** CQ Management */
>    PCREATE_CQ              CreateCq;
>    PDESTROY_CQ             DestroyCq;
>    PRESIZE_CQ              ResizeCq;
>    PREARM_CQ               RearmCq;
>    PREARM_N_CQ             RearmNCq;
>    PPOLL_CQ                PollCq;
>    PPEEK_CQ                PeekCq;
>    PGET_CQ_COMP_EVENT      GetCqCompEvent;
>    PGET_CQ_ASYNC_EVENT     GetCqAsyncEvent;
>
>    /** Protection Domain Management */
>    PCREATE_PD              CreatePd;
>    PDESTROY_PD             DestroyPd;
>
>    /** Address Handle Management. */
>    PCREATE_AH              CreateAh;
>    PDESTROY_AH             DestroyAh;
>    PMODIFY_AH              ModifyAh;
>    PQUERY_AH               QueryAh;
>
>    /** Shared Receive Queue Management */
>    //IbCreateSrq           CreateSrq;
>    //IbDestroySrq          DestroySrq;
>    //IbPostSrqRecv         PostSrqRecv;
>
>    /** Queue Pair Management */
>    PCREATE_QP              CreateQp;
>    PCREATE_SPECIAL_QP      CreateSpecialQp;
>    PDESTROY_QP             DestroyQp;
>    PMODIFY_QP              ModifyQp;
>    PQUERY_QP               QueryQp;
>    PPOST_SEND              PostSend;
>    PPOST_RECV              PostRecv;
>    PGET_QP_ASYNC_EVENT     GetQpAsyncEvent;
>
>    /** Memory Region Management */
>    PREG_MDL                RegMdl;
>    PREG_SCATTER_GATHER     RegScatterGather;
>    PGET_DMA_MR             GetDmaMr;
>    PDEREG_MR               DeregMr;
>
>    /** Memory Window Management */
>    PCREATE_MW              CreateMw;
>    PDESTROY_MW             DestroyMw;
>    PBIND_MW                BindMw;
>
>}   IB_VERBS_INTERFACE_STANDARD;
>
>#endif _IBVERBS_
>
>/*
> * CI interface GUID.  The GUID is defined outside the conditional include
> * on purpose so that it can be instantiated only once where it is actually
> * needed.  See the DDK docs section "Using GUIDs in Drivers" for more
>info.
> *
> * {B721E2B0-AF62-4320-A6AF-92F71F78789F}
> */
>DEFINE_GUID(GUID_IB_VERBS_INTERFACE,
>0xb721e2b0, 0xaf62, 0x4320, 0xa6, 0xaf, 0x92, 0xf7, 0x1f, 0x78, 0x78,
>0x9f);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofw/attachments/20051107/5961aba2/attachment.html>


More information about the ofw mailing list