[Openib-windows] [RFC] IRP-based verbs
Fab Tillier
ftillier at silverstorm.com
Fri Sep 9 17:22:23 PDT 2005
> From: Tillier, Fabian
> Sent: Friday, September 09, 2005 3:14 PM
>
> Ok, here's a shot at defining a direct-call kernel interface that supports
> verb calls at DISPATCH. Right now it's just open and close CA. Currently,
> the interface requires an input IRP. We could change this to be optional,
> in which case the implementation could block for IRQL < DISPATCH_LEVEL, and
> fail the call for IRQL >= DISPATCH_LEVEL if it would block. Making the IRP
> optional complicates the implementation a little, which is why I didn't do
> it.
Updated with a few more functions to better illustrate what's going on. Also,
resource creation functions now take as input an optional event dispatch table
for up-calls to the client for notifications. Clients are free to use either
IRP driven event notifications or callback driven.
Thoughts?
- Fab
/*
* Copyright (c) 2005 SilverStorm Technologies. All rights reserved.
*
* This software is available to you under the OpenIB.org BSD license
* below:
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
* conditions are met:
*
* - Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
*
* - Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
* $Id$
*/
#include <ntddk.h>
#include <ib_defs.h>
/****s* Verbs/CI_UDATA
* NAME
* CI_UDATA
*
* DESCRIPTION
* Vendor specific structure to facilitate user mode IO
*
* This structure is provided to assist the vendor specific user mode
* library to exchange information with its kernel mode driver. The
* user mode InfiniBand(tm) Access Layer will call the vendor specific
* module before a call is made to the kernel mode driver. The kernel mode
* driver is expected to know the format and data in the input and output
* buffers, and copy any necessary data that must be handed to the user
* mode vendor library.
*
* PURPOSE
* pInputBuffer
* UVP input buffer. May overlap the output buffer.
*
* pOutputBuffer
* UVP output buffer. NULL if no output buffer is provided.
* May overlap the input buffer.
*
* InputSize
* Size of UVP provided buffer. Zero if there is no input buffer.
*
* OutputSize
* The size of the UVP provided output buffer. Zero if no output
* buffer is provided. The kernel mode driver sets this to the exact
* size that needs to be returned to its user mode counterpart.
*
* SOURCE
*/
typedef struct _CI_UDATA
{
const VOID* __ptr64 const pInputBuffer;
VOID* __ptr64 const pOutputBuffer;
SIZE_T InputSize;
SIZE_T OutputSize;
} CI_UDATA;
/******/
typedef void IB_API
(PFN_IB_CA_FATAL_EVENT*)(
IN IB_CA* const pCa,
IN VOID *CaContext );
typedef void IB_API
(PFN_IB_CA_PORT_EVENT*)(
IN IB_CA* const pCa,
IN VOID *CaContext,
IN const UCHAR PortNumber,
IN IB_ASYNC_EVENT Event );
typedef struct _IB_CA_DISPATCH
{
PFN_IB_CA_FATAL_EVENT FatalEvent;
PFN_IB_CA_PORT_EVENT PortEvent;
} IB_CA_DISPATCH;
typedef struct _IB_CA
{
IB_CA_DISPATCH Dispatch;
} IB_CA;
/****f* Verbs/CiOpenCa
* DESCRIPTION
* Opens the HCA for client use.
*
* SYNOPSIS
*/
CiOpenCa(
IN DEVICE_OBJECT* const pHcaFdo,
IN VOID* const Context,
IN const IB_CA_DISPATCH* const Dispatch OPTIONAL,
IN OUT CI_UDATA* const pUserData,
IN IRP* const pIrp );
/*
* PARAMETERS
* pHcaFdo
* [in] Functional Device object of the target HCA.
*
* Context
* [in] A pointer to a caller-supplied context for the CA instance
* being created. The HCA driver passes this value to the client's
* event callback functions. If the client will not be using event
* callbacks, this value is unused.
*
* Dispatch
* [in] A pointer to a constant client IB_CA_DISPATCH structure. This
* structure is a dispatch table that contains pointers to the event
* callback functions for the new CA instance. Any members of the
* table can be NULL.
*
* pUserData
* [in] Descriptor for any data exchanged with the user-mode verb
* provider. The input and output buffers are shadowed by the
* access layer in non-paged pool. The HCA driver can only access
* further embedded pointers from the context of this call.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The operation completed successfully. The IoStatus.Information
* field of the IRP contains a pointer to a CA instance structure,
* IB_CA, for the new CA instance.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field of the IRP.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*********/
/****f* Verbs/CiCloseCa
* DESCRIPTION
* Closes an open instance of an HCA.
*
* SYNOPSIS
*/
CiCloseCa(
IN IB_CA* const pCa,
IN OUT CI_UDATA* const pUserData,
IN IRP* const pIrp );
/*
* PARAMETERS
* pCa
* [in] Pointer to an IB_CA structure previously created by a call to
* CiOpenCa.
*
* pUserData
* [in] Descriptor for any data exchanged with the user-mode verb
* provider. The input and output buffers are shadowed by the
* access layer in non-paged pool. The HCA driver can only access
* further embedded pointers from the context of this call.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The CA instance was closed successfully. The IRP will be completed
* with success status.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field of the IRP.
*
* STATUS_DEVICE_BUSY
* The operation could not complete because the CA instance has
* outstanding resources allocated.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*
* Any pending event notification IRPs will be completed with
* STATUS_CANCELLED before the function returns.
*********/
/****f* Verbs/CiGetCaFatalEvent
* DESCRIPTION
* Queues a request for HCA catastrophic error notification.
*
* SYNOPSIS
*/
CiGetCaFatalEvent(
IN IB_CA* const pCa,
IN IRP* const pIrp );
/*
* PARAMETERS
* pCa
* [in] Pointer to an IB_CA structure previously created by a call to
* CiOpenCa.
*
* pIrp
* [in] IRP that will be completed when a local catastrophic error
* occurs on the HCA. The IRP major and minor functions as well as
* the I/O stack must be preserved when the IRP is completed. The
* Irp.RequestorMode can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The CA instance experienced a fatal error and should be closed.
* The IRP will be completed with success status.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field of the IRP.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*
* The HCA driver must set a cancel routine to allow clients to cancel
* the notification request.
*
* A client can queue up mutiple requests for notification. Only one
* request is completed per event. All such requests are completed with
* STATUS_CANCELLED if the IB_CA instance is closed.
*********/
/****f* Verbs/CiGetCaPortEvent
* DESCRIPTION
* Queues a requests for port event notification.
*
* SYNOPSIS
*/
CiGetCaPortEvent(
IN IB_CA* const pCa,
IN const UCHAR PortNumber,
IN IRP* const pIrp );
/*
* PARAMETERS
* pCa
* [in] Pointer to an IB_CA structure previously created by a call to
* CiOpenCa.
*
* PortNumber
* [in] Port number to which the notification request is targetted.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The CA instance experienced a port event. The IoStatus.Information
* member of the IRP is updated with the IB_ASYNC_EVENT value. The IRP
* will be completed with success status.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field of the IRP and the IB_ASYNC_EVENT will be returned in the
* IoStatus.Information field of the IRP.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*
* The HCA driver must set a cancel routine to allow clients to cancel
* the notification request.
*
* A client can queue up mutiple requests for notification. Only one
* request is completed per event. All such requests are completed with
* STATUS_CANCELLED if the IB_CA instance is closed.
*********/
/****f* Verbs/CiQueryCa
* DESCRIPTION
* Queries an HCA for its attributes.
*
* SYNOPSIS
*/
CiQueryCa(
IN IB_CA* const pCa,
IN const SIZE_T AttrSize,
OUT IB_CA_ATTR* const pAttr,
IN IRP* const pIrp );
/*
* PARAMETERS
* pCa
* [in] Pointer to an IB_CA structure previously created by a call to
* CiOpenCa.
*
* pAttrSize
* [in] Size of the attributes buffer referenced by the pAttr parameter.
*
* pAttr
* [out] CA attribute of this Host Channel adapter.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The full CA attributes, including port attributes and their GID
* and PKEY tables, were successfully copied to the attribute buffer.
* The IoStatus.Information field of the IRP is updated to indicate
* the number of bytes returned.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field and the number of bytes returned in the IoStatus.Information
* field of the IRP.
*
* STATUS_BUFFER_OVERFLOW
* The operation completed but returned only partial attributes. Full
* attributes were not returned due to insufficient space. The size
* required to get the full attributes is returned in the Size field
* of the IB_CA_ATTR structure. The IoStatus.Information field of the
* IRP is updated to indicate the number of bytes returned.
*
* STATUS_BUFFER_TOO_SMALL
* The pAttr buffer was too small to return any information. At a
* minimum, the buffer must be the 4 bytes to allow the required
* size to be returned.
*
* STATUS_FILE_FORCED_CLOSED
* The CA is no longer functional. The IRP will be completed with
* failure status. The client must release all resources and close
* the CA as soon as possible.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*********/
/****f* Verbs/CiQueryPort
* DESCRIPTION
* Queries an HCA for one of its port attributes.
*
* SYNOPSIS
*/
CiQueryPort(
IN IB_CA* const pCa,
IN const UCHAR PortNumber,
IN const SIZE_T AttrSize,
OUT IB_PORT_ATTR* const pAttr,
IN IRP* const pIrp );
/*
* PARAMETERS
* pCa
* [in] Pointer to an IB_CA structure previously created by a call to
* CiOpenCa.
*
* PortNumber
* [in] 1-based port number targetted by this request.
*
* pAttrSize
* [in] Size of the attributes buffer referenced by the pAttr parameter.
*
* pAttr
* [out] CA attribute of this Host Channel adapter.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The full port attributes, including the GID and PKEY tables, were
* successfully copied to the attribute buffer.
* The IoStatus.Information field of the IRP is updated to indicate
* the number of bytes returned.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field and the number of bytes returned in the IoStatus.Information
* field of the IRP.
*
* STATUS_BUFFER_OVERFLOW
* The operation completed but returned only partial attributes. Full
* attributes were not returned due to insufficient space. The size
* required to get the full attributes is returned in the Size field
* of the IB_PORT_ATTR structure. The IoStatus.Information field of
* the IRP is updated to indicate the number of bytes returned.
*
* STATUS_BUFFER_TOO_SMALL
* The pAttr buffer was too small to return any information. At a
* minimum, the buffer must be the 4 bytes to allow the required
* size to be returned.
*
* STATUS_FILE_FORCED_CLOSED
* The CA is no longer functional. The IRP will be completed with
* failure status. The client must release all resources and close
* the CA as soon as possible.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*********/
/****f* Verbs/CiModifyCa
* DESCRIPTION
* Modify the attributes of a Channel Adapter.
*
* SYNOPSIS
*/
CiModifyCa(
IN IB_CA* const pCa,
IN const ULONG Flags,
IN const IB_MODIFY_CA* const pModify,
IN IRP* const pIrp );
/*
* PARAMETERS
* pCa
* [in] Pointer to an IB_CA structure previously created by a call to
* CiOpenCa.
*
* Flags
* [in] Combination of IB_MODIFY_CA_FLAGS values indicating which
* fields are valid in the IB_MODIFY_CA structure.
*
* pModify
* [in] Pointer to an IB_MODIFY_CA structure with the fields specified
* by the Flags parameter set to the desired values.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The CA was modified successfully.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field of the IRP.
*
* STATUS_FILE_FORCED_CLOSED
* The CA is no longer functional. The IRP will be completed with
* failure status. The client must release all resources and close
* the CA as soon as possible.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*********/
/****f* Verbs/CiModifyPort
* DESCRIPTION
* Modify the attributes of a Channel Adapter's port.
*
* SYNOPSIS
*/
CiModifyPort(
IN IB_CA* const pCa,
IN const UCHAR PortNumber,
IN const ULONG Flags,
IN const IB_MODIFY_PORT* const pModify,
IN IRP* const pIrp );
/*
* PARAMETERS
* pCa
* [in] Pointer to an IB_CA structure previously created by a call to
* CiOpenCa.
*
* PortNumber
* [in] 1-based port number targetted by this request.
*
* Flags
* [in] Combination of IB_MODIFY_CA_FLAGS values indicating which
* fields are valid in the IB_MODIFY_CA structure.
*
* pModify
* [in] Pointer to an IB_MODIFY_CA structure with the fields specified
* by the Flags parameter set to the desired values.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The CA's port was modified successfully.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field of the IRP.
*
* STATUS_FILE_LOCK_CONFLICT
* The requested capabilities conflict with existing capabilities.
* This can occur if multiple clients try to bits such as
* IB_PORT_CAP_IS_SM or IB_PORT_CAP_CM_SUP bits in the port
* capabilities.
*
* STATUS_FILE_FORCED_CLOSED
* The CA is no longer functional. The IRP will be completed with
* failure status. The client must release all resources and close
* the CA as soon as possible.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*********/
typedef void
(PFN_IB_CQ_COMP_EVENT*)
IN VOID *CqContext );
typedef void
(PFN_IB_CQ_ASYNC_EVENT*)
IN VOID *CqContext,
IN IB_ASYNC_EVENT Event );
typedef struct IB_CQ_DISPATCH
{
PFN_IB_CQ_COMP_EVENT CompEvent;
PFN_IB_CQ_ASYNC_EVENT AsyncEvent;
}
typedef struct IB_CQ
{
IB_CQ_DISPATCH Dispatch;
ULONG NumCqes;
}
/****f* Verbs/CiCreateCq
* DESCRIPTION
* Create a Completion Queue.
*
* SYNOPSIS
*/
CiCreateCq(
IN IB_CA* const pCa,
IN ULONG const NumCqes,
IN VOID* const Context,
IN const IB_CQ_DISPATCH* const Dispatch OPTIONAL,
IN OUT CI_UDATA* const pUserData,
IN IRP* const pIrp );
/*
* PARAMETERS
* pCa
* [in] Pointer to an IB_CA structure previously created by a call to
* CiOpenCa.
*
* NumCqes
* [in] Number of CQ entries requested by the consumer.
*
* Context
* [in] A pointer to a caller-supplied context for the CQ being
* created. The HCA driver passes this value to the client's
* event callback functions. If the client will not be using event
* callbacks, this value is unused.
*
* Dispatch
* [in] A pointer to a constant client IB_CQ_DISPATCH structure. This
* structure is a dispatch table that contains pointers to the event
* callback functions for the new CA instance. Any members of the
* table can be NULL.
*
* pUserData
* [in] Descriptor for any data exchanged with the user-mode verb
* provider. The input and output buffers are shadowed by the
* access layer in non-paged pool. The HCA driver can only access
* further embedded pointers from the context of this call.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The CQ was created successfully. The IB_CQ structure is returned
* in the IoStatus.Information field of the IRP. The IB_CQ structure
* contains the actual number of CQ entries allocated for the CQ.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field and the IB_CQ structure in the IoStatus.Information field of
* the IRP. The IB_CQ structure contains the actual number of CQ
* entries allocated for the CQ.
*
* STATUS_FILE_FORCED_CLOSED
* The CA is no longer functional. The IRP will be completed with
* failure status. The client must release all resources and close
* the CA as soon as possible.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*********/
More information about the ofw
mailing list