[Openib-windows] [RFC] Async kernel verbs: Memory Registration
Fab Tillier
ftillier at silverstorm.com
Tue Sep 20 10:11:38 PDT 2005
Folks,
Here are the definitions for memory registration APIs for the IRP-based direct
call interface. There are three calls:
IbRegVirtMr: Registers virtual memory, and takes as input a list of MDLs
representing the memory region being registered. The memory referenced by the
MDLs will have been pinned before the call is made, so the call only needs to
setup the mapping in the HCA using the page table in the MDL.
IbRegScatterGatherMr: Registers a SCATTER_GATHER_LIST, which are used by
DMA-enabled kernel drivers. NDIS and the StorPort port drivers provide these to
their miniports. They are also available through the DMA_OPERATIONS structure
for a DMA_ADAPTER (IoGetDmaAdapter). This facilitates physical registration by
removing the need to convert OS-provided SGLs into an intermediate format.
IbGetDmaMr: Returns a MR representing all of physical memory.
Let me know your thoughts and comments.
Thanks,
- Fab
typedef struct _IB_MR
{
IB_PD *pPd;
UINT8* __ptr64 pLocalStart;
UINT8* __ptr64 pLocalEnd;
UINT8* __ptr64 pRemoteStart;
UINT8* __ptr64 pRemoteEnd;
ULONG AccessMask;
UINT32 lKey;
UINT32 rKey;
/** NULL for non-virtual registrations */
MDL *pMdlList;
LONG volatile nRef;
} IB_MR;
/****f* Verbs/IbRegVirtMr
* DESCRIPTION
* Registers a virtual memory region with a channel adapter.
*
* SYNOPSIS
*/
NTSTATUS
IbRegVirtMr(
IN IB_PD* const pPd,
IN MDL* const pMdlList,
IN const ULONG AccessMask,
IN IRP* const pIrp );
/*
* PARAMETERS
* pPd
* [in] Pointer to an IB_PD structure previously created by a call to
* CiCreatePd.
*
* pMdlList
* [in] List of MDLs describing the memory region. The MDL's page tables
* are populated and the memory referenced properly pinned. The only
* partial pages referenced by the list of MDLs are the first page of the
* first MDL and the last page of the last MDL. All other pages are
* full pages. The page addresses are CPU addresses, and should be
* properly mapped for DMA access by the HCA driver.
*
* AccessMask
* [in] The local access rights requested forthis registered memory
* region.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The memory region was successfully registered. The IB_MR structure
* is returned in the IoStatus.Information field of the IRP. The IB_MR
* structure contains the actual attributes of the MR.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field and the IB_MR structure in the IoStatus.Information field of
* the IRP. The IB_MR structure contains the actual MR attributes.
*
* STATUS_FILE_FORCED_CLOSED
* The CA is no longer functional. The IRP will be completed with
* failure status. The client must release all resources and close
* the CA as soon as possible.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*****/
/****f* Verbs/IbRegScatterGatherMr
* DESCRIPTION
* Registers a virtual memory region with a channel adapter.
*
* SYNOPSIS
*/
NTSTATUS
IbRegScatterGatherMr(
IN IB_PD* const pPd,
IN SCATTER_GATHER_LIST* pScatterGather,
IN const ULONG AccessMask,
IN IRP* const pIrp );
/*
* PARAMETERS
* pPd
* [in] Pointer to an IB_PD structure previously created by a call to
* CiCreatePd.
*
* pScatterGather
* [in] Scatter/gather list to register. Only the first and last
* elements in the list can reference partial pages. All other elements
* must reference an intergral number of pages. If the list has more
* than one element, the first and last elements in the list must end
* and start on a page boundary, respectively. The physical addresses
* referenced by the scatter/gather list are bus-relative addresses and
* have already been properly mapped for DMA access.
*
* AccessMask
* [in] The local access rights requested forthis registered memory
* region.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The memory region was successfully registered. The IB_MR structure
* is returned in the IoStatus.Information field of the IRP. The IB_MR
* structure contains the actual attributes of the MR.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field and the IB_MR structure in the IoStatus.Information field of
* the IRP. The IB_MR structure contains the actual MR attributes.
*
* STATUS_FILE_FORCED_CLOSED
* The CA is no longer functional. The IRP will be completed with
* failure status. The client must release all resources and close
* the CA as soon as possible.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*****/
/****f* Verbs/IbGetDmaMr
* DESCRIPTION
* Get a memory region representing all of physical memory.
*
* SYNOPSIS
*/
NTSTATUS
IbGetDmaMr(
IN IB_PD* const pPd,
IN const ULONG AccessMask,
IN IRP* const pIrp );
/*
* PARAMETERS
* pPd
* [in] Pointer to an IB_PD structure previously created by a call to
* CiCreatePd.
*
* AccessMask
* [in] The local access rights requested forthis registered memory
* region.
*
* pIrp
* [in] IRP that will be completed when the request completes.
* The IRP major and minor functions as well as the I/O stack must
* be preserved when the IRP is completed. The Irp.RequestorMode
* can be used to distinguish kernel and user clients.
*
* RETURN VALUES
* STATUS_SUCCESS
* The memory region was successfully registered. The IB_MR structure
* is returned in the IoStatus.Information field of the IRP. The IB_MR
* structure contains the actual attributes of the MR.
*
* STATUS_PENDING
* The operation could not be completed immediately. The HCA driver
* will complete the IRP when the operation completes, at which point
* the status of the operation will be returned in the IoStatus.Status
* field and the IB_MR structure in the IoStatus.Information field of
* the IRP. The IB_MR structure contains the actual MR attributes.
*
* STATUS_FILE_FORCED_CLOSED
* The CA is no longer functional. The IRP will be completed with
* failure status. The client must release all resources and close
* the CA as soon as possible.
*
* Other status codes
* An error occurred. The IRP will be completed with failure status.
*
* NOTES
* This function can be invoked at IRQL <= DISPATCH_LEVEL.
*
* The HCA driver must call IoCallDriver for the input IRP so that it
* can call I/O complete request. The target of IoCallDriver must be
* the FDO of the HCA since the IRP will have only a single I/O stack
* location for the HCA to use. This implies that the IRP cannot be
* passed down a chain of layered drivers.
*****/
More information about the ofw
mailing list