<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [Openib-windows] [RFC] IRP-based verbs</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Hi fab,</FONT>
</P>
<P><FONT SIZE=2>Some more questions and comments bellow.</FONT>
</P>
<P><FONT SIZE=2>Thanks</FONT>
<BR><FONT SIZE=2>Tzachi</FONT>
</P>
<P><FONT SIZE=2>>-----Original Message-----</FONT>
<BR><FONT SIZE=2>>From: Fab Tillier [<A HREF="mailto:ftillier@silverstorm.com">mailto:ftillier@silverstorm.com</A>]</FONT>
<BR><FONT SIZE=2>>Sent: Thursday, September 08, 2005 10:00 PM</FONT>
<BR><FONT SIZE=2>>To: 'Tzachi Dar'; openib-windows@openib.org</FONT>
<BR><FONT SIZE=2>>Subject: RE: [Openib-windows] [RFC] IRP-based verbs</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> From: Tzachi Dar [<A HREF="mailto:tzachid@mellanox.co.il">mailto:tzachid@mellanox.co.il</A>]</FONT>
<BR><FONT SIZE=2>>> Sent: Thursday, September 08, 2005 11:28 AM</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> Hi Fab,</FONT>
<BR><FONT SIZE=2>>> Before we give you comments about the proposed change there are a few</FONT>
<BR><FONT SIZE=2>>things</FONT>
<BR><FONT SIZE=2>>> that we would like to :</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> 1) Do you intend that all the kernel mode ULPs to move to an IOCTL based</FONT>
<BR><FONT SIZE=2>>> model? That is do you want the IPOIB model to create an IRP every time it</FONT>
<BR><FONT SIZE=2>>is</FONT>
<BR><FONT SIZE=2>>> going to talk with IBAL?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Initially, this would be deployed just between IBAL and the HCA driver.</FONT>
<BR><FONT SIZE=2>>Over</FONT>
<BR><FONT SIZE=2>>time ULPs would be transitioned to it too.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Speed-path (post/poll) operations would still be direct-call. I'd like to</FONT>
<BR><FONT SIZE=2>>eventually have a direct call interface similar to Microsoft's new kernel</FONT>
<BR><FONT SIZE=2>>socket</FONT>
<BR><FONT SIZE=2>>model (WSK), but that would still take as input an IRP for completion</FONT>
<BR><FONT SIZE=2>>notifications.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>The IOCTL interface is a very complicated interface that is mainly used for communicating between user mode and kernel applications. Since it is so complicated there will probably by wrapper functions around it, so I believe that there will be no need to use the IOCTLs (at all) for communicating in the kernel. As for the user mode, we can create a different library that will be used to communicate with the kernel, it will use IOCTLs but these IOCTLs will be translated to the regular "kernel" interface at a high level. Please note that if we anticipate stress on the calls from user to kernel, than IOCTLs are not the best solution even when passing from user to kernel. We can of course extend this talk, if we see that there is need.</FONT></P>
<BR>
<P><FONT SIZE=2>>> Does it mean that the entire interface will change?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>All verbs that result in command interface calls would be issued via IOCTL.</FONT>
<BR><FONT SIZE=2>>It</FONT>
<BR><FONT SIZE=2>>would require the ULPs to change to allocate, format, and issue the IRPs.</FONT>
<BR><FONT SIZE=2>>It</FONT>
<BR><FONT SIZE=2>>allows using the I/O completion callback processing provided by the OS</FONT>
<BR><FONT SIZE=2>>rather</FONT>
<BR><FONT SIZE=2>>than implementing custom callback mechanisms. Think of it as evolving the</FONT>
<BR><FONT SIZE=2>>stack</FONT>
<BR><FONT SIZE=2>>to be designed for Windows.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>If you look at the windows components that are "available" to the public, you will find out that they don't use IOCTLS to communicate one with each other.</FONT></P>
<P><FONT SIZE=2>>> 2) Currently the hardware driver doesn't support operations at dispatch</FONT>
<BR><FONT SIZE=2>>mode</FONT>
<BR><FONT SIZE=2>>> (that is Create-QP will block bellow the IBAL library). As a result I</FONT>
<BR><FONT SIZE=2>>don't</FONT>
<BR><FONT SIZE=2>>> understand how you are going to establish the goal of allowing all</FONT>
<BR><FONT SIZE=2>>operations</FONT>
<BR><FONT SIZE=2>>> to work from dispatch level.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>The HCA driver will need to be fixed. Requiring all verbs to be issued at</FONT>
<BR><FONT SIZE=2>>passive, especially considering the command interface is asynchronous, is a</FONT>
<BR><FONT SIZE=2>>lousy design and imposes all sorts of restrictions on kernel clients. This</FONT>
<BR><FONT SIZE=2>>design flaw requires ULPs to have passive level threads to perform work -</FONT>
<BR><FONT SIZE=2>>work</FONT>
<BR><FONT SIZE=2>>which may be delayed by I/O completion processing from other clients. It</FONT>
<BR><FONT SIZE=2>>will</FONT>
<BR><FONT SIZE=2>>also enable new functionality in ULPs that do not have a flexible way to</FONT>
<BR><FONT SIZE=2>>get</FONT>
<BR><FONT SIZE=2>>into a passive level thread context.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Specific examples include:</FONT>
<BR><FONT SIZE=2>>- Local mad processing, which currently requires a context switch from the</FONT>
<BR><FONT SIZE=2>>CQ</FONT>
<BR><FONT SIZE=2>>callback. In a busy system, the MAD processing may be delayed to such an</FONT>
<BR><FONT SIZE=2>>extent</FONT>
<BR><FONT SIZE=2>>that the node appears unresponsive to an SM.</FONT>
<BR><FONT SIZE=2>>- Direct Data Buffer Descriptor for SRP, which can't be implemented because</FONT>
<BR><FONT SIZE=2>>all</FONT>
<BR><FONT SIZE=2>>SRP data path entry points are invoked at DISPATCH.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>The current version of the driver doesn't support this. The next version of the driver will also not support this (at least this is the plan for now). How are you going to work over this problem?</FONT></P>
<P><FONT SIZE=2>One more issue to notice is that although the command interface allows for calls to be issued at dispatch level, it doesn't allow infinite requests simultaneously. This requires locks, queues and so on which will move the complication to other places.</FONT></P>
<P><FONT SIZE=2>>> 3) What is the impact on the client that you see from this change? Will</FONT>
<BR><FONT SIZE=2>>it</FONT>
<BR><FONT SIZE=2>>> bring higher BW? Lower latency? Increase of connection rate?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>This will simplify the code base considerably by eliminating context</FONT>
<BR><FONT SIZE=2>>switches</FONT>
<BR><FONT SIZE=2>>currently required to account for a bad HCA driver model (and the reference</FONT>
<BR><FONT SIZE=2>>counting those context switches require). It also creates a common path</FONT>
<BR><FONT SIZE=2>>for</FONT>
<BR><FONT SIZE=2>>kernel and user-mode verb processing, and lets clients decide how they want</FONT>
<BR><FONT SIZE=2>>to</FONT>
<BR><FONT SIZE=2>>process their verbs. In effect, it removes policy imposed by the HCA</FONT>
<BR><FONT SIZE=2>>driver</FONT>
<BR><FONT SIZE=2>>(not the HW!).</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>There should be no negative impact on latency or bandwidth, it just takes</FONT>
<BR><FONT SIZE=2>>advantage of capabilities provided by the OS rather than duplicating</FONT>
<BR><FONT SIZE=2>>functionality, and will allow us to move to a more efficient and smaller</FONT>
<BR><FONT SIZE=2>>code</FONT>
<BR><FONT SIZE=2>>base.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>For IBAL as a client, it allows elimination of the destroy thread, the</FONT>
<BR><FONT SIZE=2>>local mad</FONT>
<BR><FONT SIZE=2>>processing thread, passive threads for verb processing, and the</FONT>
<BR><FONT SIZE=2>>asynchronous</FONT>
<BR><FONT SIZE=2>>destroy callback mechanism to name a few (I'm probably missing others).</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>For the HCA driver, it eliminates the protection context abstraction by</FONT>
<BR><FONT SIZE=2>>allowing</FONT>
<BR><FONT SIZE=2>>the driver to use the RequestorMode of the IRP, and should allow further</FONT>
<BR><FONT SIZE=2>>cleanup</FONT>
<BR><FONT SIZE=2>>of that driver.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>The change to return a busy status rather than cascading destructions</FONT>
<BR><FONT SIZE=2>>further</FONT>
<BR><FONT SIZE=2>>eliminates code from IBAL once ULPs transition to this new interface.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>I believe that although the new model will allow removing some of the threads that you have mentioned there will probably be other instead of that. I also believe that using reference count on the objects will help making the destruction of objects simpler. We have to make sure that we have a complete design before we start coding.</FONT></P>
<P><FONT SIZE=2>>- Fab</FONT>
</P>
</BODY>
</HTML>