[Openib-windows] IPOIB virtualization What was already done, what still has to be done to finish the job.

[Fabian Tillier]

On 21 Apr 2006 14:53:42 -0400, Hal Rosenstock <halr at voltaire.com> wrote:
> Hi Fab,
>
> On Fri, 2006-04-21 at 13:31, Fabian Tillier wrote:
> > Hi Hal,
> >
> > On 21 Apr 2006 09:34:27 -0400, Hal Rosenstock <halr at voltaire.com> wrote:
> > > Hi Tzachi,
> > >
> > > On Fri, 2006-04-21 at 09:23, Tzachi Dar wrote:
> > > > Hi Fab,
> > > >
> > > > The following mail summarizes the place the work that I did on
> > > > IPOIB virtualization, that is running IPOIB on Microsoft virtual
> > > > server R2.
> > > >
> > > > Please note that the current status is that ping works in all
> > > > directions, still there is a lot of work needed in order to bring it
> > > > to product quality. The biggest issue that still has to be done is
> > > > allow for packets that are bigger than 1500 bytes, and smaller than
> > > > 2048 to pass to the guest OS. Currently, I have implemented a hack
> > > > that tells windows that we only support MTU of 1500 bytes (like
> > > > Ethernet). My change assumes that all machines are windows machines,
> > > > and all have my changes, but this is not always true. One example that
> > > > breaks this assumption is Linux.
> > >
> > > Is this issue with Windows or Linux in terms of this interoperation ?
> > > Can you elaborate on this ?
> >
> > This is a Windows VM issue.
> >
> > A Windows VM can only receive ~1500 byte packets from the Windows host
> > machine.  This means that if a packet is sent using the full IPoIB MTU
> > to a guest VM, that packet will not ever make it - it gets dropped
> > somewhere between being handed off to the host network stack and the
> > guest OS.  I would expect this to be a bug in the MS virtual server
> > network emulation layer, but we haven't confirmed this yet.
> >
> > So a Linux system sending a full IPoIB MTU packet to a Windows VM
> > would not work, through no fault of the Linux machine (the same
> > applies if a Windows machine sends a full IPoIB MTU packet).
> >
> > To prevent Windows from sending a full IPoIB MTU, the IPoIB driver
> > must report its MTU to Windows as 1500 bytes, rather than 2044, but
> > then any full IPoIB MTU packet (from a Linux host for example) would
> > overrun the RQ WQE.
>
> Then it sounds like when interoperation with Linux or any other OS which
> uses the standard IPoIB MTU is desired, the IPoIB interface needs to be
> ifconfig'd down to 1500 and it would work, right ?

Yes, that should work no problem.  Still, I think the issue is with
Microsoft's virtual server.  When I have some time I'll be looking
into it so that we can use the full MTU - the last thing IPoIB needs
is to use a suboptimal MTU.

- Fab