[ofa-general] Re: [Ips] Calculating the VA in iSER header

[Pete Wyckoff]

dorfman.eli at gmail.com wrote on Wed, 16 Apr 2008 11:22 +0300:
> According to Mike's explanation below it seems that we have a bug in
> iSER initiator.
> Fixing this bug will require a fix in the stgt iSER code.
> 
> The problem is that the initiator send a VA which already includes an
> offset for the unsolicited data (which is wrong).
> In iser_initiator.c::iser_prepare_write_cmd the code looks like this:
> hdr->write_va   = cpu_to_be64(regd_buf->reg.va + unsol_sz);
> 
> we think that it should be modified to:
> hdr->write_va   = cpu_to_be64(regd_buf->reg.va);
> 
> Let's discuss this and verify we interpret the spec correctly.
> If agreed we will send a patch.

Agree with the interpretation of the spec, and it's probably a bit
clearer that way too.  But we have working initiators and targets
that do it the "wrong" way.

The transition involved in fixing both sides will lead to problems.
How does a target detect an unfixed initiator and vice versa?  A
mismatched pair will lead to data corruption.

We could address this in a few ways:

1. Flag day: all initiators and targets change at the same time.
Will see data corruption if someone unluckily runs one or the other
using old non-fixed code.

2. Rewrite the IB Annex to codify what's done in practice, and don't
"fix" any code.

3. Start using the Hello messages and extend them to specify if the
VA marks the start of the buffer or the unsol offset.

I really don't look forward to the bug reports we'll get from a
flag da approach.  Old linux versions tend to hang around for a
very long time, and people are often reluctant to upgrade.

		-- Pete

> 2008/4/15 Mike Ko <mako at almaden.ibm.com>:
> >
> > VA is a concept introduced in an Infiniband annex to support iSER.  It
> > appears in the expanded iSER header for Infiniband use only to support the
> > non-Zero Based Virtual Address (non-ZBVA) used in Infiniband vs the ZBVA
> > used in IETF.
> >
> > "The DataDescriptorOut describes the I/O buffer starting with the immediate
> > unsolicited data (if any), followed by the non-immediate unsolicited data
> > (if any) and solicited data."  If non-ZBVA mode is used, then VA points to
> > the beginning of this buffer.  So in your example, the VA field in the
> > expanded iSER header will be zero.  Note that for IETF, ZBVA is assumed and
> > there is no provision to specify a different VA in the iSER header.
> >
> > Tagged offset (TO) refers to the offset within a tagged buffer in RDMA Write
> > and RDMA Read Request Messages.  When sending non-immediate unsolicited
> > data, Send Message types are used and the TO field is not present.  Instead,
> > the buffer offset is appropriately represented by the Buffer Offset field in
> > the SCSI Data-Out PDU.  Note that Tagged Offset is not the same as write VA
> > and it does not appear in the iSER header.
> >
> > Mike
> >
> >  Erez Zilber <erezz at voltaire.com>
> > Sent by: ips-bounces at ietf.org
> >
> > 04/15/2008 06:40 AM
> >
> > To ips at ietf.org
> >
> > cc
> >
> > Subject [Ips] Calculating the VA in iSER header
> >
> > We're trying to understand what should be the write VA (tagged offset)
> >  in the iSER header for WRITE commands. If unsolicited data is to be
> >  sent, should the VA be the original VA or should it be original VA +
> >  FirstBurstLength?
> >
> >
> >  Example:
> >
> >
> >  InitialR2T=No
> >
> >  FirstBurstLength = 1000
> >
> >
> >  Base address of the registered buffer = 0
> >
> >
> >  Now, what should be the VA in the iSER header? 0 or 1000?
> >
> >
> >  We read the following paragraph in the iSER spec, but didn't get an
> >  answer from there:
> >
> >
> >  * If there is solicited data to be transferred for the SCSI write or
> >  bidirectional command, as indicated by the Expected Data Transfer
> >  Length in the SCSI Command PDU exceeding the value of
> >  UnsolicitedDataSize, the iSER layer at the initiator MUST do the
> >  following:
> >
> >  a. It MUST allocate a Write STag for the I/O Buffer defined by
> >  the qualifier DataDescriptorOut. The DataDescriptorOut
> >  describes the I/O buffer starting with the immediate
> >  unsolicited data (if any), followed by the non-immediate
> >  unsolicited data (if any) and solicited data. This means
> >  that the BufferOffset for the SCSI Data-out for this
> >  command is equal to the TO. This implies that a zero TO
> >  for this STag points to the beginning of this I/O Buffer.