[openib-general] [ANNOUNCE] Contribute RDS(ReliableDatagramSockets) to OpenIB

[Nitin Hande]

Michael Krause wrote:
> At 10:28 AM 11/9/2005, Rick Frank wrote:
> 
>> Yes, the application is responsible for detecting lost msgs at the 
>> application level - the transport can not do this.
>>  
>> RDS does not guarantee that a message has been delivered to the 
>> application - just that once the transport has accepted a msg it will 
>> deliver the msg to the remote node in order without duplication - 
>> dealing with retransmissions, etc due to sporadic / intermittent msg 
>> loss over the interconnect. If after accepting the send - the current 
>> path fails - then RDS will transparently fail over to another path - 
>> and if required will resend / send any already queued msgs to the 
>> remote node - again insuring that no msg is duplicated and they are in 
>> order.  This is no different than APM - with the exception that RDS 
>> can do this across HCAs.
>>  
>> The application - Oracle in this case - will deal with detecting a 
>> catastrophic path failure - either due to a send that does not arrive 
>> and or a timedout response or send failure returned from the 
>> transport. If there is no network path to a remote node - it is 
>> required that we remove the remote node from the operating cluster to 
>> avoid what is commonly termed as a "split brain" condition - otherwise 
>> known as a "partition in time".
>>  
>> BTW - in our case - the application failure domain logic is the same 
>> whether we are using UDP /  uDAPL / iTAPI / TCP / SCTP / etc. 
>> Basically, if we can not talk to a remote node - after some defined 
>> period of time - we will remove the remote node from the cluster. In 
>> this case the database will recover all the interesting state that may 
>> have been maintained on the removed node - allowing the remaining 
>> nodes to continue. If later on, communication to the remote node is 
>> restored - it will be allowed to rejoin the cluster and take on 
>> application load. 
> 
> 
> 
> Please clarify the following which was in the document provided by Oracle.
> 
> On page 3 of the RDS document, under the section "RDP Interface", the 
> 2nd and 3rd paragraphs are state:
> 
>    * RDP does not guarantee that a datagram is delivered to the remote 
> application.
>    * It is up to the RDP client to deal with datagrams lost due to 
> transport failure or remote application failure.
> 
> The HCA is still a fault domain with RDS - it does not address flushing 
> data out of the HCA fault domain, nor does it sound like it ensures that 
> CQE loss is recoverable.
> 
> I do believe RDS will replay all of the sendmsg's that it believes are 
> pending, but it has no way to determine if already sent sendmsgs were 
> actually successfully delivered to the remote application unless it 
> provides some level of resync of the outstanding sends not completed 
> from an application's perspective as well as any state updated via RDMA 
> operations which may occur without an explicit send operation to flush 
> to a known state.  
If RDS could define a mechanism that the application could use to 
inform the sender to resync and replay on catastrophic failure, is 
that a correct understanding of your suggestion ?

I'm still trying to ascertain whether RDS completely
> recovers from HCA failure (assuming there is another HCA / path 
> available) between the two endnodes
Reading at the doc and the thread, it looks like we need src/dst port 
for multiplexing connections, we need seq/ack# for resyncing, we need 
some kind of window availability for flow control. Are'nt we very 
close to tcp header ? ..

Nitin

.
> 
> Mike
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general