[ofa-general] Re: multiple threads posting to the same QP

[Or Gerlitz]

Roland Dreier wrote:
>  > Also do we actually want locking in the fast posting path? for example
>  > is it legal to call send(2) on the same socket fd from two threads?
> 
> Yes, I think every call must be fully thread safe, for a few reasons.
> First, if we try to make some calls not thread safe then we will
> undoubtedly has application authors creating races and reporting
> strange bugs.  Second, pushing the locking to the low-level driver
> actually allows smarter locking to be used -- cf the slightly tricky
> way that mthca/mlx4 lock CQs during QP destroy to avoid taking the QP
> table lock during poll CQ operations.
> 
> I guess it would be possible to compile a special driver library with
> all pthread calls stubbed out, for use in single-threaded
> applications, but I'm not convinced it's worth it.

Looking on the mthca and mlx4 low-level libraries I realize that you use 
pthread_spin_lock/unlock for thread safeness.

Can you spare few words on why spinning is used rather then sleeping (eg 
pthread_mutex_lock/unlock) - is it since you assume that:

A) if the lock is not contended - both calls have the same efficiency
B) if the lock is contended - it would be such for --short-- time and 
hence spinning is more efficient then sleeping (no context-switch etc)

Assuming that the locking scheme of the libraries does not introduce any 
notable overhead for single thread runs, I agree there's no need to 
provide single threaded instances as well.

> (And BTW, yes, it is possible to call send(2) in any racy way you want
> on the same FD, and the kernel's internal state will not get messed
> up)

is it documented any where? other then the kernel state, what happens if 
send(2) is called from two threads on a datagram socket? will the two 
datagrams be accepted in the remote side uncorrupted? as for stream 
socket, since the bits order in the stream is not defined in the sender
side, I "corruption" would surely be experienced by the remote side.

Or.