Open Fabrics Enterprise Distribution (OFED) SDP in OFED 1.3 Release Notes February 2008 =============================================================================== Table of Contents =============================================================================== 1. Overview 2. Bug Fixes and Enhancements 3. Known Issues 4. Verification Applications/Flows/Tests =============================================================================== 1. Overview =============================================================================== SDP in OFED is at GA level for OFED 1.3. =============================================================================== 2. Bug Fixes and Enhancements =============================================================================== * Fixes for SDP specification compliance - OOB data not marked as solicited (bug 596) - DisConn, ChRcvBuf, ChRcvBufAck marked solicited (bug 644) - Do not send DisConn if only 1 credit (bug 646) - Validate ChRcvBuf range (bug 647) * Cleanup - Compilation warnings - New kernel support * New function - SIOCOUTQ ioctl support - Add keepalive support - New /sys options: sdp_keepalive_probes_sent, sdp_keepalive_time - New options: SOCK_KEEPALIVE, TCP_KEEPIDLE - Add Zero copy bcopy support (bzcopy) - New /sys option: sdp_zcopy_thresh * Bugs fixed - Resize buffers if out of credits (bug 556) - Resize using skb_put (bug 620) - Move to accept queue on RTU drop and DREQ (bug 645) - Modify memory allocation to support in kernel users - Fix reference count but that prevents driver unload - connect() now allows AF_INET_SDP and AF_INET (bug 294) - poll() always returns POLLOUT on non-blocking socket (bug 829) - Executing netperf with TCP_CORK never ends (bug 837) - Fix bug with FASYNC - libsdp returns wrong error - multi-threaded libsdp client closes wrong socket under load =============================================================================== 3. Known Issues =============================================================================== - Each SDP socket currently consumes up to 2 MBytes of memory. If this value is high for your installation, it is possible to trade off performance for lower memory utilization per socket by reducing the value of the "rcvbuf_scale" module parameter (default: 16). Note: the minimum legal value for this parameter is 1. At this parameter value, each socket will consume approximately 128 KBytes. - Small message size performance is low when messages are sent by client at a rate lower than the rate at which they are consumed by server, and when TCP_CORK is not set. This is observed, for example, with iperf benchmark. As a workaround, set the TCP_CORK socket option to ensure data is sent in at least 32K byte chunks. - Performance is low on 32-bit kernels, as SDP utilizes high memory to ease memory pressure. Moving to a 64-bit kernel solves this problem even if the application remains a 32-bit one. - By default, SDP utilizes a 2 Kbyte MTU size. This may cause PCI-X cards using Mellanox Technologies "Infinihost" HCAs to experience low bandwidth. Workaround: reset the MTU size to 1K in this situation, using either of the two methods below: 1. Activate the "tavor quirk" workaround in opensm: a. Create an opensm options cache file (/var/cache/osm/opensm.opts): > opensm --cache-options -o b. Add the following line to /var/cache/osm/opensm.opts: enable_quirks TRUE c. Rerun opensm using your usual command line options to activate the opensm quirk option. 2. Activate the "tavor quirk" workaround in cma: set the tavor_quirk module parameter of the rdma_cm module to value 1 (default: 0). - The new BZCOPY mode is only effective for large block transfers. By setting the /sys parameter 'sdp_zcopy_thresh' to a non-zero value, a non-standard SDP speedup is enabled. All messages longer than 'sdp_zcopy_thresh' bytes in length will cause the user space buffer to be pinned and the data sent directly from the original buffer. This results in less CPU use and, on many systems, much better bandwidth. The default 64K value for 'sdp_zcopy_thresh' is sometimes too low for some systems. You must experiment with your hardware to select the best value. - Windows interoperability The Windows version of SDP does not support resizing buffers using the standard protocol messages. There will sometimes be inter-operability problems for this reason. =============================================================================== 4. Verification Applications/Flows/Tests =============================================================================== - ssh/sshd - wget/netscape/firefox/apache - netpipe - netperf - LTP socket tests - iperf-2.0.2 - ttcp - Threaded and forking echo client server examples - Various Java client server applications (SUN:jre, BEA:jrockit/WebLogic, GNU:gij/gcj) - Many UNIX utilities to verify that pre-load did not harm the applications ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Open Fabrics Enterprise Distribution (OFED) libsdp v. 9382 in OFED 1.2 Release Notes April 2007 =============================================================================== Table of Contents =============================================================================== 1. Overview 2. New Features 3. Bug Fixes 4. Known Issues 5. Verification Applications/Flows/Tests =============================================================================== 1. Overview =============================================================================== This document describes the contents of the libsdp OFED 1.3 release. libsdp is a LD_PRELOAD-able library that can be used to migrate existing applications to use InfiniBand Sockets Direct Protocol (SDP) instead of TCP sockets, transparently and without recompilations. To setup libsdp please follow the instructions below. The libsdp version for this release is 1.3. =============================================================================== 2. New Features =============================================================================== * Add support for new kernel options - SIOCOUTQ ioctl support - Add keepalive support - New options: SOCK_KEEPALIVE, TCP_KEEPIDLE * Add libsdp-devel subpackage =============================================================================== 3 Bug Fixes =============================================================================== The following list of bugs were fixed. Note that other less critical or visible bugs were also fixed. * Multi-threaded applications that are opening and closing many SDP sockets quickly using the 'both' flow would incorrectly close a valid socket. * Attempt to bind to an invalid socket using 'both' flow returns the wrong errno. * Applications using signal driven IO (FASYNC) on 'both' flow sockets would fail because the second accep() would be executed on the TCP socket. * Update error handling to set errno to a valid error code prior to returning -1. =============================================================================== 4. Known Issues =============================================================================== * libsdp cannot provide its socket switch functionality for executables statically linked with libc. * When using server to listen on both SDP and TCP, the number of sockets is doubled. =============================================================================== 5. Verification Applications/Flows/Tests =============================================================================== See the corresponding section in the SDP release notes above.