[openib-general] Re: [PATCH 10 of 18] ipath - support for userspace apps using core driver

[Andrew Morton]

"Bryan O'Sullivan" <bos at pathscale.com> wrote:
>
>  +	/*
>  +	 * This code is present to allow a knowledgeable person to
>  +	 * specify the layout of processes to processors before opening
>  +	 * this driver, and then we'll assign the process to the "closest"
>  +	 * HT-400 to that processor (we assume reasonable connectivity,
>  +	 * for now).  This code assumes that if affinity has been set
>  +	 * before this point, that at most one cpu is set; for now this
>  +	 * is reasonable.  I check for both cpus_empty() and cpus_full(),
>  +	 * in case some kernel variant sets none of the bits when no
>  +	 * affinity is set.  2.6.11 and 12 kernels have all present
>  +	 * cpus set.  Some day we'll have to fix it up further to handle
>  +	 * a cpu subset.  This algorithm fails for two HT-400's connected
>  +	 * in tunnel fashion.  Eventually this needs real topology
>  +	 * information.  There may be some issues with dual core numbering
>  +	 * as well.  This needs more work prior to release.
>  +	 */
>  +	if (!cpus_empty(current->cpus_allowed) &&
>  +	    !cpus_full(current->cpus_allowed)) {
>  +		int ncpus = num_online_cpus(), curcpu = -1;
>  +		for (i = 0; i < ncpus; i++)
>  +			if (cpu_isset(i, current->cpus_allowed)) {
>  +				ipath_cdbg(PROC, "%s[%u] affinity set for "
>  +					   "cpu %d\n", current->comm,
>  +					   current->pid, i);
>  +				curcpu = i;
>  +			}
>  +		if (curcpu != -1) {
>  +			if (npresent) {
>  +				prefunit = curcpu / (ncpus / npresent);
>  +				ipath_dbg("%s[%u] %d chips, %d cpus, "
>  +					  "%d cpus/chip, select unit %d\n",
>  +					  current->comm, current->pid,
>  +					  npresent, ncpus, ncpus / npresent,
>  +					  prefunit);
>  +			}
>  +		}
>  +	}

CPU topology is available in sysfs - it shouild be possible to push policy
decisions like this up to userspace.  If the topology info is insufficient,
we can add to it.