[ofiwg] Travis CI testing of libfabric and fabtests

[Jeff Squyres (jsquyres)]

On Oct 12, 2015, at 2:50 PM, Hefty, Sean <sean.hefty at intel.com> wrote:
> 
>> I'm not sure I understand the distinction...?
> 
> I'm referring to a script that is *stored in the upstream github tree*.  This is an 'ofiwg' script, not a vendor script.  Today that builds the sockets provider and runs all fabtests against it.

OIC.

> We won't be able to run the other providers, but we should be able to build them.  

Well, maybe.

Are you saying that everyone who does the CI testing has to use the upstream Travis config?  I don't think we want to do that.

E.g., I don't anticipate downloading and installing PSM, PSM2, uGNI, ...etc. dependent libraries.  I'll test what I can easily test, and add that to the union of what other people are testing.  I.e., as a *project* we want to ensure that all of those providers can compile and install, but you don't necessarily need to require that *everyone* compile/install *all* providers.  You only care that the union of all testing covers all providers.

I think the best bet would be to define a baseline of what you want people to test as part of the github CI stuff.  E.g., run the fabtests.  If they run exactly those guidelines, great.  If they do more than that, that's also great.

For example, my tests will likely be some variation of running the fabtests, plus perhaps a few other minor sanity checks to ensure that usnic is working properly.

> The question is what policy to use in selecting the external libraries.  That's the input I'm looking for, rather than just picking some policy on my own.

I still think that this is a decision best left to the individual testing sites.

I.e., I don't know if that travis.yml file will be suitable for everyone without any local editing for their local setup.

> Why does this matter?  It matters because this is run as soon as the pull *request* is opened.  If the test fails, the PR is flagged with the failure.  Github allows us to prevent merging PRs which have failed the pre-merge checks.  Today they are only flagged.

I saw that Github recently added this, too.  Based on our experience at the Open MPI github, false positives happen more than you would hope/like.  :-\  I.e., a Jenkins goes wonky at some particular site for whatever reason -- either the test incorrectly fails, or someone accidentally changed something at that testing site, or ...one of a hundred other reasons.

My $0.02 would be to not enable that new Github feature until you can get some confidence in everyone's CI testing.

-- 
Jeff Squyres
jsquyres at cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/