David Fletcher | 10 Dec 15:25 2003

RE: Future release strategies (was Re: latest director code changes)


>>>>> "WF" == William Fulton <William.Fulton <at> ubs.com> writes:

[ snip ]

WF> ... I'd just like to bring up the subject of the number of
WF> testcases. The time the test-suite takes to build is becoming a
WF> problem. [1] It takes a long time not necessarily due to the grand
WF> sum of lines to parse by SWIG and the compiler, but the time it
WF> takes for each app to start, create or load the shared objects
WF> with their overhead. [2] Basically, it would be a lot faster if we
WF> had one great big testcase!  Dave recently consolidated the C
WF> testcases by merging testcases and I think we should try and have
WF> a go at merging the C++ tests too.  More importantly, when adding
WF> new tests, add it to an existing testcase if possible. [3]

Here are some comments for the items I have marked:

1. The time it takes to build the test case.

   I've been using distcc (distributed compilation) for some time, and
   I've found that the tool works very well, at least for my platform
   (gcc/g++ & linux).  I hear that distcc works well for other
   platforms, too.  (Dave - OS X is supported, and I believe that a
   windows port is coming.)  See:


   This won't distribute SWIG --- only the compilation of swig, or
   compilation of the code produced by swig --- so this isn't a
   complete solution.  But, it can help significantly.  I know that
   we've reduced the time to compile large systems (such as our own,
   or Qt, for example) dramatically.

2. Running test cases.

   It sounds like the time to run the test cases is the issue, really.
   In this case, I don't have an obvious solution, other than to say:

	- There are tools to help speed up loading.  For example, see:


	- Having a central machine set up to run all of the tests (in
	  parallel) would be useful.  I know there are entire systems
	  of software devoted to software testing, and some of these
	  may have solutions to obtain faster runtimes (or parallel
	  operation).  For example, a quick query for "testing" at
	  sourceforge yielded a slew of entries.  Some of these are:


	  A query from google shows a bazillion more entries.
	  Fortunately, there are a number of "roll up" pages that
	  provide links to a number of software test tools.  For
	  example, see


	- Setting up the test machine(s) to run jobs in parallel may
	  help you out.  This is something that only a few will be
	  able to accomplish, which is why I suggest a central testing

	  For linux boxes, at least, you can get parallel execution
	  fairly easily, transparently.  See:


	  Of course, there are other, less-invasive solutions.  Check
	  out the plethora of documents on grid computing (e.g.,
	  globus, Sun's grid engine, ...)

	  Anticipating Dave's comments - yes, this would take some
	  time to set up, but it would only have to be set up once.

	  It sounds like the folks from kitware (CMake's authors) are
	  willing to help with this.  Check out their web page - they
	  seem to know a thing or two about parallel processing, and
	  it sounds like they have machine and people resources to
	  devote to this problem.

3. The only difficulty with overloading tests as you describe is that
   you may be masking or combining problems, making it harder for
   developers to quickly find and fix the problems.

   Carefully consider the suggestion to combine test cases before
   everyone runs off to do this.  You may make test cases run faster,
   but you may also make the job of finding problems and fixing code
   harder for developers.  Which is better? to force a developer to
   wait for a while, or to make a job more difficult?  (Machines
   generally have been getting faster, and more systems are moving to
   parallel execution.)

   At the very least, I suggest you create (written) guidelines for
   tests before you start refactoring.



David Fletcher                          Tuscany Design Automation, Inc.
frodo <at> tuscanyda.com                     5875 S. Danube Circle
303.690.4309                            Aurora, CO 80015-3169 USA