Re: Test case modularity

Alexandre Ferrieux wrote:
> This means that every extension of an ensemble in the core needs
> fixing test cases that are completely unrelated to the new subcommand:
> tests expecting the message
> 
>     bad subcommand "foo": must be a, b, c, or d.
> 
> must be updated to check for
> 
>     bad subcommand "foo": must be a, b, c, d, or e.
> 
> Is there a good reason to do so ? Why not [regexp {^bad subcommand}] ?

Let's take a step back away from implementation details to principles.

A test ought to exist for a purpose.  It ought to be testing something.

A test ought to be associated with some code getting tested.  The Tcl
test suite tends to follow the convention that tests in a file named
something like foo.test are for testing the code in tclFoo.c.

So if your test passes or fails based on the exact contents of an error
message, you are testing the code that constructs that error message,
whether that's your intent or not.

Thus, the text of an error message ought to be tested *at most* by
tests associated with the source code file where the content
of the error message is determined.  It makes no sense to test the
contents or formatting of an error message anywhere else.  All it does
is make the tests fragile against changes in code that in principle is
not being tested.

It may not be worthwhile to test error message contents at all, but that's
a judgment for the maintainer of the tclFoo.c file to make.

Within that limited context, yes, it makes sense to use regexp or
glob matching to check the basic formatting of the error message without
checking the entire contents in a way that requires maintenance.  It
may make sense for a very limited number of tests to check the whole
subcommand list (along with the associated maintenance) if the maintainer
has reason to guard against errors or inconsistencies in that part of
the code.  There's no need for more tests than cover the cases though.

--

-- 
| Don Porter          Mathematical and Computational Sciences Division |
| donald.porter@...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08