Re: Updating Tktest suite - GSoC project

Ania Pawelczyk wrote:
>     I think you should establish for yourself and for the sake of others
>     trying to offer help a clear sense of what you want to achieve. (DGP)
> 
> 
> I think the same! I thought about taking this as my task:
> _______________________________________________________________
> 1. Firstly bringing all tests to tcltest 2.0 syntax
>  >to be sure that the failure is not caused by dependences.
>  >to provide legibility, cohesion and get rid of
> 
>     a great mess with the tcltest package.  (Rolf Ade)
> 
>  >to learn the tests, which would be helpful in the next project step 
> that would be:
> 2. Eliminating situations when the test fails because there were made 
> wrong assumptions.
> (I don't know how much of this I'll manage to do).
> _______________________________________________________________

I said this to Ania in a separate email, but I'll reiterate here because 
it represents the value to not only the Tcl/Tk core community, but to 
the student working on it (in this case Ania).

While this may seem minimal with just 2 points, you would break down the 
reasoning and value into multiple points.

It's not just having the new test structure format, it is taking 
advantage of what that provides.  Improved clarity, separation of tests, 
robustness to the test suite, etc.

On the 2nd, you learn a lot about the language, test automation 
principles, platform discrepancies and more.

GSOC is just a summer commitment.  Consider that the tests suite for Tk 
is over 10 years of amalgamated stuff.  Having the opportunity for 
someone to give it an update with consistent style and purpose is very 
valuable.

>     The real problems with Tk's test suite are that it is highly
>     dependent on the platform that you test on (i.e. it's clean on only one
>     or two machines in the whole world!) and that it often doesn't check for
>     what *should* be the answer, but rather just what *is* the answer.
>     Fixing those things (or at least making a good step on them) will be a
>     good task. (DKF, some time ago, during application time)

This is sadly true.  I don't expect you to fully succeed in the time 
alloted to fully harden the test suites for the variety of platforms it 
could be run on.  However, moving consistently towards that path should 
provide a framework against which to improve the longevity and reduce 
the brittleness overall.

> Q:  ...trying to deal with the above actually bases on finding failing 
> test and then (if there is no wrong assumption in widget's configuration 
> and if the failure isn't caused by global dependence) I can find the 
> problem solution it in the sources?

If it isn't, don't hesitate to ask.

> As for
> 
>     2) There are tests that fail, which do imply an actual bug in Tk,
>     but don't clearly point to just where that bug is.
> 
> Q: I'm not sure what I should do in this case... Is it possible to give 
> an example of this situation and it's solution?

Find a 'knownBug' case.  Some may be registered in our bug db, some may 
predate the SF bug db, some may already be fixed and not verified, ...

> Q:  Which pattern is the most suitable?
> Q:  Do pack widgets? In the original tests they're sometimes packed. 
> Keep that the same, pack always or resign from displaying?

Sometimes pack is required to force the widget to consider geometry 
calculations that determine size.

> 1.2 Configuring widget
> In the originals there were added options to database. In fact, they had 
> no influence on any test result. They were, as described in button.test
>     # Create entries in the option database to be sure that geometry options
>     # like border width have predictable values.
>     option add *Button.borderWidth 2 [...]
> Q:  Keep those options set while creating a widget? (they don't have any 
> influence)

This is meant to enforce consistency on platforms that may have 
alternate default parameters.  For example, this way we know borderwidth 
will always be 2 for geom calculations.  Otherwise you are having to 
give variant answers depending on the platform, all because of a variant 
borderwidth default that really isn't pertinent to the test case.

> 1.3 cleanup - destroy
> I destroy widgets even if they have no right to be created.
> test button-2.7 {ButtonCreate - bad window name} -body {
>     button foo
> } -cleanup {
>     destroy foo
> } -returnCodes {error} -result {bad window path name "foo"}
> Q:  Do so?

A taste choice IMO, as long as you are clean for the next test.

> 1.4 cleanup - unset
> I've noticed that in cleanup variables should be unset, as
> p.e. when I run button-3.55 and  unsetting in button-3.54 wasn't done
> - it was passed  when I run as a single test case
> - it failed when I run all tests,
> Q: Is that thinking correct? or maybe do unsetting and destroying, (with 
> catching errors) in the setup?

This has blown test cases before (especially an array 'foo' that was 
later referenced as a scalar).  Clean is good, but you often balance 
effort against value.

> 2. Test cases
> 2.1 I split the test cases that both catch an error and obtain 
> informations (like winfo, errorInfo) to detect an error by -returnCodes.
> Previous:
>     test message-2.3 {Tk_MessageObjCmd procedure} {
>         list [catch {message .s -gorp dumb} msg] $msg [winfo child .]
>     } {1 {unknown option "-gorp"} {}}
> Idea:
>     test message-2.4 {Tk_MessageObjCmd procedure} -body {
>         message .s -gorp dump
>     } -returnCodes {error} -result {unknown option "-gorp"}
>     test message-2.5 {Tk_MessageObjCmd procedure} -body {
>         catch {message .s -gorp dump} 
>         winfo child .
>     } -result {}
> Q:  Appropriate?

Yes, this is indeed a key feature in tcltest 2.0, that you can separate 
return info from error code.

> 2.2 Checking if no error
> Q:  What to choose?

It's OK to have no result.  Some things test crash situations - we just 
hope not to crash.  ;)

> 3. Descrptions
> Q:  As there're the same test cases in diffrent tests, I thought about 
> unifying the way that cases are described. Only oes that matter? (I 
> haven't done that yet).

Not a major item.

I believe all the rest were answered.

Regards,

Jeff

-------------------------------------------------------------------------
Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
Studies have shown that voting for your favorite open source project,
along with a healthy diet, reduces your potential for chronic lameness
and boredom. Vote Now at http://www.sourceforge.net/community/cca08