18 May 2012 04:27
Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)
Ojan Vafai <ojan <at> chromium.org>
2012-05-18 02:27:19 GMT
2012-05-18 02:27:19 GMT
On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak <mjs <at> apple.com> wrote:
On May 17, 2012, at 1:42 PM, Ojan Vafai <ojan <at> chromium.org> wrote:On Thu, May 17, 2012 at 1:37 PM, Peter Kasting <pkasting <at> chromium.org> wrote:On Thu, May 17, 2012 at 1:34 PM, Ojan Vafai <ojan <at> chromium.org> wrote:2. Make outcomes optional. If they are left out, then the test is skipped (unless the test is marked SLOW, in which case it's expected to pass). There is no SKIP modifier.I don't think we should do this. It seems very subtle. I'd rather be explicit.I'm OK with the rest of your numbered proposals.I disagree, but I'm fine with punting this to the list of controversial changes that we should discuss separately. FWIW, my main motivation here is that it allows us to unify the Skipped file format with the test_expectations.txt format. But again, we can discuss that separately.
Adding SKIP (or whatever) to every line of skipped files is not a big hurdle, I think we could live with that is a transitions tep. I think the bigger hurdle is supporting chaining across multiple directories.
That's great. I don't think anyone is opposed to adding chaining and I think that's on Dirk short-list of todos.
The only potentially tricky thing here is figuring out what the platform modifiers mean for non-Chromium ports, e.g. I imagine Qt will want similar modifiers to Chromium (mac, linux, win, debug, release, etc). But I think the difficulty here is more in getting the python code right than agreeing on what the correct behavior is.
Also, currently the test_expectations.txt format requires either a bug number or a bug(ojan) entry. Would that be OK with you too? It has proven really good historically for keeping track of why a test was added to the file and for keeping track of getting the tests fixed (or, more importantly, having someone responsible for following up on it), but we could easily restrict this requirement to the Chromium expectations file if other ports dislike it.
I think with those three things and https://bugs.webkit.org/show_bug.cgi?id=86796 addressed, then the formats will be unified and the only thing to bikeshed over is the filename. :)
_______________________________________________ webkit-dev mailing list webkit-dev <at> lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev