Re: Testing Something that is Random

Ron Jeffries <ronjeffries <at> xprogramming.com> wrote:
>William Tanksley, Jr wrote:

> > This is true, and is both subtle and potentially dangerous (random
> > distributions aren't only used for games!!). I've also seen people
> > accidentally toss a random integer into a signed variable, thereby
> > chopping off the upper half of the range; such an error might never
> > get caught, even though it has enormous effects.

> Really? What is the effect on a random integer distribution of
> eliminating all occurrences of integers containing a 1-bit in one
> given location?

Obviously, half of the range is removed. In this case, the bit was the
high bit, so all the missing events were in the second half of the
event range. In this case, the 'integer' was actually an unsigned
char, so that meant that 128 out of almost 256 events would never
happen. The author tested and released, and just didn't notice that
some events didn't happen. The users noticed because some events were
twice as common as they should have been, and thanks to some other
side effects we were able to trace the cause.

If the app had been using 32-bit random numbers and gotten them
truncated to 31 bits, some apps might never notice -- others might
have tiny probability distortions -- and others might have nothing be
wrong for years, until additions to the number of events brought some
events outside of the accidentally chopped range of the RNG.

> > My point was "those types of mistakes have bitten me often in the
> > past." It's true that in a card shuffle I'm not worried about such a
> > small statistical abnormality, but other applications do worry me, and
> > frankly I'd rather not be bitten ever again in any application.

> Exactly. Your experience is that you have been bitten by RNG issues,
> so you test more extensively. Mine is that my ***, while full of
> many bites, has few bites from RNG, so I test less extensively.
> Seems to me that's as it should be. Is it not?

I don't mind it. If you ever want to do some critical work that
involves random numbers, get in touch with an expert. Until then,
don't stress out about it; that's the expert's job. So I agree with
you there.

OTOH, if someone writes to this newsgroup asking "how do I TDD a
random algorithm?", experience at randomness is going to be worth as
much as experience at TDD, and experience in TDDing randomness is
going to trump both. I'm not slamming your advice; on the contrary,
I'm taking it (you are an expert at TDD, and I am only mildly
experienced). I'm just pointing out that the answer "don't worry about
it" isn't of any value when it comes from a person who has never had
to worry about "it".

Let me quote your (presumably randomly chosen) .sig.

> My advice is to do it by the book, get good at the practices, then do as
> you will. Many people want to skip to step three. How do they know?

As with most of the things you've said, I couldn't have said it
better. This applies to more than XP.

> Ron Jeffries

-Billy

 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/testdrivendevelopment/

<*> To unsubscribe from this group, send an email to:
    testdrivendevelopment-unsubscribe <at> yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/