RandomizedTests

From APIDesign

Current revision

Most of the ways to test the application code we have discussed are useful when you find out that something is wrong and you want your fix to last and stiffen the amoeba shape closer to the desired look of the application. Tests usually do not help much in discovering differences between the specification and reality. The one exception however are RandomizedTests - they help to test the code in new, unusual ways and thus can discover new and unusual problems of the application code.

Use Randomness

The basic idea is simple, just use a random number generator to drive what your test does. If, for example, if you support operations add and remove use the generator to randomly specify their order and parameters:

Random random = new Random (); 
int count = random.nextInt (10000); 
for (int i = 0; i < count; i++) { 
  boolean add = random.nextBoolean (); 
  if (add) { 
    list.add (random.nextInt (list.size (), new Integer (random.nextInt (100))); 
  } else { 
    list.remove (random.nextInt (list.size ()); 
  } 
}

This will not invent new ways to call your code, just new orders of calls that can reveal surprising problems, because not all combinations of operations have to be anticipated by the programmer and some of them may lead to failures.

Unreproducible!

An important feature of each test is its reproducibility or at least clear failure report. It is fine that we know there is a bug in our code, but if we do not know how to reproduce it, we may not be able to analyze and fix it. The reproducibility of random tests is even more important, as in fact, we do not know the sequence of computations that is really being performed.

The first step to achieve it is to not create a random generator blindly, but use an initial seed which, when passed repeatedly generates the same sequence of numbers. If you look at the implementation of the default Random constructor, you will find out that the initial seed is set to the current time, so we can mimic the behaviour by writing:

private void doRandomOperations (long seed) throws Throwable { 
  Random random = new Random (seed); 
  try { 
    // do the random operations 
  } catch (AssertionFailedError err) { 
    AssertionFailedError ne = new AssertionFailedError ("For seed: " + seed + " was: " + err.getMessage ());
    throw ne.initCause (err); 
  } 
} 
 
public void testSomeNewRandomScenarioToIncreaseCoverage () throws Throwable { 
  doRandomOperations (System.currentTimeMillis ()); 
}

which knows the initial seed and prints as part of the failure message if the test fails. In such case we can then increase the coverage by by adding methods like

public void testThisUsedToFailOnThursday () throws Exception { 
  doRandomOperations (105730909304L); 
}

which exactly repeats the sets of operations that once lead to a failure. We used this approach for example in AbstractMutableLazyListHid.doRandomTest.

One problem when debugging such randomized test is that by specifying one number, a long sequence of operations is defined. It is not easy for most people to imagine the sequence by just looking at the number and that is why it can be useful to provide better output so instead of calling doRandomOperations (105730909304L); one can create a test that exactly shows what is happening in it. To achieve this we can modify the testing code not only to execute the random steps, but also generate a more usable error message for potential failure:

private void doRandomOperations (long seed) throws Throwable { 
  Random random = new Random (seed); 
  StringBuffer failure = new StringBuffer (); 
  try { 
    int count = random.nextInt (10000); 
    for (int i = 0; i < count; i++) { 
       boolean add = random.nextBoolean (); 
       if (add) { 
         int index = random.nextInt (list.size (); 
         Object add = new Integer (random.nextInt (100)); 
         list.add (index, add); 
         failure.append (" list.add(" + index + ", new Integer (" + add + "));\n"); 
       } else { 
         int index = random.nextInt (list.size (); 
         list.remove (index);
         failure.append (" list.remove(" + index + ");\n"); 
       } 
     } 
   } catch (AssertionFailedError err) { 
     AssertionFailedError ne = new AssertionFailedError (
       "For seed: " + seed + " was: " + err.getMessage () + " with operations:\n" + failure
     ); 
     throw ne.initCause (err); 
   } 
 }

which in case of error will generate human readable code for the failure like:

list.add(0, new Integer (30)); 
list.add(0, new Integer (11)); 
list.add(1, new Integer (93)); 
list.remove (0); 
list.add(1, new Integer (34));

We used this technique to generate for example AbstractMutableFailure1Hid.java which is long, but more readable and debuggable than one seed number.

Randomized but not Random

The randomized tests not only help us to prevent regressions in the amoeba shape of our application by allowing us to specify the failing seeds, but also (which is a unique functionality in the testing), it can help to discover new areas where our shape does not match our expectations.