Matthew Butt

Identical random values in parallel NUnit test assemblies

Posted in programming by bnathyuw on 16 November 2017

We have some NUnit cross-system test assemblies that run in parallel. They upload files and watch to see how they are processed. We were inserting a randomly generated value into the filenames in an attempt to avoid identically named files overwriting each other.

Unfortunately, this didn’t work, and we saw frequent test failures. In particular, we found that filenames generated by one test assembly were often identical to those generated by another.

I wanted to understand why this was happening, so I did some investigation.

We were getting our values fromTestContext.CurrentContext.Random, which is an instance of NUnit’s Randomizer class.

When we look at the implementation of Randomizer, we see this:

static Randomizer()
{
    InitialSeed = new Random().Next();
    Randomizers = new Dictionary<MemberInfo, Randomizer>();
}

The Randomizer is statically seeded with a value generated by the System.Random class. Because this seed is static, the same sequence of random values is shared by all references to Randomizer within each assembly, producing a sequence of values that are highly likely to be different from each other. However, references to Randomizer in a concurrently running assembly use a different instance and have their own sequence of values with its own seed.

Let’s have a quick look at how System.Random is seeded.

The MSDN documentation tells us:

  • The Random() constructor uses the system clock to provide a seed value. This is the most common way of instantiating the random number generator.

And goes on to warn:

…because the clock has finite resolution, using the parameterless constructor to create different Random objects in close succession creates random number generators that produce identical sequences of random numbers.

It would seem that our two parallel test assemblies often start executing within such a short interval of each other, and that NUnit’s Randomizer is seeded with the same value in each assembly, which means the sequences of values are identical.

There is some discussion of introducing a mechanism for controlling the seeding of Randomizer in NUnit, but in the mean time, the solution to our problem was to seed our own System.Random instances, rather than relying on NUnit’s.

Tagged with: , ,

Watch out for exceptions in NUnit TestFixture constructors

Posted in programming by bnathyuw on 16 February 2015

Don’t do anything that might throw an exception in a TestFixture constructor: your tests will be ignored, rather than failing.

My colleague Chris and I have been playing around with polymorphous testing, taking advantage of the fact that you can pass parameters to an NUnit TestFixture like this:

[TestFixture(parameter1)]
[TestFixture(parameter2)]
public class BazTests{
  public BazTests(Blah parameter){
    ...
  }
}

We wanted to use these parameters to pass in a collaborator, which would then enable us to run the same tests against different implementations of the same interface. However, we immediately encountered difficulties, as the TestFixture attribute will only take parameters that are compile-time constants, which rules out directly passing in a collaborator class. If you are adding parameters to a Test, you can use the TestDataSource attribute to pass in more complex arguments; however, no such attribute is available for TestFixtures.

To get round this, we created a static factory that would then create our dependency according to the enum value passed in:

[TestFixture(TestSetting.Foo)]
[TestFixture(TestSetting.Bar)]
public class BazTests{
  private ITestHelper _testHelper;
  
  public BazTests(TestSetting testSetting){
    _testHelper = TestHelperFactory.Create(testSetting);
  }
}

However, some of our implementations of ITestHelper require a connection string, and when it is not present, a NullReferenceException is thrown. We would have expected some of our tests to fail because of this, but instead they were just ignored, and as TeamCity disregards ignored tests, this meant that our builds were treated as successful.

It suspect the tests are ignored because exceptions in the test constructor are not sent to TeamCity as errors; rather the exception must happen in one of the attribute-decorated methods.

The code gave us the correct failures once we had rewritten it like this:

[TestFixture(TestSetting.Foo)]
[TestFixture(TestSetting.Bar)]
public class BazTests{
  private TestSetting _testSetting;
  private ITestHelper _testHelper;

  public BazTests(TestSetting testSetting){
    _testSetting = testSetting;
  }

  [TestFixtureSetUp]
  public void TestFixtureSetUp(){
    _testHelper = TestHelperFactory.Create(_testSetting);
  }
}
Tagged with: , , , ,