Time and Tests (and xUnit)

May 13, 2020 6 minutes

Time flies like an arrow; fruit flies like a banana.
– Anthony G. Oettinger

Testing time-related logic is not a new problem, but solutions vary from language to language, and even from framework to framework.

There are two main categories of time-related logic:

Getting the current time
Waiting for a certain amount of time

This post covers how to test both in .NET Core, using xUnit and good ol’ OOP.

Current Time

Using DateTimeOffset.Now is an example of getting the current time.

Examples of logic that require getting the current time are the following:

Determining if a authentication token is expired.
Calculating the age of someone based on birthdate.
Displaying either “good morning” or “good afternoon” in a dashboard page.

Using DateTimeOffset ties you to the system time.

For example, if you had a wanted to test the greetings, both tests could never pass in the same testing session; it can’t be morning and afternoon at the same time on the same place in the real world.

Fortunately, the solution is easy enough:

public interface ITime
{
    DateTimeOffset Now { get; }
}

public class SystemTime : ITime
{
    public DateTimeOffset Now => DateTimeOffset.Now;
}

We implement an interface on which time-sensitive logic will depend on, and inject the SystemTime class as its implementation during runtime.

If we want to test what happens at a specific time, we could use the following:

public class FrozenTime : ITime
{
    public DateTimeOffset Now { get; }

    public FrozenTime(DateTimeOffset now)
    {
        Now = now;
    }
}

Waiting for X Time

Using Task.Delay is an example of waiting for X time, or delay.

Examples of logic that require a delay are the following:

Waiting for some time before cancelling an operation (timeout).
Waiting before making further HTTP requests (rate limit).
Performing an action every X time (periodic execution).

Using Task.Delay also ties you to the system time, but it’s more insidious as it gives you the easy way out: just add shorter delays for testing.

Besides that, depending on your tests, they may not even be deterministic: their results may vary depending on how fast (or slow) the CI server is, on race-conditions, or just plain bad luck.

Again, the solution is fairly simple:

public interface IDelayer
{
    Task Delay(TimeSpan delay, CancellationToken cancellationToken);
}

public class SystemDelayer : IDelayer
{
    public Task Delay(TimeSpan delay, CancellationToken cancellationToken)
    {
        return Task.Delay(delay, cancellationToken);
    }
}

However, this time there are some differences on how we’ll write our test classes, depending on whether we’re going to test a single delay, or multiple delays.

Single delay

If you want a delay that completes almost immediately, then use the following:

public class NoDelayDelayer : IDelayer
{
    public Task Delay(TimeSpan delay, CancellationToken cancellationToken)
    {
        return Task.Delay(1, cancellationToken);
    }
}

If you want a delay that’s completed whenever you want, consider the following:

public class CancellableDelayer : IDelayer
{
    private CancellationTokenSource CancellationSource { get; }

    public CancellableDelayer()
    {
        CancellationSource = new CancellationTokenSource();
    }

    public async Task Delay(TimeSpan delay, CancellationToken cancellationToken)
    {
        await Task.Delay(-1, CancellationSource.Token);
    }

    public void Cancel()
    {
        CancellationSource.Cancel();
    }
}

You can emulate some interesting corner-cases when you have complete control of the sequence of events that happen before and after a delay is completed.

Multiple delays

This is where things get interesting.

Just a few days ago I had to test the following method:

public async Task ExecutePeriodicChecks(TimeSpan frequency, CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        // Perform the checks
        var checks = await _checkRepository.GetChecksByFrequency(frequency);
        var checkTasks = Task.WhenAll(checks.Select(PerformCheck));

        // Wait a bit before trying again.
        var waitingTask = _delayer.Delay(frequency, stoppingToken);

        await Task.WhenAll(checkTasks, waitingTask);
    }
}

It’s very straightforward code: it runs a process no faster than the specified frequency.

Testing that assumption is a bit harder though. Let’s say we wanted to test the following scenario: given a frequency of 1 minute, after 2 minutes and 10 seconds, 3 check cycles must have been performed. Where do we even start?

I’ll come clean now. At first I just decided to depend on system time, as follows:

[Fact]
public async Task ExecutePeriodicChecks_CheckFrequencyAsConfigured_ChecksEndBeforeTimeElapses()
{
    // Arrange
    var frequency = TimeSpan.FromMilliseconds(50);
    var handlers = ImmutableDictionary<Type, ICheckHandler>.Empty;
    var delayer = new TaskDelayDelayer();
    var repo = new Mock<ICheckRepository>();
    repo
        .Setup(x => x.GetChecksByFrequency(frequency))
        .Returns(Task.FromResult((IEnumerable<Check>)new List<Check>()));

    var doc = new Doctor(repo.Object, handlers, delayer);
    var cts = new CancellationTokenSource(140);

    // Act
    try
    {
        await doc.ExecutePeriodicChecks(frequency, cts.Token);
    }
    catch (OperationCanceledException)
    {
        // Do nothing; we're expecting the OCE.
    }

    // Assert
    repo.Verify(x => x.GetChecksByFrequency(frequency), Times.Exactly(3));
}

As expected, sometimes the tests fail at the CI server.

intermittent failure

Fundamentally, we need to abstract ourselves from the passing of time before we can test an scenario like this one. After doing so, we’ll notice that our previous statement:

Given a frequency of 1 minute, after 2 minutes and 10 seconds, 3 check cycles must have been performed.

Could also be expressed as:

After two delays have elapsed, 3 check cycles must have been performed.

Time is just an implementation detail.

What we care about here are elapsed delays, and so we implemented a class that would elapse immediately the first N times Delay was called and be of infinite duration for any further calls.

public class TimesDelayer : IDelayer
{
    private readonly object _mutex = new object();
    private int TimesRemaining { get; set; }
    public Action CallbackWhenZero;

    public TimesDelayer(int times)
    {
        if (times < 1)
            throw new ArgumentOutOfRangeException(nameof(times),
                "Times can't be less than 1.");

        TimesRemaining = times;
    }

    public Task Delay(TimeSpan delay, CancellationToken cancellationToken)
    {
        lock (_mutex)
        {
            if (TimesRemaining > 0)
            {
                TimesRemaining -= 1;
                if (TimesRemaining == 0)
                {
                    CallbackWhenZero?.Invoke();
                }

                return Task.CompletedTask;
            }
        }

        return Task.Delay(-1, cancellationToken);
    }
}

The callback is there so the calling test can be notified when Delay was called the configured number of times.

Notable differences from the previous test are the following:

[Fact]
public async Task ExecutePeriodicChecks_CheckFrequencyAsConfigured_ChecksEndBeforeTimeElapses()
{
    // We're simulating elapsed time with the TimesDelayer due to non-deterministic
    // results when using Task.Delay on slower computers / CI environment.

    // Arrange
    // ...
    var delayer = new TimesDelayer(3);
    // ...

    var doc = new Doctor(repo.Object, handlers, delayer);
    var cts = new CancellationTokenSource();

    // Act
    try
    {
        // Cancel ExecutePeriodicChecks only after delayer has been
        // called two times.
        delayer.CallbackWhenZero = () => cts.Cancel();

        var checkTask = doc.ExecutePeriodicChecks(frequency, cts.Token);
        await checkTask;
    }
    catch (OperationCanceledException)
    {
        // Do nothing; we're expecting the OCE.
    }

    // Assert
    repo.Verify(x => x.GetChecksByFrequency(frequency), Times.Exactly(3));
}

We’re cancelling ExecutePeriodicChecks after the Delay method on TimesDelayer was called for its third time. By doing so we can test that cancelling the operation while it waits works as we expect.

…but I just wanted to call Cancel more than once

public class CancellableDelayer : IDelayer
{
    private readonly object _mutex = new object();
    private CancellationTokenSource CancellationSource { get; set; }

    public CancellableDelayer()
    {
        CancellationSource = new CancellationTokenSource();
    }

    public async Task Delay(TimeSpan delay, CancellationToken cancellationToken)
    {
        CancellationToken token;
        lock (_mutex)
        {
            token = CancellationSource.Token;
        }

        await Task.Delay(-1, token);
    }

    public void Cancel()
    {
        lock (_mutex)
        {
            CancellationSource.Cancel();
            CancellationSource = new CancellationTokenSource();
        }
    }
}

Summary

Unit testing is about testing self-contained units. System time, just like databases and web services, is an external dependency that should be abstracted for testing.

In this post we saw how to abstract the retrieval of the current time, and how to perform delays in .NET Core by using the ITime and IDelayer interfaces respectively.

Finally, some example ready-to-use implementations have been provided.

blog.sinenie.cl/