Creativity in Software Development

I shared yesterday’s post with some friends, who were keen to explore what we mean when we talk about creativity in software development.

Alastair made an interesting comment:

…it made me reconsider software dev as a creative endeavour, but I think I came to the conclusion that it is. For me, I think there is a gap between a creative art like writing, especially one which has an expressive mirror like acting, and a purely creative activity like, e.g., whittling a stick or constructing a building.

I think there is value in disentangling our concepts of creativity, and I find Alastair’s distinction between the creative arts and simpler forms of creation very useful.

There’s also an ambiguity in the word ‘create’, as it can refer simply to making things, as well as to the creative endeavours we would like to characterise.

So rather than ask ‘Is software development a creative activity?’, I tend to consider a narrower question: ‘Is there a place for creative thinking in software development?’

As the most basic level, I see creative thinking as making new links between concepts. Once you have made the link, you can engage other thought processes, for example deductive thinking, to explore the consequences and implications of that link.

But because the link isn’t already there, you can’t find it by rational thought; you need a leap of imagination to reach it.

There are some sorts of problem that I can tackle best once I’ve slept. On a few lucky occasions I’ve been able to take an afternoon nap, and woken up with a new idea to investigate, but this usually means taking the idea home with me and letting it brew overnight.

Here are a few examples of problems in software development that can be tackled with creative thinking:

  • How should we name this element?
  • What is the appropriate metaphor for this system?
  • Has a similar problem already been solved? Is there a pattern we can apply here?
  • What test should we write first? What test should we write next?
  • What is the best way to split this system into smaller parts?

And of course, because software development in an organisation is a social activity, the need for creative thinking extends far beyond the design of the software.

Bring your Shadow to Work

Handrail and shadow

There‘s an idea in certain circles that we should be able to bring our whole selves to work.

There are aspects of this notion that I find unproblematically wonderful. For those of us who are invisible members of minority groups, the ability to drop the mask and be open about our identities can help us find a level of safety and inclusion at work.

There are areas that I find problematic, particularly questions about how permeable the work/life barrier should be. These are questions for another day.

The point I’m thinking about today is that there are aspects of all of us that are unpleasant. We think dark thoughts and entertain transgressive fantasies. These are parts of our whole self, but do we really want to bring them to work?

Carl Jung characterises this aspect of us as our Shadow, and sees our encounters with our Shadow as part of the way we Individuate ourselves. Repression and denial of the Shadow can lead to dysfunction: we can become overwhelmed by it and start Acting Out our fantasies, rather than enjoying them in the privacy of our mind.

So I find a tension here: being a psychologically healthy person requires us to have a healthy relationship with the dark areas of our minds, to admit these areas into our whole selves. But if we are to bring our whole selves to work, then we need to bring these dark areas to work as well.

This question becomes more pressing if we accept that there is a close relationship between our Shadow and our creativity. If we hope to do creative work, then we need to be able to dip our bucket into this dark well.

In this context, I read something interesting in an interview with Phoebe Waller-Bridge:

More than anything, she says, as a writer she wants to show women indulging their appetites and venting their grievances. “We sexualise women all the time in drama and TV. They are objectified. But an exploration of one woman’s creative desire is really exciting. She can be a nice person, but the darker corners of her mind are unusual and fucked up, because everyone’s are.” Has she always been able to say the unsayable? “Yes. As long as it feels truthful, as long as it’s pointing at the elephant, it is always exciting.” [Emphasis mine]

And this got me thinking again: Waller-Bridge is making quite a name for herself by bringing the darker corners of the mind, the Shadow, into her work. As a screenwriter this may be rather more straightforward than for a software developer, for example. But is there a way we can openly and honestly bring these aspects of ourselves to work? Is a truly psychologically safe workplace one where we can invite our Shadows?

Test code needn’t be defensive

In a code review I encountered some test code that looked a bit like this:

 var result = await _controller.Resources() as ViewResult;
 // ReSharper disable once PossibleNullReferenceException

This is a typical defensive coding pattern whose reasoning goes like this:

  • The return type of _controller.Resources() is Task<ActionResult>.
  • I need to cast the inner Result of this Task to a ViewResult, as I want to inspect its Model attribute.
  • But the Result could be a different subclass of ActionResult, so I had better use a safe cast, just in case.
  • As I’m using a safe cast, I can’t guarantee that I’ll get any instance back, so I had better do a null check.
  • Oh look! ReSharper is complaining when I try to access properties of this object. As I’ve already performed a null check, I’ll turn off the warnings with a comment.

Now, defensive coding styles are valuable when we don’t know what data we’ll be handling, but this is most likely to happen at the boundaries of a system, where it interacts with other systems or, even more importantly, humans.

But in the context of a unit test, things are different:

  • We are in control of both sides of the contract: the test and class under test have an intimate and interdependent existence. A different type of response would be unexpected and invalid.
  • An attempt to directly cast to an invalid type will throw a runtime error, and a runtime error is a meaningful event within a test. If _controller.Resources() returns any other subclass of ActionResult, then the fact that it cannot be cast to ViewResult is the very information I want to receive, as it tells me how my code is defective.

This means I can rewrite the code like this:

var result = (ViewResult) await _controller.Resources(); 

By setting aside the defensive idiom, I’ve made the test clearer and more precise, without losing any of its value.

Identical random values in parallel NUnit test assemblies

We have some NUnit cross-system test assemblies that run in parallel. They upload files and watch to see how they are processed. We were inserting a randomly generated value into the filenames in an attempt to avoid identically named files overwriting each other.

Unfortunately, this didn’t work, and we saw frequent test failures. In particular, we found that filenames generated by one test assembly were often identical to those generated by another.

I wanted to understand why this was happening, so I did some investigation.

We were getting our values fromTestContext.CurrentContext.Random, which is an instance of NUnit’s Randomizer class.

When we look at the implementation of Randomizer, we see this:

static Randomizer()
    InitialSeed = new Random().Next();
    Randomizers = new Dictionary<MemberInfo, Randomizer>();

The Randomizer is statically seeded with a value generated by the System.Random class. Because this seed is static, the same sequence of random values is shared by all references to Randomizer within each assembly, producing a sequence of values that are highly likely to be different from each other. However, references to Randomizer in a concurrently running assembly use a different instance and have their own sequence of values with its own seed.

Let’s have a quick look at how System.Random is seeded.

The MSDN documentation tells us:

  • The Random() constructor uses the system clock to provide a seed value. This is the most common way of instantiating the random number generator.

And goes on to warn:

…because the clock has finite resolution, using the parameterless constructor to create different Random objects in close succession creates random number generators that produce identical sequences of random numbers.

It would seem that our two parallel test assemblies often start executing within such a short interval of each other, and that NUnit’s Randomizer is seeded with the same value in each assembly, which means the sequences of values are identical.

There is some discussion of introducing a mechanism for controlling the seeding of Randomizer in NUnit, but in the mean time, the solution to our problem was to seed our own System.Random instances, rather than relying on NUnit’s.

How applying Theory of Constraints helped us optimise our code

The neck of a bottle of prosecco in front of a fire.

My team have been working on improving the performance our API, and identified a database call as the cause of some problems.

The team suggested three ways to tackle this problem:

  • Scale up the database till it can meet our requirements.
  • Introduce some light-weight caching in the application to reduce load on the database.
  • Examine the query plan for this database call to find out whether the query can be optimised.

Which of these should we attempt first? There was some intense discussion about this, with arguments made in favour of each approach. What we needed was a simple framework for making decisions about how to improve our system.

This is where the Theory of Constraints (ToC) can help. Originally expounded as a paradigm for improving manufacturing systems, ToC is really useful in software engineering, both when managing projects and when improving the performance of the systems we create.

Theory of Constraints

The preliminary step in applying ToC is to identify the Goal of your system. In the case of this API, the Goal is to supply accurate data to consumers.

Now we understand the Goal of the system, we can define the Throughput of the system as the rate at which it can deliver units of that goal, in our case API responses. We can also define the Operating Expenses of the system (the cost of servers) and its Inventory (requests waiting for responses).

The next step is to identify the Constraint of the system. This is the element in the system that dictates the system’s Throughput. In a physical system, a useful heuristic is a build-up of Inventory in front of this element. In our API, our monitoring helped us pinpoint the bottleneck.

The next three steps give us a sequence of approaches for tackling the Constraint:

  • First, Exploit the Constraint by finding local changes you can make to improve its performance.
  • Second, Subordinate the rest of the system to the Constraint by finding ways to reduce pressure on it so it can perform more smoothly.
  • Third, Elevate the Constraint by increasing the resources available to it, committing to additional Operating Expenses if necessary.

Exploitation comes first because it’s quick, cheap and local. To Subordinate you need to consider the effects on the rest of the system, but there shouldn’t be significant costs involved. Elevating the Constraint may well cost a fair amount, so it comes last on the list.

Once you have applied these steps you will either find that the Constraint has moved elsewhere (you’ve ‘broken’ the original Constraint), or it has remained in place. In either case, you should repeat the steps as part of a culture of continuous improvement. Eventually you want to see the constraint move outside your system and become a matter of consumer demand.

Applying ToC to our question

If we look at the team’s three suggestions, we can see that each corresponds to one of these techniques:

  • Scaling up the database is Elevation: there’s a clear financial cost in using larger servers.
  • Introducing caching is Subordination: we’re changing the rest of the system to reduce pressure on the Constraint, and need to consider questions such as cache invalidation before we make this change.
  • Optimising the query is Exploitation: we’re making local changes to the Constraint to improve its performance.

Applying ToC tells us which of these approaches to consider first, namely optimising the query. We can look at caching if an optimised query is still not sufficient, and scaling should be a last resort.

In our case, query optimisation was sufficient. We managed to meet our performance target without introducing additional complexity to the system or incurring further cost.

Further Reading

Goldratt, Eliyahu M.; Jeff Cox. The Goal: A Process of Ongoing Improvement. Great Barrington, MA.: North River Press.