Tag: programming

Deliberate Technical Debt: an example

Bank notes and coins

In a team discussion this morning we decided to incur some technical debt. This particular example is a nice illustration of the concept, so I thought I would share it.

The Context

We’re working on a proof-of-concept project that will extract some data from a file input, transform this dataset by adding data from various other places, and then load the enhanced data into an output file. (The eagle-eyed may have spotted that this is a classic extract–transform–load (ETL) pattern.)

The input data will be coming to us in a single, large file of comma-separated values (CSV). To deal with the volumes of data, we will be reading this file into a database for processing.

The additional data are extremely heterogeneous: some sources give us a single numeric value per record, one gives us more than a thousand datapoints. We expect to preserve the rough shape of the data throughout the process, but there may be some translation between the domain languages of the systems we interact with. These datasets are delivered to us in JavaScript Object Notation (JSON) format.

The output data format will also be CSV, but as the data will be significantly larger and more complex, we haven’t yet agreed how many files will be produced, or exactly how they will be structured.

There are currently plans for just one successful run of this process. As soon as we have an output file we are happy with, the project will be complete, and no further use of this code is planned. For this reason, we are not building this as a production system, but rather as a tool that can be run on a development machine. However, there’s a strong possibility that, if this project is successful, we’ll be asked to carry out similar work in future, so we want to create decently designed, adequately tested code so we can pick it up again. We also hope that this proof-of-concept will create demand for a production system with these responsibilities, so we would like to use this project as a learning exercise.

The Question

As I mentioned, one of the external datasets contains more than a thousand data points. We receive the data as a set of code-value pairs, with the same set returned for every record. We needed to decide how to store this dataset in our database.

If this dataset had had five data points, then this wouldn’t have needed discussion: we would have decided to create five new fields in the appropriate table. However, maintaining even a dozen additional fields increases the risk of errors, as you need to be able to guarantee consistency in data models, SQL creation scripts, object-to-SQL mappings, test data, and so on. Adding another data point presents further work, as you may need to make updates in all of these locations. (If our axis of changes is the set of supported data points, then this design breaks the open-closed principle, as extension of the system’s capabilities entails modification of existing implementation.)

If these are problems for a dataset with a dozen data points, then a dataset with over a thousand data points doesn’t bear thinking about. We needed to weigh up our options.

The Options

We identified three options, including our default choice for comparison:

  1. Create a field in the appropriate table for each code.
  2. Create a linked table with foreign key, code and value fields.
  3. Store the data in a blob.

We weighed these options by both their Cost to Implement, and the potential Cost of Change. We know we will incur the cost to implement, whereas the cost of change is more speculative. There is a chance that we will revisit this code in future, and that new data points may be added to the dataset, but we don’t know when or whether this will happen.

As I’ve already explained, Option 0 has a high cost to implement and a high cost of change.

Option 1 has a moderate cost to implement, as it means creating an additional table, and code for mapping code-value pairs from the application model into this database schema. On the other hand, the cost of change is fairly small: we might want to add a value or two to our test data sets. (This design observes the open-closed principle, extending the behaviour of the system to support another code can be accommodated without modifying any other code.)

Option 2 has low cost to implement, as we can create a single field and simply dump the data in there. Its cost of change seems to be low as well, as any additional data points would automatically get passed through. (It’s worth noting that this might be a terrible design for a long-running production system, as any previously stored data would be missing the new data points, and it would be pretty messy to update them; for a one-off run this isn’t a problem, and the design is perfectly acceptable.)

Based on these criteria, Option 3 seems to be a winner: it has low cost to implement, and low cost of change. What’s not to like?

Mixing Concerns

Option 2 manifests a significant design smell: it mixes the concerns of data retrieval, data storage and data presentation — the Transform and Load phases of the ETL pipeline. This smell is not shared by options 0 and 1, as they don’t make any reference to the output structure.

We receive the raw dataset as a JSON file, and will be outputting is as CSV. However, we haven’t yet agreed the exact structure of this CSV, and a huge dataset like this may raise another set of questions about file structure and normalisation. We don’t want to delay this work while these conversations continue.

Furthermore, the raw data we receive use a set of short codes that only make sense if you refer to the documentation. The recipients of our output data may be happy to receive the data in this format, but they may want us to use more explicit codes.

This has various implications:

We know for certain that our output format is different from the raw format of the dataset, so some mapping will have to take place.

We have three choices for mapping the data:

  • We could map the raw dataset to the output format before we store it, and then add it verbatim to the output. This keeps the Load phase of the ETL job very lightweight, but introduces output logic to the Transform phase. It also means the Transform phase becomes aware of the output model, which is a mixing of concerns
  • We could store the raw dataset, and then map it before adding it to the output. This keeps the Transform phase lightweight, and introduces mapping logic into the Load phase. It also means the Load phase becomes aware of the raw format, which is another mixing of concerns.
  • We could map the raw dataset to a third format before storing it, and then map it from that format before outputting it. This avoids the mixing of concerns, but at the expense of creating two sets of mapping.

All of this mapping is an extra layer on top of the object-to-SQL mapping we would already be doing, and so it adds to the cost of change for any alterations to the output structure. As we can only guess at the planned output structure, the likelihood we incur this cost is close to 100%.

Taking out the Loan

We had quite a conversation about these risks. Was this guaranteed cost of change enough to swing us towards another option?

We decided it wasn’t. Even with the additional cost of adapting this implementation to our preferred output structure, the overall cost of option 2 is less than that of the other options.

This doesn’t remove the fact that we’re incurring technical debt: we can guarantee that we will have to change this code, and that there will be rework — waste — when we do so; but we’re confident that what we gain now is worth the small additional cost we’ll pay in a couple of weeks’ time.

Tools we used to take our stand-up online

I’ve posted several times on here about team stand-ups (Reinvigorating a daily stand-up by walking the board, Quick fixes to stop ignoring your builds, Be more transparent about your meetings: put them on the board, How we collect team achievements with kudos cards). If you read these posts, you may notice that I’m a big fan of offline techniques: a physical whiteboard, index cards, sticky notes, and, if necessary, print-outs of online content. A certain global pandemic and the ensuing move to remote working made me confront this preference and forced me to find new, online, ways to visualise our work and environment in our stand-ups.

I work for a very Microsoft-centred company, and this influences our choice of tools. I hope that the spirit, if not the detail of these techniques can be easily transferred to whatever tools you have available.

Start with Coffee

We hold our stand-up every day at 11:45. Yes, this was at my instigation, as I’m not a morning person, so I like to run our stand-up at a time when I’m awake and alert. We chose a spot just before lunch (at 12:00!) so it would close off the morning’s work, rather than interrupting anything.

When we were working in the office together, we would often hold our stand-up and then drift off to lunch in small groups. Now we’re remote that’s not possible, so we’ve instituted a team coffee break at 11:30. We open an online meeting and have fifteen minutes of informal chat before we switch over to the day’s stand-up discussions.

When the time comes, I share my screen and we get started on the first discussion point.

Open an online meeting for an informal chat before stand-up.

Web Pages for Everything

I learnt long ago that it’s very easy to forget to discuss anything that’s not immediately visible on your team board. I apply this same principle now we’ve move online.

Whilst physical boards can be configured any way you choose, they only have a finite amount of space. Online sprint boards tend to be somewhat less flexible, but we can compensate for this by creating additional web pages with further discussion points on them.

Create additional web pages for further discussion points.

Use the Widgets

We track our work items in Azure DevOps, and the platform comes with a pretty flexible dashboard system, with a good selection of widgets. You can create whizzy graphs and visualise the status of your Build and Release pipelines.

However, my favourite two widgets are perhaps the most mundane: Markdown and Embedded Web Page.

We use the Markdown widget to add custom text and status messages to many of our dashboards, and I particularly enjoy the way you can create checklists and lists of links to other dashboards.

We use the Embedded Web Page widget for something even more important: a clock! I couldn’t a widget that would allow me to embed a clock on a dashboard, and I find it rather useful to timestamp the page, particularly if we’re going to screenshot it for sharing. To solve this problem, I created an Azure Function that returns a snippet HTML formatted with the same font used on the dashboard, and I embedded it on the page with an Embedded Web Page widget.

If there’s no widget support for what you want, create a web page elsewhere and use the Embedded Web Page widget.

Bookmark Everything

We currently use six separate pages for our stand-up. In Chrome I’ve created a folder in my Bookmark bar for all these tabs (to bookmark several pages at once, type Ctrl + Shift + D), and I can open them all at once by wheel-clicking on the folder.

Our six pages are:

  • Stand-up Overview
  • Team Stories Board
  • DevOps Stories Board
  • Pairing Staircase
  • End-to-End Test Status Dashboard
  • Calendar

Create a folder for all your stand-up pages, so you can open them all at once.

Stand-up Overview

This is the starting and finishing point for our stand-up discussion. It’s implemented as a dashboard with the following sections:

  • A panel with the sprint number, sprint name, and a checklist of our goals. This is implemented as a Markdown widget, and we edit the content as we complete the goals.
  • A date/time clock, implemented as an Embedded Web Page.
  • A panel with links to the other pages we use in our stand-up, and links to personal recognition pages on our company performance system. This is another Markdown widget.
  • A Sprint Overview widget.
  • A Cycle Time widget.
  • A Burndown Chart widget.
  • A Velocity widget.

We often find ourselves looking at the Burndown chart, which gives us a sense of whether our sprint is playing out as we hoped it would, and our Cycle Time chart, which is helpful for assessing our pace and flow of work.

Sometimes simple is best. Use a Markdown widget to track your goals.

Team Stories Board

This page comes the closest to a recognisable team board. We use the Boards feature of Azure DevOps, with some customisations.

  • We set our board to display stories, as this is the level at which we track our work.
  • We don’t filter by sprint, as we use this board to visualise our Product Backlog and Sprint Candidates as well as the items in our current sprint.
  • We use custom styles to indicate which stories are part of this sprint, which are overdue, and which are in our backlog.
  • We use tags and tag styles to give a quick indication of blocked stories (we also link these stories to their blockers).
  • We organise our board into seven columns, just as we did on a physical board:
    • Product Backlog. This corresponds to the New Status.
    • Candidates. A story becomes a candidate when it is refined and pointed.
    • Sprint Backlog. These are the stories chosen at sprint planning, plus any extras.
    • In Development.
    • Ready for Demo. These stories are essentially development-complete, but we demonstrate them to each other before closing them.
    • Done. This is a holding area so we can celebrate the completion of each story at stand-up.
    • Closed.
  • In addition, we split the central five columns into five swim lanes, and use these to track various other types of work:
    • Development contains our standard work items.
    • Meetings and Discussions visualises the many meetings I participate in.
    • Retro Actions
    • Kudos. As we can no longer hand each other kudos cards, we put them on the board.
    • To discuss. This swimlane is a hack. I drop an item in here if there’s something extra I want to discuss at stand-up.

Use swimlanes to visualise non-development work and topics for discussion

DevOps Stories Board

We have a DevOps team and rely on them to provide and configure infrastructure. Many of our stories are dependent on work being done by the DevOps team. We work closely with a member of the DevOps team, and he regularly attends our stand-ups.

Every day we look at the DevOps board, filtered to show only those tasks raised by our team. This gives us a change to check that we’ve provided all the information needed to complete any tasks we’ve raised.

You can filter other teams’ boards to see just those tasks raised by your team.

Pairing Staircase

In the office I would print out a templated pairing staircase for each sprint, and we used a bingo dabber to mark who had paired with whom.

When we moved online, I spruced up the Excel spreadsheet I had made for printing. I added a sum to the right of each person’s initials, and used Conditional Formatting Colour Scales to make it pretty.

A pairing staircase implemented as an Excel Spreadsheet

End-to-End Test Status Dashboard

We’ve written several suites of tests that run typical client journeys against our system. Failures in these tests can indicate instabilities across our ecosystem.

These tests run against each of our environments, and are build using Azure DevOps Release Pipelines, as these make it straightforward to run the same steps against several environments.

We use the Deployment Status widget to show the results of each of these suites of tests, and gather all these widgets on one dashboard, along with a clock (in an Embedded Web Page widget) and a notes panel (in a Markdown widget) to give details of any ongoing investigations and outstanding bugs.

We also send a screenshot of this dashboard to our colleagues every day so they can see how well we’re doing at keeping our environment stable.

Use the Markdown widget to give details of ongoing investigations and outstanding bugs.

Calendar

Our final page is our team Calendar in Azure DevOps, which automatically shows us our sprint diary. To this we add:

  • Holidays. Whenever I sign off someone’s leave, I add it to this calendar as well.
  • Office closed days and other events.
  • Our weekly build duty rota, which determines who investigates any issues.
  • Days we dedicate to working on our goals.
  • Planned releases.

By discussing this daily, we make sure it’s up to date and there are no nasty surprises.

Check your calendar daily to make sure there are no nasty surprises.

Back to the Overview

We end the stand-up by returning to the Overview dashboard.

Any stores we’ve closed will now be reflected in our charts (after a quick refresh of the page), and we take a moment to update the checklist of our goals.

We put out a final call for any other business, and wish each other a happy continuation of the day.

“Any more for any more? … Happy Wednesday!”

Me, at the end of every stand-up (give or take the name of the day!)

Quick fixes to stop ignoring your builds

A printout of a team dashboard on a team board

In software development, quick feedback is fundamental, and continuously building, testing and deploying is part of our standard toolkit.

But it isn’t enough to set up and run these processes; we need to pay attention to the results. If we can’t successfully build, test and deploy our software, then we have no idea whether we’re building it right, and we risk releasing broken components. It saddens me how often I see teams that have become completely inured to failures, and the result is a predictable decline in quality and drop in team effectiveness.

Here then are a few quick and easy techniques to start paying attention to these failures.

Continue reading “Quick fixes to stop ignoring your builds” →

A taste of graph theory

Postcard of buildings by a river. Caption reads ‘Königsberg Schlossteich’

I’ve recently been working with graph databases, which give us a powerful idiom for modelling and reasoning with highly interconnected data. Before I share some of my experiences, I would like to set the scene with a basic introduction to graph theory.

Simple Graphs

Graph theory is a relatively young branch of mathematics, traced back to 1736 and the work of Leonhard Euler.

A simple graph is defined as a non-empty set of vertices, e.g. V = {1,2,3}, and a set of edges, each of which is a 2-member subset of the vertex set, e.g. E = {{1,2},{1,3},{2,3}}.

We can visualise a graph by drawing a diagram:

simple three-vertex graph drawn as a triangle

It’s tempting, particularly for the etymologically minded, to think of a graph as drawing. It’s certainly easier to think about graphs by visualising them, but drawings as representations of graphs, have the potential to be misleading, so we need to exercise caution.

For example, the same graph can be drawn like this:

simple three-vertex graph drawn with curved edges and four crossings

Notice how this diagram shows four crossings, but is in fact equal to the above graph, which shows none.

Because simple graphs are defined in terms of sets, we can note some key characteristics:

  • Each vertex only appears once. V = {1,1,2,3} is not a valid set because a=a.
  • An edge cannot join a vertex to itself. {1,1} is not a valid set, so it cannot be a valid element in the edge set.
  • There can only be one edge between two particular vertices. E = {{1,2},{1,2},{2,3},{1,3}} is not a valid set because {1,2} = {1,2}.
  • The edges have no direction. E = {{1,2},{2,1},{2,3},{1,3}} is not a valid set because {1,2} = {b,a}.

Useful variations

These restrictions are great for pure mathematics, but somewhat restrict our ability to model real-world situations. For this reason, a typical graph database relaxes the rules in various ways:

Digraphs

It introduces a notion of direction in the edges. This means we are now dealing with directed graphs or digraphs.

We can now model a graph V = {1,2,3}, E = {(1,2),(1,3),(2,3)}:

three-vertex digraph drawn as triangle

Multigraphs

We can allow skeins: multiple edges between vertices. If we replace any edge of a graph with a skein, then we have a multigraph, and our edge set becomes a multiset, as it may contain duplicated elements.

Here we expand the basic graph V = {1,2}, E = {{1,2}} by replacing the edge {1,2} with a skein of three edges, giving V = {1,2}, E = {{1,2},{1,2},{1,2}}:

two-vertex simple graph drawn as line

becomes

two-vertex multigraph with three edges

We can also model loops by allowing single-member sets as elements of E, V = {1}, E = {{1}}:

one-vertex loop

Quivers

We can also allow directed multigraphs, also known as multidigraphs or quivers.

Degrees of vertices

The degree of a vertex, deg v, is the number of edges attached to it. In a simple graph deg v = |{e : e ∈ E, v ∈ e}|. Visually you can find the degree of a vertex by counting the edges that connect to it.

The indegree of a vertex, deg– v, is the number of edges leaving it, and the outdegree of a vertex, deg+ v, is the number of edges reaching it. If we model the directed edges as tuples, then deg– v = |{e : e ∈ E, ∃v e = (v,y)}| and deg+ v = |{e : e ∈ E, ∃x e = (x,v)}|. Visually you can find the indegree of a vertex by counting the arrows that leave it, and the outdegree by counting the arrows that point to it.

Walks

The power of graphs to model connected data arises when we start walking our graphs.

Mathematically, a walk is a sequence of vertices (v1,v2 … vn-1,vn) where each vertex vx is a member of the graph’s vertex set, and each pair of vertices (vx,vx+1) in the sequence is a member of the graph’s edge set.

Visually, a walk is found by placing a pencil on one of the dots on a graph diagram, and tracing along a line to another dot, then repeating the process.

When we come to look at graph databases, we will focus on ‘traversing’ them by walking along their edges.

Adjacency

As well as a diagram, we can represent a graph with an adjacency matrix. Consider the graph V = {1,2,3,4}, E = {{1,2},{1,3},{1,4},{2,3},{3,4}}. We can draw this graph like this:

four-vertex simple graph drawn as a square with one diagonal edge

We can also create an adjacency matrix where Aij is 1 if {i,j} ∈ E and 0 otherwise.

    ⎛0 1 1 1⎞
A = ⎜1 0 0 1⎟
    ⎜1 0 0 1⎟
    ⎝1 1 1 0⎠

The top row shows how many edges there are between 1 and each vertex: none to itself, one to 2, one to 3 and one to 4.

We can easily find the deg v by taking the sum of the corresponding row or column. We can see at a glance that the deg 1 = 3.

By multiplying an adjacency matrix by itself, we can find how many two-edge walks exist between any two vertices:

     ⎛0 1 1 1⎞   ⎛0 1 1 1⎞   ⎛3 1 1 2⎞
A² = ⎜1 0 0 1⎟ x ⎜1 0 0 1⎟ = ⎜1 2 2 1⎟
     ⎜1 0 0 1⎟   ⎜1 0 0 1⎟   ⎜1 2 2 1⎟
     ⎝1 1 1 0⎠   ⎝1 1 1 0⎠   ⎝2 1 1 3⎠

We can check that there are three two-edge walks between 1 and 1 {(1,2,1),(1,3,1),(1,4,1)}, one between 1 and 2 {(1,4,2)}, one between 1 and 3 {(1,4,3)} and two between 1 and 4 {(1,2,4),(1,3,4)}.

We can continue this trick for three-edge walks:

     ⎛0 1 1 1⎞   ⎛0 1 1 1⎞   ⎛0 1 1 1⎞   ⎛4 5 5 5⎞
A³ = ⎜1 0 0 1⎟ x ⎜1 0 0 1⎟ x ⎜1 0 0 1⎟ = ⎜5 2 2 5⎟
     ⎜1 0 0 1⎟   ⎜1 0 0 1⎟   ⎜1 0 0 1⎟   ⎜5 2 2 5⎟
     ⎝1 1 1 0⎠   ⎝1 1 1 0⎠   ⎝1 1 1 0⎠   ⎝5 5 5 4⎠

Again we can check that there are four three-edge walks between 1 and 1: {(1,2,4,1),(1,3,4,1),(1,4,2,1),(1,4,3,1)}, five between 1 and 2: {(1,2,1,2),(1,2,4,2),(1,3,1,2),(1,3,4,2),(1,4,1,2)}, five between 1 and 3: {(1,2,1,3),(1,2,4,3),(1,3,1,3),(1,3,4,3),(1,4,1,3)} and five between 1 and 4: {(1,2,1,4),(1,3,1,4),(1,4,1,4),(1,4,2,4),(1,4,3,4)}.

In general the matrix Aⁿ shows us how many n-edge walks there are between each pair of vertices.

We can perform the same trick for multigraphs and digraphs:

Here is the quiver V = {1,2,3,4}, E = {(1,2),(1,3),(1,3),(1,4),(2,4),(2,4),(3,4),(4,4)}:

four-vertex quiver drawn roughly as a square

Here is its adjacency matrix:

    ⎛0 1 2 1⎞
A = ⎜0 0 0 2⎟
    ⎜0 0 0 1⎟
    ⎝0 0 0 1⎠

Here are the next two n-edge walk matrices:

     ⎛0 0 0 5⎞
A² = ⎜0 0 0 2⎟
     ⎜0 0 0 1⎟
     ⎝0 0 0 1⎠
     ⎛0 0 0 5⎞
A³ = ⎜0 0 0 2⎟
     ⎜0 0 0 1⎟
     ⎝0 0 0 1⎠

We can also add these matrices:

              ⎛0 1 2 11⎞
A + A² + A³ = ⎜0 0 0  6⎟
              ⎜0 0 0  3⎟
              ⎝0 0 0  3⎠

This matrix tells us that from 1 to 4 there are eleven walks of no more than three edges.

Adjacency matrices can give us a useful way to reason about graphs without having to traverse every walk.

Conclusion

These concepts give us the basic tools for working with graph databases. In future posts I will look at how we can put them to work to model domains.

Creativity in Software Development

I shared yesterday’s post with some friends, who were keen to explore what we mean when we talk about creativity in software development.

Alastair made an interesting comment:

…it made me reconsider software dev as a creative endeavour, but I think I came to the conclusion that it is. For me, I think there is a gap between a creative art like writing, especially one which has an expressive mirror like acting, and a purely creative activity like, e.g., whittling a stick or constructing a building.

I think there is value in disentangling our concepts of creativity, and I find Alastair’s distinction between the creative arts and simpler forms of creation very useful.

There’s also an ambiguity in the word ‘create’, as it can refer simply to making things, as well as to the creative endeavours we would like to characterise.

So rather than ask ‘Is software development a creative activity?’, I tend to consider a narrower question: ‘Is there a place for creative thinking in software development?’

As the most basic level, I see creative thinking as making new links between concepts. Once you have made the link, you can engage other thought processes, for example deductive thinking, to explore the consequences and implications of that link.

But because the link isn’t already there, you can’t find it by rational thought; you need a leap of imagination to reach it.

There are some sorts of problem that I can tackle best once I’ve slept. On a few lucky occasions I’ve been able to take an afternoon nap, and woken up with a new idea to investigate, but this usually means taking the idea home with me and letting it brew overnight.

Here are a few examples of problems in software development that can be tackled with creative thinking:

  • How should we name this element?
  • What is the appropriate metaphor for this system?
  • Has a similar problem already been solved? Is there a pattern we can apply here?
  • What test should we write first? What test should we write next?
  • What is the best way to split this system into smaller parts?

And of course, because software development in an organisation is a social activity, the need for creative thinking extends far beyond the design of the software.

Test code needn’t be defensive

In a code review I encountered some test code that looked a bit like this:

 var result = await _controller.Resources() as ViewResult;
 result.Should().NotBeNull();
 // ReSharper disable once PossibleNullReferenceException
 result.Model.Should.BlahBlahBlah();

This is a typical defensive coding pattern whose reasoning goes like this:

  • The return type of _controller.Resources() is Task<ActionResult>.
  • I need to cast the inner Result of this Task to a ViewResult, as I want to inspect its Model attribute.
  • But the Result could be a different subclass of ActionResult, so I had better use a safe cast, just in case.
  • As I’m using a safe cast, I can’t guarantee that I’ll get any instance back, so I had better do a null check.
  • Oh look! ReSharper is complaining when I try to access properties of this object. As I’ve already performed a null check, I’ll turn off the warnings with a comment.

Now, defensive coding styles are valuable when we don’t know what data we’ll be handling, but this is most likely to happen at the boundaries of a system, where it interacts with other systems or, even more importantly, humans.

But in the context of a unit test, things are different:

  • We are in control of both sides of the contract: the test and class under test have an intimate and interdependent existence. A different type of response would be unexpected and invalid.
  • An attempt to directly cast to an invalid type will throw a runtime error, and a runtime error is a meaningful event within a test. If _controller.Resources() returns any other subclass of ActionResult, then the fact that it cannot be cast to ViewResult is the very information I want to receive, as it tells me how my code is defective.

This means I can rewrite the code like this:

 
var result = (ViewResult) await _controller.Resources(); 
result.Model.Should.BlahBlahBlah();

By setting aside the defensive idiom, I’ve made the test clearer and more precise, without losing any of its value.