On Pull, Entropy and Bottlenecks

A common scenario

Teams tend to work on features from the entry point in, pushing the behaviour through the system. In my experience this is almost always a bad idea, and I’m going to explain why, and offer some alternative approaches.

Let’s consider a fictional business: Sangeeta’s Sheet Music.

Sangeeta has focussed hitherto on selling piano music through her website and Android and iOS apps, but she’s now keen to corner the string quartet market. Now, string quartet music is sold in both score format and as parts, so there will be a certain amount of work to store information on available stock and to allow customers to choose which format they want to purchase.

Like in many successful startups, Sangeeta’s tech team has grown to some fifteen nmembers, who have been split into Catalogue, Payment, Frontend and Apps teams. She sits down with representatives from these teams, and they agree that Charlie and his team will start work on storing available formats in the Catalogue.

Two-and-a-half months later, Charlie’s team have updated the database and API schemata, and Penny and her team have enabled the shopping basket and checkout process to process different formats at different price points. At this point they approach Fatimeh from Frontend and Alejandro from Apps: “We’re nearly ready to go live with our new string quartet catalogue: we’ve got the data schema in place, our stock room is filling up with catalogue, so we just need to you to update the front end and apps, and we’re ready to start selling!”

Alejandro and his team are busy fixing a long list of bugs from the new iPhone, and won’t be able to start work for at least six weeks, but fortunately Fatimeh’s team have some capacity, so they start work. It soon becomes clear that some of the initial assumptions don’t work out on in real life: use case research shows that many orders are likely to be for all four string parts, and the shopping basket API doesn’t support this easily. Furthermore, they quickly realise that some of the catalogue doesn’t fit into the expected model: Schönberg’s String Quartet №2 has a part for soprano voice, and George Crumb’s *Black Angels* is only available in score format.

After some tedious toing and froing between the Catalogue, Payment and Frontend teams, they are happy with the functionality, but Alejandro and his team still aren’t ready implement this functionality in the apps, and Sangeeta doesn’t want to launch functionality on just the website. The feature has now been in progress for four months, yet it’s still vapourware, and the stock is sitting unsold in the stockroom.

On the face of it, there are two problems here: cross-team coordination and poorly specified behaviour. I contend that these are both results of a deeper problem: starting work in the wrong place.

Another approach

At 7digital I worked with Neil Kidd on a feature rather like this. We had observed how frequently these two problems arise, and decided we needed a different approach: we would start from the exit point and work backwards.

To do this we scripted some end-to-end User Journeys to describe the expected behaviours, stubbing out interim steps. We then worked with the responsible team to implement the final step in the process against canned data. We were then able to work backwards, working on each component in the chain to enable it to provide real rather than canned data. At each step in the process, we of course ensured that existing behaviour was unchanged, and that the new behaviour only kicked in when the previous component was ready to support it.

By starting with end-to-end User Journeys and considering the data output, we avoided nasty surprises, and our by triggering the new behaviour with data, rather than feature switches, we enabled the functionality to go live as soon as all the pieces were in place, which helps avoid producing vapourware.

I refer to this approach as Feature Pull, acknowledging the Kanban approach of pulling inventory through a manufacturing process, or work-in-progress through a development process, and I’ve been fairly vocal in encouraging colleagues to embrace it.

This concept tallies well with BDD and TDD principles, as it favours thinking about user journeys as an initial step, and testing from the desired outcome backwards.

Entropy

I wanted to find a way to illustrate this principle to colleagues outside the context of a feature, and felt a game might be the answer. I attempted to run such a game at 7digital, but the clearest lesson we drew from it was the benefit of communication, rather than anything specific to the flow of feature work.

And so at Socrates BE I ran a session in which I asked for help in coming up with a such a game.

As it happens, the interesting part of this discussion was not the proposals for the game itself, but the last few minutes, in which we started to examine *why* a pull system is often most appropriate. We alighted upon the notion of Entropy and the idea that features in a system should flow from the point of least to the point of most entropy.

If we consider that interactions with a system can often be modelled as a decision tree that branches over time, it is not surprising that the point of most entropy is at the end of a user journey, and it makes sense to start at that point.

Theory of Constraints

This felt like a nice take-away point, but I felt there was something missing. This morning I was reflecting on these ideas again, and the answer came to me: the point of most entropy is the bottleneck in the system, and by starting at this point we are subordinating our work to the bottleneck.

Think back to Sangeeta’s Sheet Music. The most obvious bottleneck in this system is clearly Alejandro and the Apps team, as they have so much other work to do. It therefore makes sense to start work by subordinating to this bottleneck, not building up inventory by implementing features that depend on this team, but rather finding ways to enable this team to overcome its workload problems.

Once the Apps team bottleneck has been tackled, then another bottleneck appears, this time one of planning: the team needs to understand and implement all the scenarios that are needed before the minimum viable feature can go live. This may involve a fair amount of planning and conceptual work, but again there is no point in doing upstream development work until this has been tackled.

Three metaphors

So here we have three metaphors for understanding how to approach features. If you like, you can focus on Pull, and start at the exit point of your system; if your mind turns to theoretical physics, then consider the entropy of your system, and start at the point where it is greatest; and if the Theory of Constraints gets you excited, then look for the bottleneck and subordinate to it. But whatever you do, please don’t just start coding!

Experience report: running the 7digital Technical Academy

Between September 2014 and April 2015 I ran 7digital’s Technical Academy: a half-year training scheme to give a new cohort of software developers a grounding in software craft. Here are a few thoughts on how it went.

Choosing candidates

Our four participants came from diverse backgrounds: two were graduate recruits, one with a degree in Forensic Biology, the other in Physics. One internal participant was a member of our QA team, while the other joined us from the Client Relations team. Three of them were women.

We explicitly opened the external vacancies to candidates with non-Computer Science backgrounds, and this gave us access to a more diverse field of candidates. We wanted candidates who had shown an interest in software development, but didn’t demand any industry experience: programming a spreadsheet to analyse coursework data, working with Access databases in a summer job, or creating a personal website could all be evidence of this interest.

At 7digital, the technical test is to pair programme on a software kata. The usual aim is to assess the candidate’s aptitude for TDD, and to see how well they work with another person. We used the same test for our Technical Academy candidates, but were fully aware that they were unlikely to have any TDD experience. Instead, we used this as an opportunity to find out how quickly they could pick up these techniques. At the beginning of the 45-minute exercise, I took a fairly directive role as Navigator: explaining what a unit test was, how an assertion worked, and telling the candidate what to type. By the end of the session, our successful candidates had demonstrated that they had picked up all the key concepts, and were capable of writing new tests with much less direction. This aptitude for learning was the most important quality we sought in our candidates.

Joining a team

Each participant joined one of the development teams, where they were expected to pair with a more experienced developer from the beginning. In addition, we ran several learning sessions a week, and expected them to work on a personal project over the six months of the academy. We suggested that they divide their time between academy and team work roughly 50:50.

New recruits at 7digital are assigned a Mentor, who has a pastoral role, catching up with them frequently and helping them settle in. We arranged Mentors for our two new joiners, and all four participants were also assigned a Tutor, whose role was to help with with any immediate technical issues during the academy, particularly in the project work.

Learning sessions

We organised a range of learning sessions, and implemented a pull system, choosing topics to address questions that had recently arisen during the participants’ project and team work. We met every Monday morning to finalise that week’s sessions, and to start planning for the following week. Apart from keeping a list of potential topics, we didn’t plan further than two weeks ahead.

Some of the sessions were hands-on practical exercises. At the beginning of the academy, we got the participants into the habit of TDD by running mob programming kata sessions: I would sit with the four of them around a computer, and they would pass the keyboard around in 5 minute sessions. As questions arose, we would sit back from the keyboard to discuss them. I introduced programming concepts and features of the IDE (Visual Studio with ReSharper) as we went along, so they could become familiar with running tests, elemental refactoring shortcuts and so forth.

Some of the sessions were more theoretical, and we enlisted colleagues to give talks and demonstrations of topics where they had particular expertise. Initially I took responsibility for organising this, but as our participants’ confidence and familiarity with their colleagues grew, I handed responsibility for this over to them.

Near the end of the course we ran some presentation sessions: one on the SOLID Principles and a couple on Design Patterns. Each of the participants chose a principle or a pattern and gave a short presentation to the others explaining it (I pitched in and gave a presentation on the Liskov Substitution Principle so as to make up the numbers). Some of the participants were rather nervous about doing their first presentation, but the results were impressive, and it was great to see their confidence boost once we had run these sessions.

Finally, we arranged a weekly reading group. During the first three weeks I led the group in a discussion of chapters 13–15 of Eliyahu Goldratt’s *The Goal*, which give the participants an accessible introduction to the Theory of Constraints, which underlies much of the Kanban methodology. After this, they started reading material on code design, and soon told me that they were happy to continue these sessions without me. The approach of discussion one or two chapters each week seemed to work very successfully, as they could engage properly with the issues.

The project

The project was a key focus of the participants’ learning. We sought ideas from colleagues across the company, and then let the participants choose a project. The person who had suggested the project acted as client, and was expected to be available to answer any questions, and to come along to demos to see how the work was progressing. Tutors were also expected to help out and give guidance.

We emphasised iterative development in the projects, encouraging our participants to get a walking skeleton running and deployed to a server on the network through our CI server (TeamCity) as soon as possible. Only then did we suggest they start fleshing out functionality. We also stressed the importance of doing TDD from the start. This approach front-loaded many of the difficulties in the projects: participants found it hard work to get going, but once they had a deployed project, they found they could pick up speed in their work.

We arranged frequent demos in which the participants could show off the progress they were making in their projects, and get feedback from their Clients and the rest of the academy.

Unlike in their team work, where pair programming was usual, much of the project work was done solo. There was a fantastic moment of revelation about a month before the final demonstrations, when it became clear that several of the participants were struggling with technical issues, and were worried whether they would be able to complete their projects in time. I suggested that rather than working solo, they work together in rotating pairs to solve these problems. At our next catch-up they reported that they had solved the problems and made more progress in those short pairing sessions than they had in the previous weeks’ solo work. For our participants to learn first-hand the value of pair programming less than six months into their careers was wonderful.

What worked

  • We found fantastic recruits and excellent internal candidates.
  • Pull learning was very successful.
  • The projects gave a good context for learning.
  • The participants developed an excellent team spirit and ability to self-organise.
  • The focus on XP and Software Craft skills equipped the participants with skills that are lacked by many developers with many more years’ experience: we created four exceptional software developers.

What didn’t work as well

  • Having a Client for each project worked better in theory than in practice; some didn’t even attend the demos.
  • Participants sometimes had difficulty getting a good balance between team work and Technical Academy work, and there were weeks in which their projects got no attention.

Would I do it again?

In a heartbeat. It was an amazing experience, and seeing the quality of our graduates is incredibly gratifying. I have now left 7digital, but there is a new Technical Academy kicking off as I write this, led by one of last year’s cohort. I wish them the very best of luck!


 

Edited on 20 Nov 2015 at 15:04 to mention the presentation sessions.

Edited on 26 Sept 2017 at 16:31 to remove a reference to an individual whose political views are not consistent with the spirit of this blog.