Secrets to building teams and software at scale: an interview with David Sale

When I first met David Sale as part of a small team building an innovative commerce platform at Sky almost 10 years ago, he was a bright graduate, eager to learn everything he could from the people we worked with. By the end of our time working together, I was the one learning things from him and asking for his expertise. I was always impressed by David’s ability to dive deep into a wide range of topics and technologies and quickly emerge with authority, new skills and pragmatic perspective.

This interview only scratches the surface on a number of interesting and important topics that David has unique and valuable views on. We talk about the ideal team size and structure for software projects; the importance of sound engineering principles in building production software; what enterprise software projects can learn from Netflix; how to focus on what matter in automated tests; as well as career and productivity tips.


The right team structure for project success

JD: From your experience as a software engineer in various settings, what correlations have you noticed between specific practices and overall project success?

DS: Initially, having the correct team size for your project is vital. Too big and you just end up with silos of people who favour each other or work better together. Too small and you don’t have a broad range of skills and experience to draw from, nor enough challenge on your ideas from other team members. I find a maximum size of 8 can work well, but no more than 4/5 developers, with the others made up of test engineers, scrum master, product owners, analysts as required.

The team needs to be built from a range of skill-sets and experience levels.  Typically a graduate, one or two software developers, a senior and/or lead often brings out the best results.  The reasons are quite simple.  By having the more junior members of the team, the experienced gain confidence and satisfaction from passing on knowledge and upskilling their colleagues.  They can also share out their workload with some of the tasks they may not have time for but which need to be done.  The graduate will learn immensely from these opportunities and being able to ask questions. The senior developers can guide the team on the way forward whilst also being challenged in fresh, innovate technologies from the rest of the team who are likely to have different knowledge and tried other tools.  This mix ensures the product has a best chance of being the optimum blend of production standard, modern technology that is maintainable for the future. 

Following on from team size would be building an actual team unit.  There are many facets to this such as:

  • Psychological safety – can all of your team speak up and debate constructively with only the success of the team/product that matters and nothing else.
  • Team spirit – doing things outside of normal work. Breakfast/lunch, go karting, nights out. Anything you all enjoy doing that doesn’t feel like forced fun.  It needs to be natural.
  • Pairing – the one activity I have seen build knowledge of each other and understanding is pairing on work for a high percentage of your week and rotating with each and every team member throughout the sprint. This activity alone forces people to work with each other in a good way, meaning that you don’t just stick with people you get on with better.  You naturally have breaks and chat around the work and learn about each other. In my estimation if a team would take 12 months to reach team unity and understanding, this practice could easily halve that time or more as opposed to working alone.

Finally, a clear, agreed vision for the product and a set of achievable milestones is critical for a successful delivery.  Each person, the team, product owners and senior management all need to be in alignment on what is required, by when and for whom.  It then needs to be revised with clear objectives and results for the team on a sprint, monthly, quarterly and yearly basis.  This keeps everyone focused on the most important next item, whilst allowing for regular celebration of success. 

JD: Pair programming is such an under-appreciated tool. I can definitely vouch that teams with a structured approach to pair programming are significantly more resilient and build momentum much faster than teams that don’t. And there are context-sensitive ways of doing this, even in the remote-first future we are heading into.

The business value of sound software fundamentals

JD: I know you’re saying all of this from experience, because we’ve both seen it done right and also seen it done wrong (or not done at all). Speaking of your experience, what would you say is the most interesting project you’ve worked on and in what ways has it changed the way you see the world?

DS: I think the most interesting project in my career so far would have to be my current one [at Booking.com]. We are building a configurable, lightweight micro service which needs to connect to hundreds of car rental companies API’s around the world, all of whom have their own way of doing what is essentially the same operations. We have to write our service in such a way that we can reuse the components we develop so that you can build a companies connection form the “lego box” we provide. It’s a great challenge and the team work hard to write as little new code as possible to keep the solution elegant and allowing us to deliver new integrations to the business fast from our already built toolkit.

I think that is the biggest thing this project has taught me. When you leave university, you think you need to be writing masses of code and delivering all the time. This project is about delivering more by writing smarter code. Reusable code, so that when a feature is built, many companies API can reuse it.

Secondly it has provided us an opportunity to learn about the world. As we connect to companies literally world-wide, you learn about where places are, the customs in that location (did you know in Israel it is illegal to charge an extra fee to young drivers, whereas virtually all companies worldwide charge for this “service”) and generally just get to speak and work with a global network of companies. All with the same goal of delivering a good experience for the customer.

JD: This is precisely what I love about our trade – being exposed to a variety of industries in different geographies and different ways of doing things enriches you so much. Especially when you begin to notice patterns of how something that is a best practice in one industry can be transferred to another – there are immense opportunities for value creation in this way.

Best practices for software reliability and resiliency

JD: Let’s get a bit more technical. You’ve written a great book on software testing automation with Python. What are some under-appreciated ideas and areas in that field which you don’t feel are getting enough attention in practice? (Doesn’t have to be Python-specific)

DS: I think there just isn’t enough focus from developers in the industry on the shipping of the product and it’s usage in production.  We need much more coverage and guidance on performance testing, cloud readiness, 12 factor apps, docker and the list goes on.  A lot of the books and tutorials focus on writing code to fix problems.  But I think there is a lack of focus on the quality and performance of code under real world strain.  

Services these days need to support high availability, high traffic 24/7. I still see so many third party services we consume in our business sending emails informing us of “scheduled downtime” in 2020.  Netflix are one of the gold standards in this regard.  They have developed tooling to actively break their systems in production so that no matter what they face, the applications are stable and ready to serve customers who just won’t accept being unable to stream at any time of day or year. As developer’s I think we should be looking towards these leading companies and trying to adopt their best practices into our projects regardless of having one customer or millions.

JD: I think we’ve seen this even more clearly with the dramatic shifts in demand around the pandemic and the economic effects around it. Even some of the biggest and most resourceful companies have had their IT systems fail them at critical moments in ways that, from a technical perspective, just shouldn’t be happening if they were better prepared.

Moving faster by testing what matters

JD: In terms of the ability to deliver reliable, maintainable production software – what aspects of programming languages and DevOps tools in general do you feel could be improved or better integrated?

DS: For me, I always build on or introduce a behaviour testing suite into my projects. Right now, that is Cucumber for Java.  If you have never used behaviour driven development or the tools that go with it, I highly recommend reading up on the practice.  I have some articles on the web on the subject and my book also covers it.  With the right coverage in a BDD suite, you almost remove the need for unit tests in my opinion.  They form a contract that your application must meet to be able to be pushed to Git.  If you cover all the behaviour in your system then you freely change the implementation, be it lines of code, frameworks, DB, even language and still make them pass.  This is much the same at the unit level to a smaller degree.  This is very achievable for a backend service where ultimately your JSON input and output is all that matters between you and the client application.

I’d like to see one standard tool for doing this practice that is agnostic of language.  That way the entire development community could contribute to such a tool and have it cover many use cases and languages.  To a degree Cucumber does this but it has language specific implementations.  I think a tool could be built that doesn’t even need programming language steps behind it, but just allow you to set expected methods, url, payloads and assert on the outputs.  I think over time we see this used more and more.

JD: This is so on-point in my experience as well! Many software engineers I talk to seem to find it difficult (probably based on training and conditioning) to make a psychological shift away from unit tests onto higher level functional tests. People have looked at me funny when I express my belief that unit tests can be counter-productive, in that they tend to restrict your ability to refactor freely and move forward with momentum. BDD tests, on the other hand, tend to enhance this ability. It’s not that unit tests are a bad thing – there’s definitely a place for those – it’s more about making a clear distinction about testing what’s important in the business context (the what and the why) versus testing implementation details (the how).

To get ahead in your tech career, focus on the business context

JD: What advice do you have for junior or mid-level software engineers who want to take their career to the next level? In your experience, what specific skills, ways of thinking or ways of working make someone more valuable in that respect?

DS: I think it’s vital that engineers at all level understand the entire picture.  They need to be able to service the code from design, planning, creation, testing, deployment and delivery to customer.  That means being comfortable with knowing the code, testing practices, deployment pipelines, servers/cloud infrastructure, source control, logging and alerting/monitoring.  I would advise them to dive into all these areas and learn from the people who are setting them up.  Pair with them as they make changes. Do these changes yourself whilst they guide you.  Become the go to person on all these topics and then teach everyone so you are not the bottle neck.  

I live by the mantra of “teach people to fish, don’t give them a fish” on every aspect of software development.  If you teach them, they can serve themselves so if your team is not doing this and you are always going to that person, get them to show you. 
In addition, you need to be able to see and design the larger picture. It’s great being an expert on your specific system.  But how does it fit into the wider organisation.  What is the future for your project? Is it legacy and needs replacing or is the Greenfield project that the company is looking for.  Each has their own challenge and you need to be able to come up with solutions and designs that fit into the overall plan and drive real change forward.  If you want to progress in your career you need to be able to work with many different teams and people outside of just software engineers. Can you tailor your conversation to a dev, a tester, a product owner or analyst. What about the head of engineering or leadership team.  All of these people have different concerns and levels of understanding but you need to explain yourself clearly so they can buy into your project and get you support you need.

It is not enough to be a great coder.  In 2020, you are expected to wear many hats and cover many different responsibilities if you want to progress.

JD: From talking to other highly experienced software engineers, this seems to be recurring theme indeed. Adriana Vasiu gave very similar advice in our first interview here. It’s obviously important to be good at coding and to understand the machine, the operating system, the technologies you work with. But what seems to be disproportionately more valuable in terms of your career progress is your ability to take a step back and consider all of this in the wider context of the business processes, objectives and interactions that created the need for your work to begin with.

Get more done with less stress by owning your time

JD: Let’s finish with another question I like to ask. Do you have any productivity tips that you’ve recently found practical and useful?

DS: Owning your own time. I see this every day in my career. People complaining they don’t have enough time to complete things or having back to back meetings all day. I find it baffling. Your time is your own to manage and to push back where necessary. Simply because we are always on tools such as slack or email doesn’t mean you have to immediately respond to things. Similarly regarding meetings, when you receive a request you need to question at first why am I being invited, what can I offer to this meeting. If you find yourself thinking I have nothing to offer, seek clarification as to why you are needed. Maybe you can provide some quick info or where the person can go for that info, removing you from the actual meeting. I think a lot of it comes down to politeness, but I have seen stats about the wasted time and money on people in meetings they do not need to be at, simply because they don’t want to say no to someone.

I find blocking out time in my calendar really helps. So I now have 1 or 2 hour blocks virtually every day marked as “focus time”. Most people check calendars and will work around them, otherwise if they book in those slots I decline and 99% of the time it is moved to a time when I am free in my calendar. Only if it is fire alarm type emergency do I break these rules, such as key production issues or other pressing matters. But this is very rare.

I have found adopting this approach has resulted in much better output from myself. I am able to get done what I need to in my focus times and I am then less stressed in meetings and more present. Switching off from Slack in those focused times really helps too. As we work from home more in this new world you can be bombarded by people on messaging apps who have no context on the other messages you are present in. Reply to them in your own time. If it is that urgent they will find another way to get you.


If you’d like to work on the same team with David Sale, Booking.com are now hiring: https://careers.booking.com

David has a lot more valuable content over at his website: http://www.dsale.co.uk/ You can also follow him on LinkedIn and Twitter.

For even more insight from David and other leaders in tech and business, subscribe to this blog.