In August 2015, Edgeware made the decision to change the way we develop and deliver software and hardware. We started a lean and agile transformation, and this blog post describes where we are and what we’ve learned three months into the process.
Lean and Agile
First, let us define what we mean when we say “lean” and “agile”. Lean and agile are complex areas, so let’s confine the discussion to a few selected topics.
From lean we learn that flow efficiency is very important. It doesn’t matter how efficient every department of a company is unless we collaborate well to solve the customer’s problem. For example, it doesn’t matter if the different parts of our organization are very efficient executing on their backlog unless we can effectively collaborate to quickly get things out the door to the customer.
From agile we understand that the ability to learn and adapt is very important. This affects everything we do, from the way we get feedback on code changes, to the frequency at which we deliver to customers and get their feedback. When we learn, we can also take action to change the plan. We also understand that in agile, value is at the heart of every decision. We focus on delivering the most valuable items at every point in time, again, even if it means changing our plans.
From lean and agile, we also learn that continuous improvement is essential to both. We need to constantly reflect on what our biggest problems are and take action. We cannot tell upfront what actions will help, so we run experiments and evaluate the outcome.
Expected Outcomes
When applying an agile methodology like Scrum, it is important to understand why we do the things we do. We are looking for a particular result or outcome from our organization.
First, in our case, we want to reduce the lead times from initial idea to customer delivery. This goes for everything from implementing and releasing new features to handling customer support cases. Secondly, and related, we want to be able to quickly respond to market needs. We need to be able to change our plan if we realize it will not produce the most valuable results. Third, we want to build a learning organization that improves from every failure.
The outcomes you are looking for will affect the way you set up your organization and will influence daily decisions.
Our Organization
Our R&D department is divided into three parts: The Platform team, multiple feature teams and the Shield team. The Platform team selects and evaluates the underlying hardware (motherboard, circuitry, flash DIMMs etc.). They also write the FPGA VHDL code for the hardware acceleration used to achieve our market-leading streaming performance. Hardware development cycles are often longer than software development cycles which presents interesting challenges when it comes to agile development.
The Shield team, as the name implies, shields the rest of the organization by resolving external issues. For example, they collaborate closely with Customer Support to resolve customer issues. How to get good flow efficiency for customer errands is an interesting topic on its own that we will cover in a future blog post. People rotate in and out of Shield on a four week schedule and as we continue to staff up, full feature teams will rotate in and out of Shield. Time spent in Shield brings valuable insight into customer setups and issues.
A feature team is a cross-functional teams of 5-6 developers. The team’s competencies span the technology stack from the driver layer (C), to the Linux application layer (C++) all the way up to the cloud micro services layer (Python). The team is autonomous, meaning that team members are self-organizing in terms of processes, roles and responsibilities. That said, the teams all contribute to a shared set of tests, test frameworks, continuous delivery pipeline, definition of done, definition of ready and so forth. The teams also comply to some common rules, such as the requirement to track velocity and “happiness index” (see below) over time.
All feature teams have a tester. The tester is a developer that brings “the tester mindset” to the team (How can this possibly break?) and is responsible for championing quality. He or she also mentors the developers in techniques such as exploratory testing. All team members are expected to contribute to test automation (Python) and exploratory testing. The team tests the feature it develops on unit, component, integration and system level. They test functional as well as non-functional aspects such as stability and performance.
The Product Owner sits with the team to ensure close collaboration. The feature team is responsible for the full user experience and quality of a feature across all layers of our technology stack (also, if hardware is involved, Platform is involved in the daily stand-ups etc.).
Working with Agile
In each of the feature teams, one of the developers has the additional role of agile coach. We use Scrum and the agile coach has similar responsibilities as a Scrum Master. Most importantly, the agile coach helps the team increase the amount of value delivered with the same effort over time. We run two-week sprints synchronized across the teams. Two times per week, we do a Scrum of Scrum around the release board (Figure 1) to update the progress and sync on any stoppers we might have.
At the end of the sprint, each team has a sprint retrospective. There we reflect on how to improve the process and the way they work. An important part of this is the happiness index (inspired by Crisp): “On a 1-5 scale, how do you like working at Edgeware? What would make you score higher? Out of everybody’s suggestions, which one do you think would help increase the team’s average score the most?” The most popular suggestion is taken into the next sprint as an action. We track the happiness index over time to make sure the trend is improving.
The team also has a sprint review. There we review the work that has been completed and reflect on how to make the product backlog better and more valuable for the next sprint.
At the end of each sprint, we have a Fix-it-Friday (fix whatever you think is most painful) or a Lab Day (implement any product or tooling ideas you have). Both end in a Demo Friday event where all teams demo the things they finished in the sprint (and the Lab Day). This helps spread the knowledge and is a possibility to celebrate our successes.
Planning and Adjusting the Plan
We have a joint backlog for all feature development at Edgeware called the feature backlog. It contains epics and features. Epics are big things, possibly many months of work. When we plan for the next 3-6 months of deliveries, we break down the epics into features, which are 1-4 team-weeks of work. Product Management and Product Owners groom and order the backlog every week in order to deliver the most value at every given time and to be able to forecast the content of future releases.
In a release planning event held each quarter (inspired by SAFe), the feature teams pick features from the feature backlog. In order to estimate when a feature will be delivered, the team breaks down the feature into user stories, which is typically 1-3 team days of work. A user story is a thin slice of functionality that spans the technology stack. The result of the estimates is the release planning board seen in Figure 1. Each feature is placed in the week (column) it is assumed to be completed. This is our best estimate at the time of the planning, and naturally, we re-visit the plan frequently in order to meet deadlines and market commitments.
We are gathering data on feature completion times. The uniform 1-4 week size of the features will allow us to use historical data on feature throughput to forecast the scope of future releases. More on forecasting in a future blog post.
Delivering Often and On Time
The quarterly release at the end of each release cycle is tied to important marketing events. Despite this, we want to get feedback early and often from customers. To that end, we release software to customer labs throughout the release cycle. This provides us with valuable learning which allows us to adjust the plan throughout the release cycle.
The relatively small size of the features allows us to burn down features at a steady pace throughout the release cycle. This in turn allows us to detect when deadlines are at risk, which is incredibly important. If we get an early heads-up, we have a variety of tools to apply to adjust the plan. For example, we might de-scope individual features to save time, we might re-order features to meet deadlines or we might assign more feature teams to an epic to speed up development. This together with the fact that we build working software across the stack every sprint (as seen in the Demo Friday) allows us to deliver on time.
Unblocking the Engineers
In order for the engineers to feel and be effective, we work hard to remove blockers. The time we invest now in fixing problems is re-payed in increased productivity and happier engineers. We track technical debt in our improvement backlog. The teams are expected to spend 20% of their sprint time on fixing things from the improvement backlog. The Fix-it-Fridays and Lab Days mentioned earlier are also used to address improvements. For bigger things, we bring them in together with feature work.
We also have Engineering Tooling developers working on unblocking the teams. Currently, the most important effort is to take the various test environments and continuous integration efforts we have and converge them into a continuous delivery pipeline. Developers need fast, early and accurate feedback on the changes they make. The continuous delivery pipeline will provide that, as well as well-tested binaries at the end of every night. We have also moved to a single Git repository to simplify cross-stack development and reflect our thinking around system releases.
Ah Yes, One Last Thing
An agile transformation is not a quick fix. It takes a long time for a new culture and mindset to settle. It requires commitment and support from staff and the leadership team alike. Fortunately, we have no lack of either. Three months in, we come across new problems and challenges every day that we address and learn from. That said, we are already seeing benefits from the new way of working, such as reduced lead times and increased flexibility in planning. With the people we have and the rapid progress we’ve made, I have no doubt in my mind that we can build a world-class performant product delivery organization. If you are passionate about software and/or hardware development and want to join our agile transformation, contact us!
Author:
Johnny Bigert
VP R&D
Fill out the form below and we will get in touch with you.