Making Snow Angels in a Brownfield

Thinking about my last post triggered me to finally read Kill it with Fire by Marianne Bellotti. The book is all about modernizing legacy systems, which is something developers have to do frequently. Often, systems that are working well tend to get ignored until they fail. By then, the people who built them have moved on or retired. The current developers don't have the institutional knowledge to easily fix or extend that system. But it doesn't have to be that way. Just like regular and preventative maintenance for your car, developers can conduct regular and preventative maintenance on software to extend its useful life.

For most of us, maintaining our car is about changing the oil and replacing worn parts. Maintaining software is a bit more like maintaining a race car, tweaking settings and trying custom parts to get the highest possible performance. In software, the improvements can be about adapting to a changing environment, or fixing a bug, but the lion's share of change requests are about altering or extending the system based on feedback from customers or other stakeholders.

Given that software maintenance often costs as much as or more than the initial build cost, it makes sense to spend time making that process as efficient as possible. So how can organizations prepare for the inevitable need to refactor or upgrade existing systems? How can developers do the work in a way that makes updates and extensions easier.

Place a high value on maintainers

Developers love starting new projects. The alluring fresh scent of a greenfield. Clean slate. A chance to make better choices this time. We are culturally conditioned to place more value on the new and novel. New projects and new technologies contribute to our professional growth and our resume. Organizations need to counterbalance our culture's newness fetish by placing increased importance and value on those who take on the maintenance burden. Bellotti's book has a section in chapter 8 about being seen. As managers we need to find a way to recognize the accomplishments of those doing the unglamorous work. The team responsible for keeping that old app running like a well oiled machine need to be recognized and remunerated at least as much as the team building the fancy new thing. If the fancy new thing fails, the organization can regroup and try again later. If the legacy system goes down, everything comes to a screeching halt until it is fixed.

Place a high value on maintainable code

Systems usually start out fast, elegant and fairly simplistic. As new edge cases and exceptions to normal processes are added, the system grows ever more slow, byzantine and complex. The more often some "quick and dirty" change is added, the larger the maintenance burden. Organizations that place a high value on maintainable code will think carefully about how changes affect the quality and complexity of the application even when a big customer is pushing for that new feature yesterday. Sometimes those market pressures require a less than optimal solution, so smart organizations add an item to their backlog to revisit the feature as soon as possible to refactor it to better mesh with the rest of the system. One organization I'm aware of schedules regular "Tech Debt" sprints to tackle just these kinds of issues.

Heed the warning signs

Systems rarely fail out of the blue. Typically errors have been spewing ino the logs for weeks or months before catastrophe happens. Developers should be scanning error logs (preferably with tools 😉) and categorizing errors. One of the best first steps when dealing with a legacy system is to catalog errors and triage them. Fixing the most common ones will begin a virtuous cycle towards a system the runs reliably and correctly.

Having a test suite with reasonable coverage and keeping it up to date is critical to making maintenance more efficient. Not only do tests warn you when a change breaks a contract with another part of the system, it also acts as documentation for the new team members so they can see how things really work.

Also heed your developers when they tell you something is starting to be a problem. They see aspects of the system you never will and they know when tasks are taking longer than feels reasonable. Add those items to your backlog and prioritize them high enough that they get addressed sooner rather than later. When you go for a drive and hear the brakes making a funny sound, you don't have to attend them instantly, but waiting too long could make for a memorable day. The same is true for software.

Don't settle for "It works"

When we build software, we often have time constraints that push us to stop when a new feature works. We need to resist that pressure at least a bit. There are a whole raft of questions we need to ask ourselves about whether this solution is maintainable. Does this component communicate it's intent clearly? Are there other components in the system that behave or look the same? Is it reliable? What can make it fail? What can we do to avoid or mitigate those failure modes? Does it put unnecessary pressure on other parts of the system? Does it follow community best practice? If not, why not? Maintainable code is written to communicate well to other developers, not just to get the computer to do something.

Don't rewrite, refactor

When we come upon a gnarly piece of code, we often think it would be easier to scrap it and start over. That is almost never the case. It is gnarly for a reason. There are probably edge cases or constraints you don't know about yet. Refactoring, including writing comprehensive tests, will make sure you know exactly what it is doing and probably why. Every component has a contract with other parts of the system. If we give up on understanding and just rewrite we are bound to break that contract and make other parts of the system fail in unexpected ways.

Leave it better than you found it

This is a mantra many of us heard growing up in scouting. It is just as true in software. Most of us don't love cleaning up our campsite, but we really enjoy coming to a nicely kept one. In software, it is often our teammate or even our future self we are making things better for. Eight years ago I inherited a project that was 10 years old at the time and looked every bit of it. By following this rule, the system gained capability while shrinking about 10% in code size and complexity. When I left the project a year ago it didn't look new, but didn't look like 2003 anymore.

Know when a subsystem needs to be redesigned

Most of the time we can refactor our systems to improve functionality. But once in a while we find that one of our components has a design flaw that we just can't work around. Rebuilding something is a tool of last resort, but necessary none the less. When that happens, the first thing to do is analyze that component's contract with the rest of the system and work to isolate the component and simplify that contract as much as possible. That work, along with the new design constraints, will set the team up to correctly specify the replacement component and add unit and integration tests for it. Even with all that work the team could cause breakage due to an incomplete understanding of the contract or uncaught side effects of the component. The team needs to plan for extra manual testing and potential cleanup when this path becomes necessary.

Take pride in our work

Let's get real. We all have code in production that we're not thrilled with. That's just a fact of life. External constraints, insufficient understanding of the feature and our own tendency to have a bad day here and there lead to bad code. But we don't have to let that code live for 20 years. When there's time, our teammates can help us out during pull request review. Otherwise we need to be transparent enough to add the non-optimal implementation to the backlog for refactoring.

Pair whenever possible

There's an old saying that "Two heads are better than one" and this is never more true then when developing. Yes, it adds coordination complexity, but having two people working together on a problem can more than double the chance of an excellent outcome. If you are a sports fan, you know that players can have an ego as big as a house off the field, but at game time they have to leave their ego on the sidelines and do whatever helps the team to win. Pairing is similar. When we come to pairing acknowledging our weaknesses and appreciating the strengths of our partner, better work comes out much faster than either would produce on their own.

Go out there and make some snow angels

Wow! We covered a lot of ground in these paragraphs. I hope you found a useful nugget or two in here. We all spend time maintaining systems that were built by others or even by our past selves. By doing that maintenance carefully we can build bright new snow angels in our brownfield projects. When we've built enough of them our brownfield can turn into a beautiful place to work and our projects can live long useful lives.