How to develop unmaintainable software

I get paid to take on technical debt. In my work I see a lot of hard-to-maintain code, and I see many of the same avoidable problems over and over.

I specialize in debugging, fixing, maintaining, and extending legacy software systems. My typical client has a web site or internal application that works, more or less, but the original developer isn’t available. Business requirements have changed and the software hasn’t kept up. Or my client has something that is “almost finished” but they parted ways with the developer after using up their budget and schedule. Usually there’s a list of missing features and bugs.

My typical client usually has been told by other developers that they need to throw everything away and start from scratch. Most programmers don’t like maintaining code, and they especially don’t like maintaining someone else’s code. When writing code programmers often ask the wrong questions when they talk about maintainability—see Matt DuVall’s article The myth of maintainability for a good description of how that happens.

Here are some things you can do in your own software projects to keep me in business:

Don’t use version control

I’m always surprised to find big projects written in the last few years that aren’t under source code version control. When you don’t use version control the next programmer to come along has to figure out which files are part of the current system and which are obsolete or backups. The next programmer won’t have any commit messages or change logs to get the code’s history from. I covered how not to use version control in my article Introduction to Abject-Oriented Programming.

Customize your development environment. A lot.

Don’t make it easy for the next programmer to start working on the code. Require specific versions of languages and tools, and make sure they conflict with the versions that come with the operating system. Customize your Eclipse or VisualStudio or vim environment like crazy, then write macros and scripts that only work with that setup. Don’t have a disk image or scripts to replicate your development environment, and don’t bother writing any documentation—it will be intuitive.


Create an elaborate build and deployment environment

For a web site deployment to a staging or production server should look like one of these:

svn up
git pull
hg pull

Programmers can argue about simplicity and elegance in code, and then turn around and build the most elaborate and baroque build and deployment systems. This is one of the off-the-radar things that programmers do without their client or project manager reviewing or understanding it, so it easily gets out of control. While you’re chaining together eight different tools with various scripting languages remember to leave out the documentation.

Don’t set up testing/staging platforms

Changing a production system is exciting. Don’t bother with a testing/staging server. Instead have secret logins and backdoor URLs to test new features. Mix test data with real data in your database. Since you aren’t using source control keep copies of previous versions just in case. Don’t build logging into the code. Don’t disable outgoing emails, credit card authorizations, etc. during testing.

Write everything from scratch

Don’t bother with a well-understood framework like Django, Rails, or CakePHP. You can write a better templating engine, ORM, sort or hashing algorithm. I scratch my head when I see code with comments like “faster than native dictionary method” or “replacing PHP library function because parameter order sucks.”

Add dependencies to specific versions of libraries and resources…

Throw in as much third-party code as you can. Link to as many shared libraries as you need to. I’ve seen code depend on large external libraries just to get access to one function. Modify the source code of the third-party libraries so they can’t automatically be updated to a newer version, but don’t bother forking or keeping your modified versions under source code control. I will be able to diff your version with the original and figure it out.

 … But don’t protect or document those dependencies

I get more urgent calls because of updates and upgrades gone wrong than for any other reason. A seemingly innocuous WordPress upgrade, Linux package update, or new jQuery release will trigger a chain reaction of failures. Don’t make your code check for the specific version or modified copy of your external resources or third-party libraries. Don’t even add a comment to remind yourself.

Use a bunch of different programming languages and stay cutting-edge

Every day HackerNews and Reddit buzz with cool new languages. Try them out on your client’s time. Any decent programmer can learn a new language in no time, so if the next programmer who gets to work on your code is a newb that’s not your problem. The boundaries between the languages, the incompatible APIs and data formats, the different server configuration requirements are all fun challenges to overcome and post about on StackOverflow. I’ve actually seen PHP web sites with pieces of Ruby wedged in because everyone knows PHP sucks and Ruby is better. Half-baked caching, aborted Rails and Node.js projects, and especially NoSQL solutions (“It’s more scaleable!”) are my bread and butter.

Where’s the programming advice?

It won’t matter much if your code is beautifully object-oriented or shiny and functional—programmers pretty much always see spaghetti code when they have to maintain a legacy system. I am good with diff and grep and Ctags, tracing through code, refactoring, and debugging. I’ll figure out the code. The most beautiful, elegant code is still hard to work with if there’s no source control, too many dependencies and customizations, and no testing/staging facility. It’s like finding one clean and nicely-decorated room in a house straight out of “Hoarders.”

Related and possibly interesting: The joys of maintenance programming.

52 comments ↓

#1 mihai on 10.14.13 at 9:56 am

Great article and very applicable for consultants. Main point here… is that you have the ability to decipher others code and new developers should practice working on others code as a lesson. Even with the issues you mentioned you can get through it all, just more time is required thus higher costs, which works in your favor as a consultant :)

#2 Jon on 10.14.13 at 11:05 am

I recently was hired by a startup to finish an Android and iOS app that was initiated by a consulting company. After getting my hands on the source code, I was shocked by the poor quality of the code. There was commented out code everywhere, no source control, duplicated code everywhere, lack of any discernable design patterns, and perhaps worst of all, modified libraries that were not properly forked.

I wasn’t completely sure that I made the correct call when I decided to start from scratch, but your article has me thinking it probably was for the best.

Great read!

#3 greenone on 10.14.13 at 1:34 pm

thats what it feels like programming magento… no documentation and horrible dependencies… often feels like what you describe

#4 The internet on 10.14.13 at 2:33 pm

Hey look, its a rewording of half the joel test

#5 Darth Continent on 10.14.13 at 3:36 pm

tl;dr: Job security.

#6 web technologist on 10.14.13 at 9:31 pm

[…] By the way this is the post. […]

#7 Greg Jorgensen on 10.14.13 at 11:03 pm

Thanks for reading and commenting. Joel covers a lot of same ground (http://www.joelonsoftware.com/articles/fog0000000043.html).

This stuff has been discussed since the dawn of programming but it still happens every day. There are always new programmers who have never heard of Joel Spolsky, to say nothing of Knuth or Brooks.

#8 magentoman on 10.14.13 at 11:45 pm

Yep, this looks exactly like Magento developing!

#9 Michael Jones on 10.15.13 at 1:01 am

I liked the article. Excellent stuff, though frankly quite concerning :)

Completely unrelated, I wanted to say that I found it very hard to find a visited link on your page. It blends in almost perfectly with the plain text. I can kind of see the logic in that but it was not helpful when I wanted to find the link again and was scanning for coloured text. All good though, just wanted to read more of your stuff!

#10 Belarius on 10.15.13 at 1:43 am

I’ll just leave this here.

http://web.archive.org/web/20130930122540/http://thc.org/root/phun/unmaintain.html

#11 Greg Jorgensen on 10.15.13 at 2:09 am

@Michael Thanks, the visited link color was bugging me too. I fixed it.

@Belarius Thanks, much more complete list of bad practices to avoid.

#12 Jamie on 10.15.13 at 3:25 am

I never post on blogs but this is just hilarious. Great read!

#13 name on 10.15.13 at 1:35 pm

thakhna

#14 Henryk on 10.15.13 at 3:57 pm

I wonder if you really think “svn up” is always the best way to deploy a site. I wish you were right.

#15 Bob on 10.15.13 at 4:31 pm

Thanks for giving me the information I need to maintain job security!

#16 Gareth on 10.15.13 at 4:36 pm

I used to do something similar… I ran into a super smart architect who had decided to re-write the IBM JVM garback collector to fix his app… he spent a morning describing the work to me.

I stepped in that afternoon and within 1-day had reduced the garbage created by about 95%. The gc was not longer an issue!

#17 egocmsuser on 10.15.13 at 4:40 pm

I would add:
1. Do i18n with branching logic, because English, German and Turkish does it all.
2. Use multi-tier infrastructure architecture to bill more. Then setup staging servers, but automatically sync every content to production, because WYSIWYG you know?!
Workflow and versioning are hard, let’s go shopping…

#18 Lou Collins on 10.15.13 at 4:41 pm

Loved it!
I once had a client with a vital data analysis program written in FORTRAN by a user who’d learnt some programming. EVERYTHING was hard-coded, and every time the rates for any of their products changed, he would edit the program (in multiple places). The thing was full of magic numbers and COMPLETELY uncommented. After he died unexpectedly, I had to work out what the code did, working jointly with his former colleagues who I needed to recognise tables of apparently random numbers – “Oh, that looks like the prices for next-day interstate delivery for parcels under 500g”
It was like being on a treasure hunt. I reckon that job had more “Ah-ha!” moments than any other I’ve had.

#19 Marshal on 10.15.13 at 5:54 pm

In a wierd way, is it good for the economy and society to have some “spagatti” programmers in financial industry that can engineer wealth un-hoarding by top 1%? :). They might induce the company to create “new initiatives” new projects because old ones don’t work, decomissioning-projects, integration-projects etc that keeps more people in job [the middle/poor family have bread on the table]. Of-late I see finaincial industry kicking out old guys [i.e 35+ ] and getting in interns-with-peanuts – that’s a yearly ritual in many financial institutions. Hmm.. I know that this arguement sidelines the “clean code” concept, but can’t stop fathoming the other dimensions of life like greed/shut-down. Obviously we all like “efficient software that requires less people to maintain, and runs for decades”.

#20 Fill on 10.15.13 at 5:58 pm

I inherited a failed project once that was written in Rails. It was developed full time for 3 years. They named methods, variables and etc. using nonsensical code-names. In spite of using a MVC framework, all the controller code was mashed into one giant controller! Views were stored in a database, and they caught all exceptions (including legit ones generated by bugs) and redirected the user in their own custom type of routing (I imagine due to some assumption that errors were the users’ fault).

On the flip side, I’ve worked with brilliant programmers who take things to the extreme adding worthless overhead for no practical gain. For example, forking the code at every feature addition even though there’s only 2 programmers, or demanding daily scheduled ‘scrums’ even though, again, there’s just two programmers and they are in constant communication during the work day anyways.

#21 Maintenance Man on 10.15.13 at 9:29 pm

It is very lucrative to have to clean up messes left by other programmers. Gets real fun when they are large systems. Been paying the bills for over 10 years doing this very thing.

#22 Paul Colombo on 10.15.13 at 10:39 pm

I work at a digital agency that has inherited or updated no less than 6 websites in the past 2 years. Not to mention projects we’ve had to outsource and then maintain. I have seen everything on this list first hand, and thank you for posting it. I struggle to get our account, PM and finance teams to understand the importance of allocating time to do things right, and that cheaply outsourced projects cause long-term headaches.

You sir, are my hero.

#23 Typical Programmer – How to develop unmaintainable software | Alvaro Uribe on 10.16.13 at 12:19 am

[…] Typical Programmer – How to develop unmaintainable software. […]

#24 Antti Hatinen on 10.16.13 at 12:40 am

I would like to add a specific programming style I’ve met with Indian spaghetti-code : avoiding bugs by catching all exceptions at all levels, and showing the pages to the user like nothing bad happened (like database connection is not available) . It works great if the testers / customers don’t really understand how the software should actually work, and the app just looks like it’s almost right (but in reality doesn’t do anything). I think it’s related to avoiding “loss of face” via having a “national?” strategy of putting more effort in hiding mistakes than actually writing code that works.

#25 Trevor Turton on 10.16.13 at 1:23 am

Great post, I recognised lots of the problems that you described. I’ve even written a few in my time. It would be very funny except it’s tragic. To call such code “spaghetti” is flattery, rather call it macaroni because it’s full of holes.

#26 David Gerard on 10.16.13 at 3:11 am

Don’t forget the documentation to match!

HOWTO: write bad documentation that looks good

#27 PlainSimpleDeveloppper on 10.16.13 at 3:36 am

I jumped from a company in june because of this kind of ‘politics’ ; their products were such a mess, coding for them was a nightmare. For instance the guys who conceived the database didn’t use any relation between their 100 tables. why so sicne they could code their own sets of triggers to assume referential integrity ? :O try to play in a rdbms with more than 400 triggers ready to shoot at any move…
some love to suffer

#28 Greg Jorgensen on 10.16.13 at 3:49 am

@Henryk of course svn up (or equivalent) is not always the best way to deploy–I was use hyperbole to make a point. In fact a lot of web sites can be deployed with svn up (or svn export if you prefer), but there are things like database changes, possibly race conditions on a busy site, language translations, etc. that may require a more sophisticated build/deployment system. What I am complaining about is unnecessarily complicated and clever build/deployment systems that only one person understands. I’ve worked at software companies where a build/deploy was a half- or full-day operation.

#29 Greg Jorgensen on 10.16.13 at 3:54 am

@Marshal It’s bad enough to find these things in older legacy apps, but I have seen some of these mistakes in code that is still under development. It’s a “first big system” problem, where a group of smart but inexperienced programmers get carried away and try to out-clever each other.

@Fill I have a theory that programmers suffer from OCD at a much higher rate than the overall population. It makes us good at programming, but also makes us prone to ritual behavior like forking every change, obsessing over tabs vs. spaces, holding pointless scrums, etc.

#30 Greg Jorgensen on 10.16.13 at 3:57 am

@PlainSimple I’ve written about RDBMS abuse. More than a few times I’ve been involved with projects where an RDBMS was rejected or abandoned because it was too slow or wouldn’t scale (theoretically). Some NoSQL replacement was introduced, but then the data quickly became inconsistent and corrupt. When I look at the abandoned “relational” solution I see every beginner mistake that made it slow and un-scaleable in the first place.

#31 Greg Jorgensen on 10.16.13 at 4:17 am

These comment threads are far more entertaining:

http://www.reddit.com/r/programming/comments/1ofkgu/how_to_develop_unmaintainable_software/

http://developers.slashdot.org/story/13/10/15/2159253/how-to-develop-unmaintainable-software

Disappointed there were no comments on Hacker News. I don’t know if this is just not interesting to that crowd, or if the article was lost among the breathless venture capital/startup news.

#32 carlo on 10.16.13 at 4:24 am

Or the opposite. Im working on a system that uses J2EE, JPA, JSF, Eclipselink, JMS and Oracle (in other words ‘prevalent frameworks’). It’s all ‘Java’ version controlled. We have a test server and standard IDE and build sequences. There’s even documentation.

But its still a terrible piece of software.
Why? Because its a monolithic hotchpotch with little to no architectural or design oversight.

#33 Richard on 10.16.13 at 4:52 am

We wrote a lot of stuff like ORMs back in the late 90s when the available platforms were somewhat more scant. It means we still have a lot of code on that basis, though it is documented and maintainable. We still build on it today. Going to something like Hibernate would be prohibitively expensive.

#34 Mr. Bear on 10.16.13 at 5:41 am

Oh, I love those NoSQL kool-aid drinkers. They think they are so incredibly brilliant because they have discovered a database table with a ‘key’ column and a ‘value’ column.

#35 davidwr on 10.16.13 at 7:07 am

Writing unmaintainable code is easy.

The hard part is realizing why it is unmaintainable so you don’t do it again.

This – or something like it – should be required reading for every new programmer and everyone in leadership/management/executive offices regardless of experience.

#36 Joe on 10.16.13 at 10:34 am

Great article. Could use some clean up. For example in the “svn up” section you break character. It would be great if you kept up the farce throughout so that a credulous reader thinks this a bona fide recipe for writing software. Do that and I’m sure some manager will mail this around as a codification of all his hopes that all that “fluff” his/her devs are busy with is an unnecessary waste of time.

#37 Greg Jorgensen on 10.16.13 at 1:28 pm

@Joe I already wrote that article: Introduction To Abject-Oriented Programming. Too many people took it seriously, I’m worried I did some damage to my profession.

#38 Greg Jorgensen on 10.16.13 at 1:30 pm

@Bear Wait until the NoSQL people (re)discover fixed-length records with fields at specific byte positions, and embedded pointers to the physical position of related records. That will rule because it’s fast and scales.

#39 Fill on 10.16.13 at 3:42 pm

@greg Oh, I’ve definitely had my fair share of hires and friends who were OCD. I think that can help make for a good programmer/engineer, but it can also cause problems with staying on task and keeping a grasp of the big picture as they work. There’s what I call “going down the rabbit hole”, as we touched on, where somebody gets so entrenched in the details of something that they spend all their time over-engineering it and lose grasp of the bigger picture. I’ve also had a hire go home early sick with “OCD” (that was a new one, I had never heard of somebody doing that before). On the other hand, that hire would occasionally elect to do an all-nighter to crank out a project from soup to nuts, which was great.

#40 Totally Unmanageable Code In 8 Easy Steps on 10.17.13 at 10:34 am

[…] translates into job security for him and others like him. To make it all easy, Glen put together a list of simple steps that any organization can follow to code themselves into a corner and end up calling someone like […]

#41 engineeringevil.com on 10.17.13 at 7:50 pm

Hmm it seems like your site ate my first comment (it was super long) so I
guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your
blog. I too am an aspiring blog blogger but I’m still new to
the whole thing. Do you have any suggestions for newbie blog writers?
I’d certainly appreciate it.

#42 Greg Jorgensen on 10.18.13 at 3:11 am

@NoMoreH1Bs I hate to remove comments but this isn’t the place to disparage programmers based on their ethnicity or nationality.

#43 Alessandro Gasparini on 10.18.13 at 6:19 am

Hi Greg,
I must tell you that your article is fantastic, really funny, to the point that I’d like to hire you to make a review of our product LogicalDOC. . Anyway I want to recommend you to add some social widgets to your site, such as Facebook like, Twitter or other, in order to increase (if possible) its popularity.

#44 Alessandro Gasparini on 10.18.13 at 6:22 am

In fact I saw now that you have a twitter followme button, but I suggest you also add a button like Twitter as on our website

#45 Steve Nelson on 10.18.13 at 8:00 am

Great Article! I was shacking my head (up and down) and laughing at the same time only because I’ve seen the same thing so many times.

#46 Addinall on 10.20.13 at 4:34 pm

“Don’t bother with a well-understood framework like Django, Rails, or CakePHP. ”

Over the last decade I have seen some REALLY poor systems. Maintaining the stuff was just out of the question. The REALLY REALLY bad ones all used well known MVC Frameworks and produced a horrible complex mess then the result should have been some light footprint OOD/OOP PHP.

#47 How to develop unmaintainable software on 10.21.13 at 3:51 am

[…] by gregjor to programming [link] [102 […]

#48 Ironic Programming How To Guide Teaches Programmers How Not To : Stephen E. Arnold @ Beyond Search on 10.28.13 at 9:02 pm

[…] article from Typical Programmer titled How to Develop Unmaintainable Software is written by a programmer who spends most of his workdays debugging, maintaining and fixing […]

#49 Cristina on 11.11.13 at 2:10 pm

Hi! I know this is kinda off topic nevertheless I’d figured I’d ask.
Would you be interested in exchanging links or maybe guest authoring a blog article or vice-versa?
My site discusses a lot of the same topics as yours and I feel we could greatly benefit from each other.
If you happen to be interested feel free to send me an email.
I look forward to hearing from you! Wonderful blog by
the way!

#50 eugel on 11.26.13 at 1:05 am

Another case. Consider an established enterprise. All the code is documented and controlled for decades (25+ years). Somebody writes a function and it works for 10 years. Then a precision bug occurs in this function. One cannot change the code without impacting the previous behavior and data. So they fork. And another decade pass. The issue is documented but the people are gone. One day you decide to call the “new” function and discover that it is slow and its precision isn’t sufficient. However the “old” function gives you what you want though you know that once in 10 years it fails… It’s a true story from my real life.

#51 Zol on 05.08.14 at 12:31 am

One way to write unmaintainable code is use short names and no documentation, and print same content in logs from three different methods.

#52 Jak pisać kod niemodyfikowalny | Dawcy Mózgu on 11.02.14 at 2:01 am

[…] How to develop unmaintainable software – Greg Jorgensen, 2013 […]

Leave a Comment