A Startup Serial Killer Masquerading as a Software Architect
10 June 2021
A few years back, I joined a startup in the Cloud Security space. I was hired to take ownership over the React app they had built. On my first day, I met the rest of my new co-workers. One of which was responsible for the API that I needed to interact with, let's call him Happy.
I had enough problems to deal with in getting their web application ready for prime time, so I didn't focus a ton on the backend. I assumed that it was being competently built. That is until I started having to nag people to make the endpoints I need available. This pushed me to start understanding how the backend worked.
Building cloud-native applications, was not new to me. I hadn't done a ton of work on AWS, so that part was new to me, but Happy seemed to know what he was doing. I was a JavaScript developer and the backend was entirely Python, so I didn't have the confidence to speak up right away when I first felt that something was amiss.
As problems started to compound and our system couldn't even handle a single user, I knew it was time to do something, so I started challenging the current architecture that looked something like this:
- Every entity in the system had a micro-service, which were more like nano-services, that was responsible for transforming request data into a database object and then from a database object back into response data. I'd guess there were roughly 10 of these.
- We had a micro-service for the API that would terminate HTTP requests.
- We had a micro-service for the Database that all DB communication had to go through.
Feeling icky yet?
These services would talk via RabbitMQ (a standard message queue), the path a request for a User took through the system was:
API -> RabbitMQ -> User Micro-service -> RabbitMQ -> Database Micro-service -> RabbitMQ -> User Micro-service -> RabbitMQ -> API
This wasn't specific to the user entity or just fetching data. Every request for an entity or to mutate an entity would follow a similar path. Please, please do not do this.
One reason Happy chose his overly complicated architecture was due to using a Python Framework called Nameko. I highly suggest that if you hear someone discussing Nameko to run in the other direction as fast as possible.
For the majority of tech startups, this should instead look more like:
API -> DB -> API
You might have the API put tasks on a queue like RabbitMQ, or something like Kafka, bound for other micro-services. But, waiting for that task to complete and respond with a message on another queue is a recipe for disaster. If you need the response from a micro-service in order to respond to the request, that means you probably shouldn't be communicating with that micro-service over a message queue, but instead making a direct API call.
I call this the hub-and-spoke architecture and, in my humble opinion, it is ideal for early-to-mid stage products.
Maintaining a bunch of different micro-services is a real pain for a small team, but sometimes slow deployments or other things are very good reasons to split something out into its own micro-service. The primary benefit of micro-services is it better enables more work to happen in parallel, but on a small team, this usually isn't a big problem.
With Happy's architecture, our problems only continued to get worse. Eventually, our CEO asked Happy what we'd need to do to get the system ready for production, and Happy's response was: "Re-write it in Java."
Happy got canned that day and we started re-writing code for the backend, because sending anything besides a non-trivial request to the system would cause it to fall over.
If that was the end of the story, I wouldn't be writing this. But, here we are.
Last night, I was at a networking event. And I was talking to the former CTO of the company who hired Happy after us. Out of curiosity, I asked him what he thought of Happy. After a bit of back and forth, I mentioned RabbitMQ, and the CTO's eyes got big and he says, "Happy and fucking Rabbit, man."
To which I responded, "oh my god, he did it to you too didn't he?"
Happy had left one KC-based startup on life support because of his software architecture decisions. He then went and did the exact same thing to another KC-based startup without learning from his experiences at the first that his approach to micro-services was completely and fundamentally broken.
I totally understand that people get in over their heads and are afraid to tell people that they don't know. But, when you are being hired for portraying yourself as a competent software architect and clearly not willing to learn from your incredibly costly mistakes. And when that arrogance then causes startups in Kansas City to be significantly more likely to fail, well I do have a problem with that.
I know Happy will strike again, but maybe reading this will give you the confidence to put a stop to it in time. Other than trying to increase awareness that choosing a bad software architect (/lead developer/CTO, whoever has that role) can really set your tech startup back. The only other thing I can think to do is make this offer:
If you are a KC-based non-technical founder/CEO/concerned party, and you'd like a neutral third party to evaluate your web application's software architecture, please reach out.
I've worked in startups virtually my entire career and I definitely understand that sometimes the code isn't going to be pretty. However, that's not the same thing as making sure to get the things that are really hard to change aren't wildly wrong up front.