The Radar vol 25 (Kubernetes, Kafka)

Every six months, Thoughtworks publish, radar, a review of software technologies. If you don’t know about it, or haven’t read it, then I’d recommend starting. The team at thoughtworks absolutely rock. They’re thought leaders, so sign up.

This month’s paper has some interesting notes around [[ Kafka ]] and Kubernetes. Both technologies appear to have become the “standard“ for high performance messaging and container orchestration, respectively.

# Message buses

Kafka’s position lead wasn’t guaranteed a few years ago, with RabbitMQ being a popular choice for many, including companies I worked for. We chose RabbitMQ over Kafka for simplicity and guaranteed message delivery, but Kafka caught up. RabbitMQ has the edge for more complex message routing use cases, but if you’re after scalability, Kafka is hard to beat. It’s easy to fall for a more scalable technology, but many use cases don’t require the sort of scalability that Kafka can deliver. There is no “best” technology for every use case. Understand the pros and cons of each and choose wisely for your specific use case. Once you decide, you’ll have to live with that decision for a long time. I’ve listed some references at the end of this post.

# Kubernetes

We first adopted Kubernetes back in 2014, but unlike today, you had to manage the entire stack yourself, with very few clouds [[ PaaS ]] options available at the time. A colleague once referred to as an “investment”. That was an understatement. I resisted using Kubernetes on numerous projects because of its complexity. Azure AWS and GC all now offer “managed” Kubernetes services, making it an easier choice for a wider range of projects.

There’s a section in the radar, called “Clever tech we shouldn’t need” (more on this section below). Regarding [[ Airflow ]]:

For example, we see clever workflow management tools such as Airflow or Prefect that are overeagerly used to manage complex data pipelines through orchestration.

Yep….we were overeager. It’s easy to forget [[ Airflow ]] is an orchestration tool and not a data crunching/processing/munging/cure famine/end all wars tool! We fell into the trap of using [[ Airflow ]] to do everything the rest of the stack couldn’t and it hurt. [[ Airflow ]] uses [[ Python ]] for scripting, which is a great language…..if you use it properly. Running gianormous queries for thousands of concurrent transactions per second isn’t using it “properly”. And if you’re running your own instance of [[ Airflow ]] prepare for hell once you hit performance limits. My advice, if your [[ Airflow ]] scripts are more than 10-30 lines, or if the word [[ Airflow ]] comes up 3x more than any other part of your stack, be scared. Be very scared. If you’re the CTO, starting reading up, and asking deeeeeep questions unless enough of your team have experienced the [[ Airflow ]] torture chamber. And don’t assume every understands what “orchestration“ REALLY means. An alternative stack to [[ Airflow ]] is [[ Conductor ]]…and the clues in the name. Conductors don’t try to play every f**king instrument at once, by themselves.

# Clever statements we shouldn’t need

The section “Clever tech we shouldn’t need” is a cold bucket of common sense. Don’t you just hate hearing:

”..teams often don’t realize they’re doubling or tripling down on needless complexity without stepping back to look at the big picture and question whether the current solution is worse than the problem.”

No shit. Easier said than done when you’re staring down the barrel of a deadline, and a backlog growing faster than a Swiss Alps avalanche. Of course the 45% APR remortgage (aka technical debt) we took wasn’t our best decision. It’s the opposite of what we intended, but try explaining refactoring and complexity to a leadership team who can barely operate a calculator, and under pressure from investors. You hear your own voice tailing off as you suddenly remember you’re part of the leadership team. Specifically, the part that’s gotta explain it…to the investors.

“Stepping back”….I’ve got [[ ADHD ]]….. We don’t step back. Ever. We eat people who aren’t stepping forward, which is only 96% of the population…..🤨. “Stepping back“…..that’s like telling my wife to “calm down“ when she’s annoyed. I’d rather take my chances filling up the car with a lit cigar hanging out of my mouth. (In case you missed the joke, don’t think EVs or Tesla 🙄).

# Remote spontaneous huddling interruptions

Radar, Section 14:

“We’re seeing continued innovation in remote collaboration tools. The new Huddles feature in Slack provides a Discord-like experience of persistent audio calls that users can jump in and out of at any time. Gather provides a creative way to emulate a virtual office with avatars and video. IDEs provide direct collaboration features for pairing and debugging: we’ve previously blipped Visual Studio Live Share and included JetBrains Code With Me to the list in this edition.“

As if getting interrupted every 30 seconds (on a good day) by Teams, Slack, Jira, and of course email, isn’t enough, please eat up the last 100 microseconds of time I have left with more “collaboration”. Innovation would be a big red button to shut down all collaboration tools in one fell swoop! Stop interrupting me. It’s not cool to vomit your conscious stream over me….I know, it ain’t the tools, it’s the way we use them. Why don’t we insist on some mandatory don’t-do-this training before we let everyone loose? It feels like handing a prefrontal-cortex-still-developing 17yr old a loaded gun, minus the safety briefing. Good. Luck.

For an industry in love with Agile / LEAN and talk of eeking out a few secs here and there (cos it all adds up), we do some counter-intuitive stuff. Are we deliberately trying to make “flow“ harder? Is this a test to keep us on our toes? Let me introduce you, to a new concept. I call it “going solo”. It really works.

# Enough

It’s Sunday, I just needed to vent. I’ll be back to my normal, responsible self on Monday.

# References

Notes mentioning this note

There are no notes linking to this note.