Chasing fads or solving problems?

Sample chat between two engineers

Engineer 1: We should use Hadoop/Kubernetes/Cassandra/insert-random-cool-tech.
Engineer 2: Why?

Engineer 1: Well, all the cool guys rave about it: latest-fad-tech is the next big thing on the tech horizon. All the hot startups are using it too!
Engineer 2: So what will we use it for?

Engineer 1: Well, we could use latest-fad-tech to revamp our entire data processing stack. It's so much easier than the stable-reliable-well-known-and-proven platform we currently use.
Engineer 2: So what are the benefits of moving to latest-fad?

Engineer 1: We'll look modern like the hot startups, use the latest stuff and not have to worry about data loss any more.
Engineer 2: Do we have a data loss problem now?

Engineer 1: Errm..., we don't have a data-loss problem. Do you know that latest-fad also promises automatic resource management? Our infrastructure staffing needs will go down.
Engineer 2: Our current resource management costs come to about 5 - 10% of our staffing needs. Are you telling me latest-fad has no adoption and management cost at all?

Engineer 1: Probably not - nothing is free! But it is a one-time cost and the investment will eventually pay off.
Engineer 2: OK, fair point. How does latest-fad behave under extreme load? Are failure modes well-known?

Engineer 1: Well, it is the in-thing! There are loads of people running it so I think you'll be able to Google search for problems...
Engineer 2: What happens if you run into an edge case that no one has ever run into before? Who are you gonna call?

Engineer 1: I don't know...
Engineer 2: So you're saying we should take a plunge from a stable less-cool platform to an unproven cool one?

Engineer 1: Errm, well I don't like the less-cool platform because it is not as cool as latest-fad. And latest-fad is open source too!
Engineer 2: Have you thought about the cost of moving to latest-fad? The risks of rewriting code, redesigning our stack and the productivity dip during the learning phase has to be weighed against potential benefits.

Engineer 1: That sounds like a lot of work but we really should use latest-fad.
Engineer 2: What benefit will accrue to our business and customers by switching? What needle does it move?

Engineer 1: But it'll make our developers more productive.
Engineer 2: Well, there are other tasks that will significantly increase our productivity e.g. better documentation, release automation and investing in tools. I admit they aren't as 'cool' as latest-fad though.

Engineer 1: ...
Engineer 2: Why not do a spike and come back with a comprehensive plan covering what it'll take to adopt latest-fad?

Shiny new kid on the block

If you’ve been a software engineer for some time, you probably have been engineer 1, engineer 2 or listened on a similar conversation. It’s the classic conversation along React vs Angular vs Vue; micro-services vs monoliths, AWS vs Azure themes etc.

Most times we name-drop new technologies to impress. At other times, we want to satisfy our itch and try out new stuff: when you have a new hammer, you go about finding nails to drive in!

This attitude is risky because the eagerness to wield the new hammer can divert engineers from truly focusing on the problem at hand. Problem identification and isolation should be starting point: technology is a means and not the goal.

By all means, use Hadoop, neural networks or Cassandra if they are the best tools for the problem space. However, if they aren’t, then you’ve worsened the situation for the following reasons:

1. You still haven’t solved the original problem
2. You have a new problem: maintaining a complicated technology platform
3. Supporting the ill-fitting abstraction that the platform in 2 provides for 1

Congrats! You just played yourself!! You bought two more problems for the price of one!!!

Cost vs benefit vs risk

There are three factors to be considered every time you make a technology decision:

1. Cost (C): What will it take to adopt or implement the technology?
2. Benefits (B): What benefits will accrue?
3. Risk (R): What is the risk to existing business value?

My heuristic is to green-light full adoption only if the long-term benefits outweigh the costs and risks; i.e: B > C + R. Thus a short-term dip in productivity (i.e. high cost) is totally acceptable if, in the long run, the increased developer output makes up for it.

I know that shiny is exciting but the goal of software engineers is to solve business problems and bring value to the customer/business. Technology choices are expensive and you have to think of the second-degree and third-degree impacts of your choices as technology leaders.

But I want to try out new things!

Every now and then, some new technology comes up that solves your business problem perfectly, you might outgrow your current design or even have outdated technology. Here are a few suggestions on how to avoid this:

1. Do hackathons – they are great for synergistic and symbiotic discovery of great solutions!
2. Encourage engineers to explore new approaches to problem solving.
3. Allow independent evaluations, proposals and implementations.

Think again: “What problem am I solving?”

Related

1. You Are Not Google
2. Choose boring technology
3. What you should do before you start reforms