Why computer science matters for software developers


I used to think computer science never mattered because I rarely used algorithms and never saw the value of algorithm-based interviews (I still don’t ūüėČ ). The few folks I asked also concurred so I felt I was right.

September 2016

My team got reorged and our goal was to build and deliver a brand new SaaS offering. As part of that move, I made a conscious decision to switch to full stack engineering + dev ops. My experiences have made me re-evaluate my initial stance and realize I was wrong.

Computer Science does matter, a lot! Having that foundation empowers you to

  • Make better tradeoff decisions
  • Innovate new ways of solving problems
  • Spot design pitfalls early; e.g. a design that violates CAP theorem is a disaster waiting to happen.
  • Avoid solving impossible problems e.g. trying to parse HTML with regex.

The following paragraphs show how computer science principles underpin common concepts.

1. File systems and the HTML DOM

What do hierarchical file systems have in common with the HTML DOM? Simple, both are based on trees. Some common operations on trees include reparenting, search and walks. The table below shows how file system and DOM operations can be tied back to basic tree operations.

Tree Operation File system DOM
Search File search Search
Traversal Directory listing Layout rendering
Node reparenting File move Hiding and showing sections of the DOM

Having this foundation allows you to ask deeper questions. For example, let’s analyze the reason behind the discouragement of excessive DOM manipulations.

It’s not the DOM – tree operations are fast. The challenge lies in repainting the entire HTML layout when deep-lying nodes change. If you keep moving nodes around, the browser has to play catch up and this adds up over time.

The ‘best practice’ is to detach the node, manipulate it and then re-attach it. This approach means the browser repaints twice – on attach and detach. The detached node can change several times without triggering DOM reflow since it’s no longer attached.

2. Solving scheduling problems

I wrote a timetable generator in PHP as an intern 8 years ago. Yes, it was a brute force solver and took about an hour to generate a timetable for an average-sized school.

A quick inefficient solution can get you to the market fast and might even work well at small scale. However, any attempt to extend or improve such solutions would be prohibitive. My brute force solution of 2009 would have broken for larger problem sets; a trivial ask such as introducing a new constraint e.g. multiple teachers for multiple classes, would have necessitated a rewrite.

The timetable problem is a constraint satisfaction problem (CSP). Other popular CSP problems include appointment scheduling, map colouring and the 8 queens puzzle. Backtracking search is a standard way to solve CSPs. Solvers can also leverage greedy search, minimum-conflicts heuristics and simulated annealing.

This approach separates the problem from the solver; thus it becomes easy to change the constraints and extend the solver to new scenarios.

3. Avoiding stack overflows

How would you make sure your recursive function never runs out of stack space? This might not be a problem if the language optimizes tail calls but alas most languages don’t.

  1. You could catch the stack overflow exception but that doesn’t¬†mean your computer can’t calculate the value. It just means the recursive implementation exceeded the computer’s stack memory limit.
  2. You could convert the recursive function to a loop. This would require passing in values around and might not be so elegant.

A trampoline solves the problem beautifully and can be reused for all recursive functions and across languages. Read this post to learn how a trampolined factorial function allows you to compute the factorial of 30000.

4. Consistently parsing HTML with Regex?

A common joke is that experienced programmers send newbies on a wild goose chase by asking for a regex parser for HTML. It’s been long since I did any automata course but the short answer goes thus:

  • Regular expressions are a regular grammar (Type 3)
  • HTML is a¬†context-free¬†grammar (Type 2)

Type-2 grammars encompass Type-3 and are thus more complex. See the Chomsky hierarchy.

It might be safe to do this for a small set or to extract data out of pages. But saying this is something that’ll work all the time is not true.

Tip: Know the class of impossible computing problems and save yourself from wasting time on futile challenges.

Conclusion

Concepts like design patterns, computer networks, architecture, etc. all matter in the software engineering profession. However, having a solid CS background is key.

There are loads of opportunities to apply computer science principles to daily tasks – you just have to make the push.

Thoughts?

Related

The Effective Programmer – 3 tips to maximize impact


Effectiveness, (noun) : the degree to which something is successful in producing a desired result; success.

Over the years, I have tried experiments, read books and watched several talks in a bid to improve my effectiveness. After a series of burnout and recovery cycles, I finally have a 3-pronged approach that seems to serve me well.

1. Learn to estimate well

2. Use the big picture to seek opportunities

3. Continuous Improvement

Lets discuss these three.

1. Estimation – the bane of software development

Reliable coding estimates accurately¬†forecast when feature work will be done. But when is a feature done? Is it when it is code complete?¬†Test complete? Or deployed? Most developers wrongly associate code complete with test completion or deployment ready.¬†This explains¬†arbitrary estimates like: “Oh… I’ll be done in 2 hours”; such estimates typically miss the mark by wide margins due to error compounding. Let’s take a simple bug fix scenario at a¬†fictitious¬†software engineering company.

  • Bug is assigned to developer John¬†SuperSmartz
  • John¬†SuperSmartz reads the bug description, sets up his environment and¬†reproduces it
  • He identifies the cause but¬†does some light¬†investigation to find any¬†related bugs¬†(he’s a good engineer)
  • John designs, implements and verifies the fix
  • Gets code review feedback and then checks in

Any of the intermediate steps can take longer than estimated (e.g. code reviews might expose design flaws, check-ins might be blocked by a bad merge, newer bugs might be discovered in step 3. etc). Without such an explicit breakdown, it becomes difficult to properly give estimates.¬†Don’t you now think the 2-hour estimate is too optimistic?

Personally, I use kanbanFlow (I love their Kanban + pomodoro integration) to decompose work into small achievable 25-minute chunks. For example, I might break down some feature work into 8 pomodoros as follows:

  • Requirements clarification – 1 pomodoro
  • Software design and test scenario planning¬†– 2 pomodoros
  • Coding (+ unit tests) – 3 pomodoros
  • Testing and code reviews – 1 pomodoro
  • Check-in + estimation review – 1¬†pomodoro

Some of the things I have learnt from using this approach:

  • I grossly underestimate feature work – the good side though is that this planning¬†enables me to improve over time
  • I know when to start looking for help – as soon as a task exceeds its planned estimate, I start considering alternative approaches or seeking the help of a senior technical lead
  • Finally, it enables me to make more accurate forecasts¬†– e.g. I can fix x bugs per week…

2. See the big picture

A man running around in circles covers a lot of distance but has little displacement. In optimal scenarios, distance covered equals displacement while in the worst scenario, it is possible to cover an infinite distance and have a displacement of zero.

Imagine working for several days on a feature and then discovering major design flaws that necessitates a system rewrite; a lot of distance has been covered but there has been little displacement. Working on non-essential low-impact tasks that no one cares about is neither efficient nor effective. Sure they might scratch an itch but always remember that the opportunity cost is quite high; the lost time could have been invested in higher priority tasks with a larger ROI.

Whales periodically surface for air and then get back into the water to do their business; so should engineers¬†periodically verify that priorities align with company’s goals. It’s possible to get carried away by the deluge of never-ending feature requests and bugs fixes; an occasional step back is needed to grasp the whole picture. Here are sample questions to ask:

  • Where are the team’s goals?
  • Does your current work align with company goals?
  • Skills acquisition and obsolescence check
  • Opportunities for improvement?

Personally I try to create 3 to 4 high-impact deliverables at the beginning of each week and then focus on achieving these. Of course, such forecasts rely heavily on productivity estimates.

3. Continuous Improvement

Athletes consistently hold practice sessions¬†even if they don’t want to because it’s essential to staying on top of their game. The same applies to pretty much any human endeavor – a dip in momentum typically leads to some loss in competitive edge.¬†The software engineering field, with its rapidly evolving landscape, is even more demanding – developers have to continuously and¬†relentlessly learn to stay relevant.

Staying relevant requires monitoring industry trends vis-√†-vis blogs, conferences and newsletters. There are a ton of resources out there and it’s impossible to follow every single resource, rather it is essential to separate the wheat from the chaff and follow a select high-quality few.

Learning and experimentation with new technologies naturally follows from keeping abreast of developments. A developer might decide to learn more about the software stack his company uses, logic programming or even computer science theory. Even if his interests are totally unrelated to his day-to-day job, independent learning would expose him to new (possibly better) ways of solving problems, broaden his capabilities and might even open up new opportunities. I prefer learning established CS concepts to diving into every new db-data-to-user-moving framework.

Opportunities abound such as learning IDE shortcut keys, terminal commands, automating mundane tasks and so on. Ideally you want to start simple by selecting the area with the highest impact-to-effort ratio and then dedicating a few minutes to it daily. Over time, the benefits start to pay off.

And that’s about it! Do you have other suggestions?

Like this post? Follow me on Twitter here or read some of my top posts.

1. Code is poetry: 5 steps to bulletproof code

2. So you want to become a better programmer

Are developers better than testers?


A lot of people view testers as second-class citizens; probably because QA guys rarely ‘create’ stuff like devs do (although they sometimes do original work creating test frameworks and platforms). They are more similar to editors who verify the work of authors. However, testers are just as important as developers; without QA the end product is bound to be buggy and might even miss out on the requirements!

Developers usually believe they write bullet-proof code, (after a couple of ‘subjective’ tests which assume some ‘condition’ will never happen and so on…); however as we all know it; this is not true.¬†I believe every software project requires testing ‚Äď yes QA is that critical.¬†You can’t be really sure that some complex piece of code works properly, you can only be sure that you didn’t find bugs. Even well-known APIs, languages and frameworks have issues, for example, a¬†bug¬†was found in the Java implementation of binary search some time ago.

Testing requires a different skill set from development and all programmers will benefit vastly from spending some time as testers. In fact a Microsoft Test lead said the best developers he knows all started out as testers. This is not far-fetched; such test-devs do more automation of mundane tasks, know more about potential failure scenarios and try to guard against it early in the writing process; moreover they test their own more thoroughly leading to more reliable software.

Testers have to make sure the product is robust, matches the customer’s requirements and find out flaws the developer probably overlooked while writing his code (e.g. passing in wrong parameters, graceful degradations, bad failures etc). They always have to devise ingenious testing scenarios that expose flaws and defects in system design and/or implementation. Contrary to opinion, testing might involve writing code to test code (automation) and building testing frameworks/platforms; some testers probably get to write more challenging code than developers.

Why do devs/PMs etc dislike QA?

Imagine having to tell proud parents that their beloved child is ugly? Sounds tough right? QA guys have the unpleasant duty of pointing out the flaws in the work of devs and PMs. Well, no one likes to be criticized or told they made mistakes after working so hard on a feature. It’s not difficult disliking guys who routinely point out the flaws in the design, system implementation or find out bugs in your code.

Pros of QA work:

  • Testers understand the project requirements and have a wider knowledge of the product life-cycle.
  • They ideally work well and interact with a lot of team members ‚Äď PMs, devs and other QA guys.
  • They learn to be patient carrying out repetitive work and come to appreciate automating lots of things.
  • They have great analytical skills ‚Äď well that‚Äôs all what testing is about right?
  • They learn how to communicate, how to be impersonal and tactful and how to diffuse tense situations when they happen.
  • Testing skills come in handy if you ever switch to development.
  • Replacing good testers can be a very difficult task… talk about job security ūüėČ

Cons:

  • As I said earlier no one likes being told their brainchild is ugly even though it‚Äôs true. You might have to handle friction.
  • Yes, it might get extremely boring repeating the same things but you can always up your game.
  • Testing experience might not count as development experience.

Well, if you still believe developers are better than testers; that’s fine. Just learn how to test your code properly and save us all the pain of bad software and evil failures. Moreover, don’t feel bad if a tester shows you 7 bugs in your one-of-a-kind-awesomeness-embedded-wonderful-piece-of-code.

Still think testers are inferior to devs? Have your say in the comments.