The difficult parts of software development


Time for a classic rant again; yeah it’s always good to express thoughts and hear about the feelings of others – a good way to learn.

Lots of people think the most difficult aspects of software development revolve around engineering themes like:

  • Writing elegant pieces of code that are a joy to extend and scale beautifully
  • Crafting brilliant algorithms that can land rockets on small floating platforms (yup, SpaceX, I see you…)
  • Inventing new cryptographic systems (please don’t attempt this at home…)
  • Building and maintaining massively parallel computation systems

Agreed, these are extremely challenging and sometimes it is difficult to find a perfect solution. However, since most of these deal with code and systems, the required skills can be learned and there are usually workarounds available.

There is more to software development than engineering and these other facets can spawn tougher (or even impossible-to-solve) challenges.

Here are three of those difficult areas:

1. Exponential Chaos

The combinatorial complexity of code grows exponentially. It’s well nigh impossible and also futile trying to exercise all possible execution paths of a program. Equivalence class partitioning helps a lot to cut down the test space but invariably, we still miss out on a few.

A single if statement with one condition has two paths – the condition is either true or false. Let’s assign the simple one-condition if check code above a theoretical complexity  value of 1. If that if statement is nested in another if statement, the number of paths explode to 4; ditto for two conditions in the if condition check. Re-using our complexity model, this comes to a value of 2 or so.

Most codebases have loads of conditional branching, loops, multi-threading, various components and what have you. So we can safely our complexity values for such code bases in in the high millions or thereabout. Scary? Not yet.

Now imagine what happens when there are hundreds of developers working in that same codebase and making a few hundred check-ins daily? Maybe the complexity value should sky-rocket to the high billions? Trillions?

Given the rapid rate of change and inherent complexity, how do you ensure that quality is maintained? How do you enforce consistency across large groups? A very difficult problem to solve – there are approaches to mitigate the risk but I do not know of any foolproof method that works all the time. If you know of a way, do let me know.

2. I’ll know what I want when I see it

We all think we know what we want – alas, we typically don’t until we see the finished product. Let’s take the following series of interactions between Ade who wants a new dress and his tailor.

Ade: I want some beautiful dress that fits me, is wearable all year round and casual. I want it in 2 weeks.

Tailor: Aha, so you want something casual that fits you, is wearable all year round and need it in 2 weeks.

Ade: Yup right on point

2 weeks later

Tailor: Here it is! (Beaming proudly)

Ade: (Not impressed); I like the fabric and design. But I don’t like the colour, the sleeve length and it doesn’t fit me quite right. Can you change it to black?

Tailor: here, it is in black

Ade: On second thoughts, black would be too hot, could you change it to brown?

Tailor: here it is in brown

Ade: Great! Could the sleeves be shortened by 2cm?

Tailor: Done

Ade: Hhmm, could you revert the sleeves to their original length? I think I now like the earlier design.

Tailor: Done!! (getting annoyed probably)

Ade: Great! This is becoming awesome, could you please reduce the width of the dress? It’s too wide…

Tailor: @#$@#$@$#!!!

Most people usually don’t have physical products tailor-made to their desires. We go to the store (to meet a car dealer, a tailor or an architect) and choose one of the several options there. We can take a car out for a ride, walk through a building or try on a new dress. This helps a lot as we know if we want that product or not.

In software development, it’s a different story – we want something tailored but we cannot express that need accurately until we see the product. Invariably, our descriptions do not match what we desire. To  restate: it’s highly probable that you wouldn’t like a dress you described in its entirety to a tailor when you try it on.

Figuring out what users actually want is a difficult problem – probably why agile methodologies are popular. Less difficult way? Do the minimum possible thing and allow users to play with it. For example, the tailor could have given Ade a paper dress to evaluate all the styles and all that.

Let’s play a simple game: when you start your next project, make sure you document all user requests, also record every update as you go along. I am pretty sure the new requests will significantly differ from the original one. The end product might be totally different from the initial ask even.

3. No laws – it’s the wild wild west out there

If I release my grip on an apple, I expect it to fall down – why? Gravity of course. Most interactions in the physical world are bound by these models. Imagine that a car manufacturer has to design a new super car for some super-rich guy. Mr-rich-guy asks for the following:

  • Must be drive-able by adults, teenagers and infants
  • Must work on Earth, Venus and Mars
  • Can run perfectly on gas, water or coal

The manufacturer can tell him it’s impossible since the current physical models make it extremely difficult to achieve the three impossible orthogonal requirements; maybe if he wants a movie though…

Let’s go to the world of software; consider the typical AAA game, to capture the largest possible market share, products have to be usable on:

  • Multiple consoles (XBox, PlayStation, Nintendo etc)
  • Other architectures (e.g. 32-bit and 64-bit PCs)
  • Operating systems – Windows, Linux
  • Various frame rates

There are also limitations in software (hardware limits, processors, memory etc) but we often have to build ‘cars’ that can be driven well by people in various age groups living in multiple planets!

The support matrix explodes all the time and generating optimal experiences is an extremely difficult task. In fact, most times, the workaround is to have a limited set of supported platforms.

Alas, it’s the realm of 0s and 1s, so software engineers have to conjure all sort of tricks and contortions to make things work. Maybe some day, cars would run everywhere too…

Conclusion

So apart from the technical challenges, there are a few other just-as-challenging (or even more challenging areas) in software development. These include

  • Ensuring your requirements match hidden customer desires
  • Working to meet various regulations and ensuring proper support across platforms
  • Managing technical debt and reducing risk in heavily changed code bases

Your thoughts?

A simple explanation of 1s complement arithmetic


I remember taking the digital systems course in my second year of university and being exposed to concepts like k-maps, 1s and 2s complement arithmetic. These building blocks served as the foundations for bigger concepts like adders, half-adders, comparators, gates and what not.

It was pretty much fun doing the calculations then even though I didn’t fully realize why I needed to add a 1 while doing 1s complement arithmetic. Until I stumbled upon the excellent explanation in Charles Petzold’s Code; a great book that uses very lucid metaphors for explaining computing concepts. As is usually the case – the best explanations are typically so simple and straightforward that anyone can grasp them.

Even if you already know about 1s and 2s complement arithmetic; still go ahead and read this, you might find something interesting.

Subtraction – the basics

Assuming you need to find the difference between two numbers, e.g. 173 and 41; this is pretty straightforward, you do so

minuend 174
subtrahend 041
difference 133

Aside: Minuend and subtrahend are valid names, the names of parameters for mathematical operations are given below.

Expression

First number (i.e. 5)

Second number (i.e. 3)

5 + 3

Augend

Addend

5 – 3

Subtrahend

Minuend

5 * 3

Multiplier

Multiplicand

5 / 3

Dividend

Divisor

This is the simple case, how about the carry scenario when you need to ‘borrow’ from preceding digits?

minuend 135
subtrahend 049
difference 086

Aha the pesky borrowing! What if there was a way to avoid borrowing? The first thing to think of is the ceiling of all 3-digit numbers i.e. the smallest possible number that would require no borrows for any 3-digit number. We use 3-digits since we are taking a 3-digit example; were the subtrahend to be a 5-digit number, we would need the smallest 5-digit number value too.

That smallest ‘no-borrows-required’ number is 999. Unsurprisingly, it is the maximum possible value in base ten if you have only 3 digits to use in the hundreds, tens and units positions. Note: In other bases, the values would also be the penultimate value e.g. for base 8, it’ll be 777.

Now, let’s use this maximum value as the minuend

minuend 999
subtrahend 049
difference 950

Since we are using 999 as the reference value, then 49 and 950 are complements of each other; i.e. both add up to give 999. So we can say 49 is the 9s complement of 950 just as 950 is the 9s complement of 49.

Awesome! We can now avoid the annoying carries!! But knowing this is useless in itself unless we can find some way to leverage this new-found knowledge. Are there math tricks to use?  Turns out this is very possible and straightforward.

Math-fu?

Lets do some more maths tricks (remember all those crafty calculus dy/dx tricks)…

135 – 49 = 135 – 49 + 1000 – 1000

= 135 + 1000 – 49 – 1000

= 135 + 1 + 999 – 49 – 1000

= 135 + 1 + (999 – 49) – 1000

= 136 + 950 – 1000

= 1086 – 1000

= 86

QED!

What’s the use of such a long process?

We just found a very long way to avoid carries while doing subtraction. However, there is no motivation to use this since it is quite inefficient in base 10. So what’s the reason for all this?

It turns out that in computer-land, counting is done in 0s and 1s. The folks there can’t even imagine there are numbers other than 1 and 0. As you’ll see, there are some great advantages to using the 1s complement approach in this area.

Lets take a simple example e.g. 11 – 7

minuend 1011
subtrahend 0111
difference ????

Applying the same trick again (this time the minuend will be 1111 instead of 9999).

minuend 1111
subtrahend 0111
difference 1000

Do you notice a pattern between the subtrahend (0111) and the difference (1000)? The complements seem to be ‘inverses’ of each other.

The 1s complement of any numeric binary value is just the bitwise inverse of the bits in the original value. Calculation is just a matter of flipping each bit’s value, a linear  O(n) operation that can be quite fast. That’s a BIG WIN.

Continuing the process again with the addition step this time:

Augend (difference from step above) 01000
Addend (of 1) 00001
Addend (of original 11 value) 01011
Sum 10100

Finally the subtraction of the next 1* number that is larger which will be 10000 (since we used 1111).

subtrahend 10100
minuend 10000
difference 00100

And there is the 4! answer done!

How about negative numbers? Simple, just do the same process and invert the answers.

Hope you found this fascinating and enjoyed it. Let me know your thoughts in the comments.

How to detect page visibility in web applications


You are building a web application and need the application to pause whenever the user stops interacting with the page; for example, the user opens up another browser tab or minimizes the browser itself. Example scenarios include games where you want to automatically pause the action or video/chat applications where you’d like to raise a notification.

The main advantage of such an API is to prevent resource wastage (battery life on mobile, internet bandwidth or unnecessary computing tasks). Definitely, something to have in mind especially for developers targeting mobile devices. So how would you this?

Can I use event listeners?

Technically, you could use a global event listener on the window object to listen for focus/blur events however, this can not detect browser minification. Also, the blur/focus event would be fired whenever the page loses focus; however, it is possible that a webpage is still visible despite losing focus – think about users having multiple monitors.

The good news is that this is possible with the PageVisibilityAPI which comes with the browsers and this post shows how to use this.

Deep dive into details

The Document interface has been extended with two more attributes – visibilityState and hidden.

Hidden

This is true whenever the page is not visible. What counts as being not visible includes lock screens, minimization, being in a background tab etc.

VisibilityState

This can be one of 4 possible enums explaining the visibility state of the page.

  • hidden: page is hidden, hidden is true
  • visible: page is visible, hidden is false
  • prerender: page is being pre-rendered and not visible. Support for this is optional across browsers and not enforced
  • unloaded: page is being unloaded; hidden would also be false too. Support for this is also optional across browsers

Show me some code!

document.addEventListener('visibilitychange',function(){
    if(document.hidden) {
        console.log('hidden');
    } else {
        console.log('visible');
    }
}, false);

Browser support

You can have it in nearly all modern browsers Opera mini. Also, you might need to specify vendor prefixes for some of the other browsers. See this.

Conclusion

There it is; you now know a way to effectively manage resource consumption – be it battery, internet data or computing power.

You can use this to determine how long users spend on your page, automatically pause streaming video/audio (with some nice fadeout effects for audio especially) or even raise notifications.

Did you enjoy this post? Here are a few more related posts:

How to track errors in JavaScript Web applications


Your wonderful one-of-a-kind web application just had a successful launch and your user base is rapidly growing. To keep your customers satisfied, you have to know what issues they face and address those as fast as possible.

One way to do that could be being reactive and waiting for them to call in – however, most customers won’t do this; they might just stop using your app. On the flip side, you could be proactive and log errors as soon as they occur in the browser to help roll out fixes.

But first, what error kinds exist in the browser?

Errors

There are two kinds of errors in JavaScript: runtime errors which have the window object as their target and then resource errors which have the source element as the target.

Since errors are events, you can catch them by using the addEventListener methods with the appropriate target (window or sourceElement). The WHATWG standard also provides onerror methods for both cases that you can use to grab errors.

Detecting Errors

One of JavaScript’s strengths (and also a source of much trouble too) is its flexibility. In this case, it’s possible to write wrappers around the default onerror handlers or even override them to instrument error logging automation.

Thus, these can serve as entry points for logging to external monitors or even sending messages to other application handlers.

//logger is error logger
var original = window.onerror; //if you still need a handle to this
window.onerror = function(message,source,lineNo,columnNo,errObject){
    logger.log('error', {
        message: message,
        stack: errObject && errObject.stack
    });
    original() //if you want to log the original
    return;
}

var elemOriginal = element.onerror;
element.onerror = function(event) {
    logger.log('error', {
        message: event.message,
        stack: event.error.stack
    });
    elemOriginal();
    return;
}

The Error Object

The interface for this contains the error message and optional values: fileName and lineNumber. However, the most important part of this is the stack which provides information about the stack.

Note: Stack traces vary from browser to browser as there exists no formatting standard.

Browser compatibility woes

Nope, you ain’t getting away from this one.

Not all browsers pass in the errorObject (the 5th parameter) to the window.onerror function. Arguably, this is the most important parameter since it provides the most information.

Currently the only big 5 browser that doesn’t pass in this parameter is the Edge browser – cue the only ‘edge’ case. Safari finally added support in June.

The good news though is there is a workaround! Hurray! Let’s go get our stack again.

window.addEventListener('error', function(errorEvent) {
    logger.log('error', {
        message: event.message,
        stack: event.error.stack
    });
});

And that wraps it up! You now have a way to track errors when they happen in production. Happier customers, happier you.

Note: The eventListener and window.onError approaches might need to be used in tandem to ensure enough coverage across browsers. Also the error events caught in the listener will still propagate to the onError handler so you might want to filter out duplicated events or cancel the default handlers.

Related

Tips for printing from web applications

Liked this article? Please share, subscribe or drop a comment.

Maturing as a software engineer


Looking back on my time as a developer, there are a lot of things I would have avoided doing if I had as much knowledge and maturity as I did now.

While I am grateful for the experiences and don’t regret them; I felt it would be a good idea to share these. These might motivate others or at least speed up their careers.

Here goes!

1. Patterns, patterns, patterns

When I take part in code reviews, I tend to look for recurring style patterns. Why? This helps to reduce the cognitive load on readers of the code (after all, code is written to be read).

I am  not advocating for bad software patterns rather having a plethora of ways for doing the same thing in a codebase creates confusion and productivity losses. How do you determine the ‘right’ pattern?

For example in JavaScript, there are several ways for creating an array.


var a = [];

var a = new Array();

var a = new Array(3);

Having a haphazard mixture only takes away brain processing cycles. Rather, have your team decide on a style and stick to it.

By the way, the first style is the ‘expected’ and preferred approach although there might be use cases for the latter two.

Ever wonder why the Google codebase is rated to be easy to work with? Well, think about consistency and established patterns.

2. Break the big picture down and make incremental progress

Building and distributing the smallest software piece you can imagine requires more effort than you would think. It is much more efficient to break down the big picture into small chunks of work that can be completed in an hour or less. Such breakdowns make you more effective and help in understanding progress and forecasting completion times (which is a tricky problem to solve).

I used to break down only the code pieces before (which itself was an improvement over my earlier dive-into-code-and-figure-it-out-as-you-go approach). Nowadays, I try to take some time and reflect on the end product itself: its behaviour, look and feel and how users would interact with it.

For a typical software project, such road maps covers:

  • Testing – unit tests, continuous integration,
  • Documentation – extensibility guides, tooling
  • Implementation
  • Discoverability and Distribution – release targets, getting started articles
  • Maintenance – handling bugs, user feedback etc

Sounds like too much work? Well, just focus on one small bit at a time and keep making progress.

3. Be lazy – start first on tasks with the largest impact/effort ratios

Two things matter: results and impact. There is no point in slaving for 20 hours to choose between blue and light blue if it has no impact on the users. Ditto for spending endless hours ‘arguing’ over what language should be used. Just choose the best usable one and deliver results.

My heuristic for tasks is thus:

  • Does completing the task move me closer towards the big picture?
  • Is this the easiest-to-achieve task with the biggest impact?

If so, I pick up that task and just do it – the goal is to maximize the impact/effort ratio.

Before I’d just stick to a task and spend endless hours on it even if it was something as trivial (and probably low-impact) as beautifying test scaffolding test output and elaborately designing test functions. Now? Common, my time is more valuable than that – I get the test functions right and try to get the coverage I want but won’t spend too much time once that is achieved and is readable for others.

Excessive polishing time can be spent on other more impactful pursuits like having fun with family or delivering high impact features.

4. Technical skills plateau

Sooner or later, you’ll get to the technical plateau. By that time, you’ll have so many successes under your belt and can detect potential pitfalls easily. Then, what next?

There are tons of ways to extend your impact and that is the way people become even better engineers. For example, I doubt if Anders Helsberg is still writing a lot of code, yet his ideas continue to empower and influence millions around the world.

Think about that, how do you scale your influence and make it possible to touch the lives of thousands of people? Are there engineering problems crippling your organization? Process pitfalls to improve with huge impact? Education ramp ups? There are always challenges to solve and problems to fix.

5. Choose career investments carefully

How would you set up an investment portfolio? Would you just go about investing in everything? Nope, you would evaluate the risks and benefits, consult experts and then invest in a select few areas while ignoring other areas.

You could spread out your risk by investing in a wide area but doing this excessively dilutes your returns. Conversely, investing in only company could be very risky too. Thus, it’s generally advised to spread  out your investment portfolio

Careers are investment portfolios. A typical career spans a long period ( upwards of 30 to 40 years) and shares some similarities with investments:

  • technologies, frameworks etc -> investment options
  • time -> funds

Just as you wouldn’t jump on every new fund, why would you do the same with your career? There is no harm in taking measured risks in careers but you should be strategic and know what your end goal is.

Every now and then a new framework pops up in the news. Before, I’d hop on the bandwagon and try figure it out. Nowadays? Well, if it really piques my interest, then I might spend some time learning about its core design principles and problem-solving approach.

If it neither solves any of my problems nor brings anything new to the table, then no thank you; I’d rather continue nurturing my current investment portfolio and hedging my bets.

Think about your bets and stick to them.

Conclusion

I am still learning and pray I continue. One thing that has struck me as being really critical is the will to try. We don’t know if something would work out or not however we can always try and then learn from the outcome (success or failure).

Don’t give up – continue learning and growing.

Understanding Bit masks


Bit masks enable the simultaneous storage and retrieval of multiple values using one variable. This is done by using flags with special properties (numbers that are the powers of 2). It becomes trivial to symbolize membership by checking if the bit at a position is 1 or 0.

How it works

Masking employs the bitwise OR and AND operations to encode and decode values respectively.

New composite values are created by a bitwise OR of the original composite and the predefined flag. Similarly, the bitwise AND of a composite and a particular flag validates the presence / absence of the flag.

Assuming we start off with decimal values 1,2,4 and 8. The table below shows the corresponding binary values.

Decimal Binary
0 0000
1 0001
2 0010
4 0100
8 1000

The nice thing about this table is that the four bits allow you to represent all numbers in the range 0 – 15 via combinations. For example, 5 which is binary 101, can be derived in two ways.

5     -> 1 + 4

or

101 -> 0001 | 0100

       -> 0101

7 which is 111 can also be derived in the same form.

Since any number in a range can be specified using these few ‘base’ numbers, we can use such a set to model things in the real world. For example, let’s say we want to model user permissions for an application.

Assuming the base permissions are read, write and execute, we can map these values to the base numbers to derive the table below:

Permissions Decimal Binary
None 0 000
Read 1 001
Write 2 010
Execute 4 100

Users of the application will have various permissions assigned to them (think ACL). Thus a potential model for visitor, reader, writer and admin roles with our base permissions is:

Role Permissions Decimal Binary
Visitors None 0 000
Readers Read 1 001
Writers Read + Write 3 011
Admins Read + Write + Execute 7 111

Noticed a pattern yet?

All the binary values can be obtained by ‘OR-ing’ the base binary values. For example, admins who have read, write and execute permissions have the value obtained when you do a bitwise OR of 1, 2 and 4.

The UNIX model uses the same numbering system. E.g. 777 translates into 111 111 111 which grants owners, groups and others read, write and execute permissions.

Checking Access

Now, the next question is how do you check if a composite value contains a particular flag? Going back to the binary data basics, this means checking if a bit at some position is a 1 or 0.

The bitwise AND operator comes in handy here – it guarantees a 1 when both values sharing the same index are 1 and 0 in all other cases. Thus, ‘AND-ing’ a composite value and the desired flag would reveal the desired outcome. If we get a value greater than zero as the result, then the user has the permission, otherwise a zero means there is no permission.

The admin role has a bitmask value of 111. To check if he really ‘has’ the execute permission we do a bitwise AND of 111 and the execute flag 100. The result is 100 which proves the permission.

More tables! The two tables show how to check for 2 users; one with the read + write + execute (111) permissions and another with the read and execute (101) permissions.

Read + Write + Execute (111)

Permissions 111 bitmask Binary Has permission?
Read 111 001 Yes: 111 & 001 → 1
Write 111 010 Yes: 111 & 010 → 1
Execute 111 100 Yes: 111 & 100 → 1

Read + Execute (101)

Permissions 101 bitmask Binary Has permission?
Read 101 001 Yes: 101 & 001 → 1
Write 101 010 No: 101 & 010 → 0
Execute 101 100 Yes: 101 & 100 → 1

See? Such a simple way to check without having to make unnecessary calls to the server.

Usage

This post shows the permission model however bit masks can be applied a wide variety of scenarios. Possible examples include checking membership, verifying characteristics and representing hierarchies.

A possible use in games could be to verify the various power-ups an actor has and to add new ones rapidly. Eliminates the need to iterate, use the bitmask.

Hierarchical models, where higher members encompass lower ones (e.g. the set of numbers) can also be adequately modeled using bitmaps.

Language support

Explicit language support is not needed to use bitmasks, the rules to know are:

  • Use increasing powers of 2 – ensures there is only one flag at the right spot
  • Create basic building blocks that are easy to extend and combine
  • Remember to watch out for overflows and use values that are large enough to hold all possible bit values. For example, C’s uint_8 / unsigned char can only hold 8 different flags, if you need more, you’d have to use a bigger value.

Some languages provide extra support for bit mask operations. For, example, C# provides the enum FlagsAtribute which implies that the enum would be used for bit masking operations.

Teaser

Q: Why not base 10? After all, we could use 10, 100, 1000 etc.

A: The decimal system falls short because you can have ten different numbers in one position. This makes it difficult to represent the binary ON/OFF state (which maps well to 0/1).

 

Safely handling destructive loops


Collections abound everywhere in the real world and loops are very common in programming. A simple loop is shown below:

var arr = [1,2,3,4];

for(var i = 0, len = arr.length; i < len; i++) {
    console.log(arr[i]);
}

//Output
1,2,3,4

Destructive loops

Simple loops can be boring, after all, why not have the collection change during a looping operation for more fun? Won’t happen? Believe me it does, have run into such at least twice.

An example scenario involves collections of listeners (say for example in a pub-sub implementation). Ideally, you want to go through all the listeners and invoke them. However it is possible a listener can either directly or indirectly remove itself while the loop is processing. For example, it might call a destroy function on some object which in turn leads to a unsubscribe call.

When this happens, it’s all chaos. Funny things happen or even worse – bugs.

Suddenly, the loop invariants are all wrong and you get funny results and/or errors (most likely errors). Let’s take an example:

function createListenerArray(arr) {
   for(var i = 0; i < 4; i++) {
       arr.push(function(i) {
           console.log(i);
           if(i === 2) {
               arr.splice(i, 1);
           }
       });
   }

   return arr;
}

The createListener function pushes 4 functions onto the array reference it gets. One of these functions (the third one) will remove itself from the array once invoked. To make tracking things easier, each function will log its index to the console.

var arr = [];
createListenerArray(arr);
for(var i = 0, len = arr.length; i < len; i++) {
    arr[i](i);
}

0
1
2
//TypeError: arr[i] is not a function

Oops! Blows up…

The loop was proceeding smoothly until it got to the function at index 2. Invoking that function changed the array by removing the corresponding entry from the array. Thus the valid length of the array decreased from 4 to 3.

The next step of the loop tried to access the element at index 3; however that index is now an invalid position since the element at index 3 got moved to index 2 when the array shrunk. arr[3] is undefined and invoking undefined as a function leads to the TypeError.

Imagine the ripple effect this can cause in larger arrays

Let’s try forEach.

var arr = [];
createListenerArray(arr);

arr.forEach(function(val, i){
    val(i);
});

0
1
2
undefined

This is not the desired result but on the bright side, at least there were no Exceptions. So what other approaches exist?

1. Clone

One way is to create a clone of the array to provide a validation checker before invocations as shown below.

var arr = [];
createListenerArray(arr);
var clone = arr.slice(0);

for(var i = 0, len = clone.length; i < len; i++) {
    if(arr.indexOf(clone[i]) > -1) {
        clone[i](i);
    }
}
//
0
1
2
3

This guarantees that the functions would always exist; to verify, we can re-run the snippet again to prove that the value at index 2 is now gone.

for(var i = 0, len = clone.length; i < len; i++) {
if(arr.indexOf(clone[i]) > -1) {
        clone[i](i);
    }
}

//Output
0
1
3

2. Count down

Cloning can lead to new issues with memory allocation and garbage collection. There is another approach that is cleaner, doesn’t require cloning and should not leave danglers around.

The fix relies on understanding the true cause of the looping issue. The problem occurs because the removal of a loop element affects unprocessed elements in the loop (by changing the array length).

We can circumvent this by counting down from right to left – if an element removes itself, the unprocessed elements to the left are not affected since their indices are still valid (compare to the left-to-right count-up approach).

Now, even if a function should remove itself, things are still alright, since its index would be removed and not be valid anymore.

let arr = [];
createListenerArray(arr);

for(var i = arr.length - 1; i >= 0; i--) {
    arr[i](i);
}

//Output
3
2
1
0

console.log(arr.length);
//3

Neat! Now, this is another reason to use utils like _.forEachRight.

Liked this trick? Let me know your thoughts.

Influential Books for programmers


I try to read a lot of books. Over the years, my ‘taste’ for books has been refined and some of my criteria are listed below.

Book Impact Scale

  1. Length: The 200 – 300 page range is just about right for a technical book. Longer books contain a lot of fluff and repetition. Sometimes I get the feeling that the author had to meet some minimum page count.
  2. Thought-provoking: I love technical books that challenge the way I think, enable me to see things in alternative light and exercise my ‘thinking’ muscles. Isn’t that the goal for reading after all? To broaden horizons and learn alternative viewpoints.
  3. Inclusion of challenges: I like books that provide practice exercises. This cements the new ideas – it is easy to ‘think’ you already know something when you don’t. Solving a problem is the litmus test for understanding.
  4. Enduring: Every field has its ‘classics’ and they exist for a good reason too. If a 30-year old book is still in print, has lots of good reviews and teaches fundamentals; then chances are high it is a GOOD book. Add bonus points for language-agnostic themes.
  5. Simple: Most books fall short in knowledge delivery by employing obscure terms. This might be due to a misassumption of expected readers’ skill levels. Clear straightforward explanations make for easy reading.

Now, here come the books (listed in no particular order).

1. The Pragmatic Programmer

Excellent read. Medium sized at 352 pages but bursting at the seams with sagely advice. The authors used lots of interesting stories to drive home the points. Some of the lessons that have stuck with me include are:

  • Coincidental programming – you don’t why or how the code ‘works’. If adding that statement makes it ‘work’, then it might also cause bugs you can’t explain.
  • Learning:
    • Learn a new language every year
    • Master a single editor (got me to learn vim)
    • Become a command line expert
  • The broken windows theory – quickly leads to software rot
  • Programming close to the problem domain (SICP book did this too)
  • Being proud of your work – craftsmen are supposed to be proud of their work.
  • Bringing about the change you want
  • Test your software else your users will

Definitely influenced me in a big way and probably due for a reread now.

2. Structure and Interpretation of Computer Programmers (SICP)

A master classic – one for anyone who wants to be a master programmer and has the patience to complete all the exercises. Reading the book without solving the questions would make it difficult to fully absorb the goodness.

The authors painstakingly organized the book – the pages flow so smoothly that the increase in complexity is almost imperceptible. You get exposed to a huge slew of programming paradigms (logic, functional, OOP), build cool stuff (8 queens solver, calculus solve, an interpreter and a compiler) and also learn some CS stuff (streaming, Ackermann functions, Huffman etc).

The biggest change for me is getting to understand that programming is all about solving the right problems using the right abstractions and in a logical and well-ordered manner. Languages don’t matter that much – they are just tools for shaping processes and conveying our thoughts. Java, JavaScript, Python, C, C++ are all beautiful languages. If you don’t know how break down problems, evaluate alternatives and create solution pipelines, then it doesn’t matter what tool you use – the end result may work but wouldn’t be a masterpiece.

Give a master carpenter the minimum tools he needs and he can create something beautiful. Give me a fully stocked wood workshop and while I can craft a table it probably would not come close to what the master did with his limited tools. My table might work but I probably won’t get any appreciation for it. Think this same way about programming.

For example, here is how they explained list reversal:

To reverse a list, just append the first element to the end of the reversed sublist

Be warned though, this is probably a year-long journey but one that you must should undertake. It’s freely available and you can check out my repo of solutions too.

3. Code Complete 2

I had always wanted to read this book but I couldn’t bring myself to do that because it was over 1000 pages long. Finally, I gratefully found a strategy that worked – committing to a 30 min read window daily; mostly on the bus commute home.

The points from the book:

  • Avoid complexity at all costs; software is already difficult so make it as dumb as possible. Again, make it plain simple.
  • Always care about quality regardless of your state in software development
  • Create programming models in your language i.e. try to create domain abstractions in the language; this makes it easier to maintain and express ideas
  • Programs are meant to be read by humans first and then for computers next
  • Being open to other ideas and influences – no need getting stuck to some particular tools
  • Testing styles to ensure code coverage

It is a great book – a bit verbose at times but one that you should read too.

 4. Programming Pearls and more Programming Pearls

A bit old but still readable – it made it obvious how programming had advanced and developers could avoid knowing intricate computer details.

The two books are small and contain a lot of ‘pearls’ that make you think deeply; most revolve around events that happened during the authors’ time at the Bell labs. The coding examples might be outdated but the solid principles in the book aren’t.

For example, a book teaching the basics of motion from the 60s would probably still be applicable today; motion fundamentals haven’t changed much. Same way the author’s approach of problem definition, isolation and iterative improvements can’t be faulted.

Highlights

  • Back of the envelope calculations – made me realize how these axioms could influence the way you program. Now you know why these questions are popular at interviews.
  • Over-engineering a solution might help it cope with unforeseen load
  • Iterative improvement – first solve the problem, then improve the algorithm if needed.
  • Confirm algorithmic correction – it’s very easy to assume your algorithms work when they infact contain bugs.
  • Programming for resource-constrained environments – how do you sort 4 million nodes on a limited memory computer?

5. The Little Schemer + The Reasoned Schemer

The style was nothing like any of the others and I surprisingly found it enjoyable. It flows as a series of questions and answers although readers are encouraged to try before taking a peek at the answers.

The first few chapters were easy to breeze through (coming from a SICP background probably helped too) however things get more interesting from about chapter 9 or thereabout.

  • Provides a ‘formula’ for nailing recursion: fix the base case and then handle the recursive case

Learning recursion is easy, master the first part and then you can learn the remaining parts (pun intended).

  • Good explanation of the Y-combinator – slow build up but still challenging explanation of a hard-to-grasp concept.
  • Covers CS concepts like generators, memoizations, closures, Church encoding, currying, continuation passing and more.

Worthy of mention

1. The Algorithm Design Manual (TADM)

Normally algorithms books are all full of mathematical proofs (really important to understand). However, I like the TADM because it focuses more on applying those algorithms in practical context. The various stories by the author and his humorous quips make it all the more fun too.

2. Eloquent JavaScript

The book that exposed me to functional programming and started the long JS adventure. I always recommend it since it is an excellent beginner book, free (despite its high quality) and is just awesome.

Conclusion

Looking back, it is interesting to see the subliminal influence these books have had on me. I only realized some of these while writing this review.

What are your favorite books and how have they influenced the way you write programs? Share in the comments.

Advice for aspiring programmers


I have made a couple of mistakes over the years and wanted to share those pitfalls so upcoming programmers know what to avoid and what works.

1. Focus

You can’t do everything – I don’t think it is possible for one person to simultaneously be a pro at writing high quality code, network engineering, computer hardware and machine learning.

Some years ago, a much younger and naive me wanted to do everything in software engineering, use several operating systems and know enough about network engineering. Did I succeed at that? No, I just diluted my efforts and wasted loads of time (was fun though…).

Laser-like focus is essential as the field of computing is extremely wide. Decide the field you are interested in, narrow down a learning path and get your hands dirty. For example, if you choose software engineering, you’ll have to select one of three tiers: web/mobile/desktop. Once decided, just work on completing a sample project. The more projects you have under your belt, the better you become and the more learning you do.

2. Dedication

You probably won’t become good enough in one or two years. That learn * in 24 hours book? Gives you a feel for what the language syntax is but there is more than that to mastery. You need to learn language idioms, problem analysis and clean architectural design.

This take time – a lot of time: Norvig put it at 10 years; just like mastery in any other field. It is a long haul so set your mind at it and you should hopefully start seeing results in about 5 years.

3. Consistency

Little drops of water consistently dripping on the same spot can wear down rocks.

The rush of excitement you feel at the beginning is eventually going to wane. Once it becomes a dreary old routine, what will keep you going at it? Consistency and habits can compound in surprising ways.

Assuming you are 1% better each day; after a year; you’ll be about 37 times better. Conversely, if you lose 1% everyday, then after a year you’ll be 0.02 of what you were. This is an extreme example but …

Small incremental progress over a long period of time leads to big increases. For example, a developer that dedicates 25 minutes daily to improving his skills would have spent ~183 hours honing his skills after a year. That’s nearly a month of intense study time. Can you study intensely for one month without burning out?

Can’t afford the 25 mins a day? How many minutes do you spend daily on Facebook? Multiply that by 365 to see where your time is going… Oh.. add in some more hours for games, TV and Twitter too. You see little things add up quickly over long periods of time.

Now once you are consistent, what’s the next step?

4. Seek mastery before moving on

Now that you have focus, the next obvious step is to stick to a path. Frameworks pop up every second in software development but hopping around from one framework to another is ill-advised: you would likely end up as a jack of all frameworks and master of none. I agree and encourage exploratory learning but jumping on every fad bandwagon is going to get you nowhere. Rather, make a strategic choice, stick to it and commit to learning it deeply.

Does this make you less marketable? Afterall, you want to be that guy who knows how to write code in 10,000 languages. Mastering a few increases your worth – you can now charge people more for expertise in your domain. Think of it this way, if you want a dining table masterpiece, what kind of carpenter would you prefer? One that specializes in dining tables or a generalist?

5. Learn the fundamentals

I can’t emphasize this enough. Let’s take a story: EmberJS and AngularJS brought about widespread adoption of 2-way binding across UI interfaces and this in turn triggered the proposal for object.observe as a JavaScript language method. More recently, the new path has been reactive programming and virtual DOM, eliminating the need for observables. You guessed it, object.observe got retired from the language. Now, a counter example, Dijkstra’s algorithm has been around since 1956 and it is still in use after 60 years! Can you beat that?

You have to know algorithms, data structures, operating system fundamentals, caching, Mathematics, etc. These concepts are language agnostic and provide a solid bases for solving new problems. That cool functional reactive somersaulting framework you are harping about today? It may be gone in 10 years. Fundamental CS knowledge? Evergreen, safe investment.

6. Be Humble

There are two things you should keep in mind:

  • There would always be a programmer better than you
  • You probably overestimate your skills

Do you disagree with me? Check out your ranking on the programmer skills matrix. Next, read Micheal Church’s excellent adders analogy. Unless you are already at 2.5+, there is no need to brag about your skills. Instead, quitely keep pushing the limits and some day you hopefully would earn bragging rights.

Humility is a great characteristic to have – there are a ton of things you do not know and need to know. And it takes time to reach the levels others have reached. Even if you think you are a rockstar, people generally appreciate a humble confident programmer who doesn’t rub his ‘magnificence’ into their faces all the time.

Need a yardstick to measure your progress? Periodically check back on both lists to track your progress.

7. Learn how to communicate

Most of software engineering is not about writing code – a large amount of time is spent discussing projects, evaluating designs and interacting with customers. The ability to clearly convey thoughts is one most programmers lack and it is a clear differentiator.

Being able to express yourself means you can get help with problems because you can explain them to others, it also means you can convince people to join your open source project or even adopt your architectural design. All these come from being able to sell yourself and your thoughts.

Communication is not limited to speech only though, writing well is another part. Want to improve? Write more, speak more, volunteer to be a spokesperson – practice makes perfect.

8. Get a mentor & be visible

Stand on the shoulders of giants to see farther. You need to avoid making pitiable mistakes if you can, accelerate your growth and ensure you are on the right path. Great mentors help you grow, introduce you to their networks and provide insightful feedback. They have a lot of experience which you don’t.

Don’t hide under a rock! People should know you for what you do!! Would you buy Microsoft Word if you never heard of it? Seek out peers in the community, interact with them and share ideas. Help one another grow and help influence the community. Being visible makes it easier to find mentors, jobs or even new freelance opportunities. Don’t want to go out? Then create a blog, comment on blogs, release software on Github or participate in fora.

9. Be open

No, your favorite language/IDE/tool is not the best thing since bread and butter. It has its own flaws and strengths.

You can have pretty strong opinions on anything, for example, favorite languages. However what you shouldn’t do is get so tied to some language that you refuse to try out any other language. There would always be situations where your favorite tool doesn’t cut it; in such scenarios, find the best tool and deploy it.

A Toyota Camry can probably carry logs of wood but you are better off moving such with a heavy freight truck.

I used to think I understood recursion until I read SICP; I was forced to write programs without resorting to variables and suddenly, simple toy programs became quite challenging. Exposure to various tools will empower you with various problem solving approaches. This is always a good thing since software engineering is all about solving the right problems.

10. Startups can be overglorified

I think most times, the start-up concept is overglorified – it involves a lot of work, huge risk and great rewards on success. However, the idea that once you make that awesome piece of software, it’ll automatically sell itself is mostly false. That rarely happens.

The saddening part however is that most of us do not know how to deliver good software, talk less of selling software. Yet, they want to run the business. Imagine you handing over a truck on the highway to an unqualified driver who barely knows the ropes. What’s that? A recipe for a crash right? Yup, then why do we do the same with startups?

Software is typically not the critical part of most businesses, rather meeting the user’s need is. You need to learn how to write software first, then learn about business and entrepreneurship – creating a MVP, build-measure-learn cycles, customer interactions, acquisition funnels, maintenance and support, delivery pipelines, execution and meeting competitors. Now, you know this, you can see the bigger picture – the software is not the goal. The business is the goal – the best piece of software that no one uses is a failure.

There is nothing bad with working in an established shop for some time to learn the ropes, creating connections and learning how business is run. You can even start testing the waters and at least you have a safe cushion if things go south.

And that’s it! Did I miss out on anything? Do share your thoughts in the comments or follow me on Twitter.

Code complete != ship ready


I used to work for a team where whenever an engineer said he was done, the next question would invariably be are you ‘done done‘?

Do you find yourself always under-estimating the amount of time needed to complete some software project? Does it take more time than what you planned? Or do you say you are done but need to change just one small feature.

Accurate estimation is difficult but again an unclear definition of what done means also contributes to this. So what does done mean?

Checklists and Processes

Boeing introduced preflight checklists in 1935 after a prototype crash killed two pilots. Investigations revealed that they had forgotten to carry out a critical step before takeoff. Thus, the fix was to introduce a preflight checklist to avoid this (they learnt from the mistake and became better for it).

A catastrophic software failure can have disastrous outcomes – it is expensive, can harm (imagine if embedded car software were to fail), and is bad PR for the company (how many security breaches have you heard of in recent times?).

So, do we need such heavy formal checklists for software development? Some companies require standardized processes while others don’t. However, it is always good to have a mental model of the various stages of the software shipping pipeline.

Here is a checklist to guide you through the various phases of done; you can now know which stage you are at any given point in time.

1. Code complete

You have written the code (and hopefully added unit tests) but have not done any deep testing. This is what most developers erroneously refer to as done. No, code complete does not mean done. It is the first milestone in a long journey.

Always have tests for code completion

2. Test complete

Thorough exploratory testing of the feature has been carried out. This covers extreme conditions, edge cases and load/performance scenarios.

It’s an exercise in futility to attempt to find all bugs and exercise all possible code paths. Rather, the goal of testing is to reduce the chances of a user finding a glaring simple-to-discover error that should have been found earlier. It also surfaces other issues (e.g usability, security holes etc.) that might have been latent.

Every discovered bug should ideally be fixed and a unit test added to prevent future regressions. However, not all bugs are fixable (and those turn into features…). Early knowledge of these issues enable developers to proactively prepare mitigation plans, workarounds and also notify users.

If you don’t have dedicated testers (they are worth their weight in gold and very good); you  can use James Whittaker’s testing tours method to get some fun while doing this and also ensure a good coverage.

3. Integration complete

You have completed the various isolated features, written unit tests and also done exploratory testing. You have a high confidence in each of the individual features however your components work exactly like the similitude below:

So get to it and do integration of all the disparate components and ensure that things still work as they should. More testing, more discussions with collaborating software engineers and maybe some more glue code.

Why do I keep emphasizing on testing? Have you ever had a bad experience with some software? It’s nearly always a quality issue.

4. Ship ready

This is the cross your t’s and dot your i’s moment; verify that the code solves the problem, issues discovered during testing have been fixed and components have been integrated. Code complete, test complete and integration complete? Great, time to ship.

Some organizations enforce strict limits on some parameters like performance, security, reliability and compliance. If so, you should do another pass to ensure that your software meets these criteria.

The product owner / customer likes the software and viola! Your software piece rolls out like a new car off the production line.

5. Shipped (maintenance ready)

Software is never done. When it is shipped, customers would start using it in ways you never imagined and will find issues. Some can be petty – for example, can you change move this button by 0.2 pixels? While others would be pretty serious stuff.

In maintenance mode, the software already exists but there are new bugs to fix, new feature extensions to make and behavioural modifications to make.

And when you think it’s all done, then maybe it’s time to release a new version. Start at number 1 again.

Enjoyed this post? Follow me on Twitter or subscribe for weekly posts. No spam for sure.

Related Posts

  1. The Myth of Perfect Software
  2. The Effective Programmer – 3 tips to maximize impact