Testing (Part 2)

In the previous post, we've looked at what to test. Now it's time to see how to test. There are several different aspects:

• Regression Testing - Here, we test the result of current test with its previous results. This ensures that the bugs we fixed today didn't break the things that were working yesterday.
• Test data - There are two types of test data: real world data and synthetic data. Both are important, because each of these tests different aspects and behaviour of the system.
• Exercising GUI Systems - Testing GUI required special testing tools. Some of them record the events and play whenever required, while some others are based on a script. Writing code that is decoupled from GUI helps us to write better tests, because we can test the backend without interacting with the GUI.
• Testing the Tests - How do we test the tests? The easiest way is to introduce bugs intentionally in the tests and see if they complain. 
• Testing Thoroughly - How do we know if we have tested our code thoroughtly? The answer is we don't, and we never will. But we can use coverage analysis tools to keep track of it.

- summary of Ruthless Testing, from The Pragmatic Programmer: from Journeyman to Master 

Testing (Part 1)

Generally, developers hate testing. They hate to see their code breaking.

But we don't. We test every bits and pieces in our code. Why? Because we know that this is one place where we can see Butterfly effect in action! One simple mistake can cause large (and very bad, of course) effects in the future.

Test Early, Test Often, Test Automatically

Many teams develop elaborate test plans for their project. But automated tests, which run with every build is far better and successful than test plans that sit on a shelf. The earlier we find the bug, the easier the fix. In fact, a good project may have more test code than production code.

Just writing tests is not enough. We have to make sure that we run that often. We cannot say that coding is done unless all the tests are passing. These are some major types of test we need to perform:

• Unit Testing - Tests a module. Because if the module can't work alone, it cannot work with others as well.
• Integration Testing - Tests how the subsystems play and interact with each other.
• Validation and Verification - Ensures that what is build is what the user needs. Checks the functional requirements.
• Resource Exhaustion, Errors and Recovery - Tests how the system behaves in real world conditions under varying factors like memory, CPU bandwidth etc.
• Performance Testing - Stress testing, in which we tests our system and how it behaves when there is heavy load.
• Usability Testing - Different from all the above tests. Here we test the system with real users. It helps us to know how easy it is for them to use it.

- summary of Ruthless Testing, from The Pragmatic Programmer: from Journeyman to Master

Automation

When the first Model-T car was released, the instructions for starting the car were two pages long! Creeppy!! We get irritated if the car takes more than 5 seconds to start when we turn the key. A great example to show how we realised the importance of automation over the course of time.

Why do we need automation? It ensures consistency and repeatability. Manual procedures ensure consistency up to an extend, but not always repeatability. Because the interpretation varies slightly for different people. People are not repeatable as computers are.

Many tools are available for handling repeated tasks. One such example is cron. It allows us to schedule tasks to run periodically. We use cron to automatically backup data everyday at midnight, or to generate reports at the end of every month. We use makefiles to compile projects. It runs all the tests before build and compiles the project automatically with a single command.

Another great example is build automation. We use Continuous Integration Servers to deploy code to production servers. We use API doc generators to generate documentation automatically from source code.

Automation is inevitable. Let the computer do all the repetitive, mundane tasks. We've got better things to do...


- summary of Ubiquitous Automation, from The Pragmatic Programmer: from Journeyman to Master

Pragmatic Teams (Part 2)

Orthogonality

Traditional team organization is based on the waterfall model. The team includes individuals who are assigned roles based on different job functions. This is more strict in some cases, where one set of people are not allowed to talk to another!

This is a great mistake. It is a misconception that different aspects of development such as analysis, design, coding and testing can happen in isolation. The decisions taken in such cases may not be accurate.

Organize Around Functionality, Not Job Functions

Dividing based on functionality favours in many ways. When there customer wants to change the database, only one team gets affected. The sense of ownership increases when the team is aware of the fact that only they are responsible for that functional aspect.

Automation

A great way to achieve consistency and accuracy. Why do all the work manually when your editor can do a lot of things for you? Automation is an essential part of every process. We'll discuss about that in detail in the coming section


- summary of Pragmatic Teams, from The Pragmatic Programmer: from Journeyman to Master

Pragmatic Teams (Part 1)

So far, we have seen so many techniques that can help you as a better individual. But the value and outcome of these techniques are multiplied manyfold if you are working in a pragmatic team. Here are some of those techniques we already discussed in terms of teams:

No Broken Windows

Remember the broken window which you were too lazy to fix? It can happen to teams also. It is difficult for a pragmatic programmer if she joins a team which doesn't care about the quality. Some teams apply a quality manager, to take care of the issues. This is absolutely ridiculous! Pragmatic teams know that quality is the result of contributions from each and every member in the team.

Boiled Frogs

You might as well remember the frog which doesn't notice the gradual change in the water and ends up cooked. Being concious and mindful about all the varying aspects is not always easy, even for a team. Sometimes occurs from the assumption that someone else is already looking into the issue.

Fight this. Make sure that everyone monitors the change. If not, appoint a chief water tester, who constantly checks for variations. It is not necessary that you have to reject the change. At least, you should be aware of it.

Communicate

Good teams communicate well! Customers love to talk to such teams because their meetings are well organized. Discussions are live and the team speaks with one voice.

Don't Repeat Yourself

Chances of duplication is more in a team. A good project librarian can help in such situations. If a team member is looking for something, she must know that she has to talk to this person first. What if the project is too large for one person to handle? Divide it based on functional aspects so that one person can handle one particular aspect of the entire project.


- summary of Pragmatic Teams, from The Pragmatic Programmer: from Journeyman to Master.

Elixir Language (Part 4)

Binaries

In Elixir, strings are UTF-8 encoded binaries.

What is a binary?

A binary is just a sequence of bytes.  By default, each character in a string is internally represented in memory using 8 bits.

Let's take a simple example. The string "hello" in UTF-8 encoded binary form is 104, 101, 108, 108, 111. As you can see, each character uses a number (code point) between 0 and 255, which is represented with 8 bits or 1 byte.

You can define a binary using <<>> as shown below.

iex> <<0, 1, 2, 3>>
<<0, 1, 2, 3>>

We have the string concatenation operator <> in Elixir, which is actually a binary concatenation operator.

iex> "hello" <> "world"
"helloworld"
iex> <<0, 1, 2, 3>> <> <<4>>
<<0, 1, 2, 3, 4>>

Let's see "hello" in Elixir's binary form:

iex> "hello" <> <<0>>
<<104, 101, 108, 108, 111, 0>>

Here, I just used a concatenation operator to append a binary to a string. It converted the string to its internal binary representation.

The unicode standard assigns code points to many of the characters we know. For example, a has code point of 97. Here, as you can see, h has a code point of 104.

All commonly occuring characters have code points between 0 and 255. But there are characters whose code points are above 255, which cannot be represented in memory using a single byte. In the next post, I'll explain how they are represented and stored in memory.

Read more: Binaries, strings and char lists

Don't be a Slave to Formal Methods

There are so many well-described software development methodologies available: from structured development, CASE tools and waterfall to UML and object orientation.

Remember this. never get addicted to any of them!

It doesn't mean that formal methods are bad. But considering one without putting in your effort to analyse based on the context and developmental practices is a waste.

Don't Be A Slave To Formal Methods

Formal methods have some serious shortcomings:

• Most of the formal methods use diagrams to capture requirements from the user. No matter how simple it is, most of these look like an 'alien image' to the user. Finally, what the user understands is the designer's interpretation of it. Use prototypes whenever possible and let the user play with it.
• Formal methods seem to encourage specialisation. One group of people work on one diagram, another group on another.. and it goes on. This leads to poor communication. We prefer to be together and love to understand the whole system we are working on.
• We like to write adaptable, dynamic systems that allows us to change the character of applications at run-time. Most of the existing methodologies are incapable of doing that.

Should we completely avoid formal methods? No, but analyse before you use. Look at the methodologies critically. Never underestimate the cost of adopting new tools and methods. And also, never think about the cost of the tool when you look at its output. Extract the best out of all.

- summary of Circles and Arrows, from The Pragmatic Programmer: from Journeyman to Master

Specification Trap

Program specification is an important part of software development. It acts as the bridge between requirements and program development. It is also an agreement with the user - a contract which ensures that the final system developed will be in line with the requirement.

The problem is, some designers find it difficult to stop. They will not be happy until they pin point every single detail of the problem. This is a big risk. Why? These are the reasons:

• No matter how well you try, different people will have different perspective of the requirements. The user may not get 100% of what he/she really asked. Changes at the last stage will always be there.
• Pen is mightier than a sword. Languages are powerful. But there are cases where a natural language cannot explain what we need. Don't you agree? Then try writing a set of instructions for this problem: How to tie bows in your shoelaces? You may stop before finishing. Even if you complete it, other people reading the instructions may not get what you meant. Here's a different saying: Sometimes, it is easier done than said!
• Programmer, who is implementing the requirement might have a better idea to do it. Writing down every detail is like restricting their ideas and creativity.

These reasons don't imply that specifications are bad. There are cases where clear and detailed specifications are required, depending on the working environment and the product you develop.

Rely not only on requirements. Always go for prototypes or tracer bullets. Because, sometimes, it is easier done than said!

- summary of Specification Trap, from The Pragmatic Programmer: from Journeyman to Master

Elixir Language (Part 3)

The Match Operator

In Elixir, we use = operator to assign value to a variable:

iex> x = 1
1

What's the big deal? Well, in Elixir, = is not just an assignment operator. It's a match operator.

iex> x = 1
1
iex> 1 = x
1

As long as the left hand side and the right hand side are equal, Elixir will not complain. But when a mismatch occurs, it raises a MatchError:

iex> 1 = x
1
iex> 2 = x
** (MatchError) no match of right hand side value: 1

For assigning value to a new variable, the variable should always be on the left hand side of the operator.

The match operator can be used to destructure complex data structures into its components. Here is an example where Elixir uses pattern matching on a list:

iex> [a, b, c] = [1, 2, 3]
[1, 2, 3]
iex> a
1

Here is a common question that pops up in mind when we talk about the match operator. If we are using the expression x = 1 in Elixir. How does elixir know whether we want to do a match or assignment? 

By default, Elixir always performs assignment if the variable is on the left hand side. If you specifically want to perform match, you can use the pin operator (^).

iex> x = 1
1
iex> ^x = 2
** (MatchError) no match of right hand side value: 2

Here I explained the parts which I liked the most in Elixir. This is only meant for an introduction into the language. All the examples are taken from the Elixir's getting started guide. 

Read more and learn here: Elixir: Pattern Matching

Elixir Language (Part 2)

Here are the few things I liked about the language at the first glance:

Lists and Tuples

Square brackets are used in Elixir to specify a list of values. They can hold values of multiple data types. 

iex> [1, 2, true, :atom, 3]
[1, 2, true, :atom, 3]

They are represented in memory using linked lists. This means accessing the length of a list is a linear operation.

Tuples, unlike lists, store elements contigiuously in memory. They can also hold values of any type.

iex> {:ok, "hello"}
{:ok, "hello"}

Short-Circuit Operators: and, or

By default, and and or operators are short-circuit operators in Elixir. That means, the right hand side of these operators will be executed only if the left hand side is not enough to determine the result.

iex> false and error("This error will never be raised")
false

iex> true and error("This error will never be raised")
** (RuntimeError) undefined function: error/1

Here, the first expression returns false because 'false and anything' is always false. But in the second part, the left hand side is not enough to get the result. Hence, it tries to execute the right hand side, and it fails because there is no function called error/1.

In the next part, I'll share what I learned and liked about the match operator in Elixir.

Read more:

• (Linked) Lists
• Tuples
• Basic Operators
• Short-Circuit Evaluation (wikipedia)

Elixir Language (Part 1)

After listening to the Introduction into Elixir language, by Dave Thomas, in Bangalore Ruby Users Group meetup, I always wanted to learn the language, or at least functional programming part of it. I did a failed attempt the same day evening! Yesterday, I started again with the tutorials. I am sharing some of my learnings here.

Elixir Language

Elixir is a simple, easy to learn functional programming language build on top of Erlang VM. It is a dynamic language which allows metaprogramming and can be used for building concurrent, distributed and fault-tolerant applications with hot code upgrades.

Installing Elixir in Ubuntu/Debian

The only prerequisite for installing Elixir is Erlang, version 17 or later. Precompiled packages are available in Erlang Solutions website. Follow the instructions on their website to install Erlang. Then, install Elixir by cloning their repository and generating binaries:

$ git clone https://github.com/elixir-lang/elixir.git
$ cd elixir
$ make clean test

Make sure to add the bin directory to the PATH environment variable ( Add it to ~/.bashrc file for bash, ~/.zshrc file for ZSH ... )

$ export PATH="$PATH:/path/to/elixir/bin"

This is a distilled version of installation instructions for Debian/Ubuntu users (like me!). Refer the original version for detailed instructions.

In the next part, I'll share the basic constructs I liked in Elixir language on the first day of learning.

References: 

• Elixir: Getting Started Guide
• Elixir Primer and Resources
• Elixir (Programming Language)

Wait for the Right Moment

Imagine the picture of a lion hiding behind bushes to catch his prey. He is waiting.. waiting for the right moment to jump on to the prey for a perfect catch.

Well, that was a horrible example! It's just to convey this idea: There is always a right time to begin. Learn when to start and when to wait.

We are developers. Listen to the inner voice that whispers to us ‘wait’, before jumping on to the keyboard and start writing the code.You might have a nagging doubt in our mind. Listen to it before proceeding. This may not always be a big deal. Sometimes, it could be just a disturbing thought. Give it some time. It will crystallize into something more solid. Or it will solve the great mystery that prevents you from continuing.

There is always a starting trouble while beginning a new project. But, how do we know if reluctance to start is a good decision, or we are simply procrastinating?

There’s a technique answer the question and solve the problem. If you are not sure when to start or how to start, simply take a part of the problem which you think is difficult to solve. Start developing a prototype for it, which doubles as a proof of concept. This will result in either of the following:

• You’ll feel that you are just wasting our time. No need to think twice. Throw away the prototype and start working on the actual code.
• You’ll find that there’s something wrong with the basic premise. You'll also get to know the right way to start with. Your instincts were right. Now you can launch the project in the right direction.

Prototype always helps. This is much more acceptable than simply announcing, ‘I don’t feel like starting’.


- summary of Not Until You’re Ready, from The Pragmatic Programmer: from Journeyman to Master

Solving Impossible Puzzles

Many times, we have stumbled upon a problem which seems impossible to solve. They look like a hard nut to crack, but are they really? Sometimes, the solution to the problem lies elsewhere, in a different path. We make ourselves limited by imaginary constraints. Separate them out. Hand pick the ‘real’ ones from the pool of all limitations and constraints.

We use the term ‘think outside the box’ to recognize and eliminate constraints which are not applicable in our problem. But, if the box is the boundary of our constraints and conditions, the real challenge is to find the box! The box could be larger than what we think it is.

Don’t think outside the Box - Find the Box

Next time, while encountering a difficult problem, iterate through all possible solutions. Some solutions may seem stupid. Think twice before ruling them out. Remember the Trojan Horse story? How do you get the troops into a walled city without being discovered? It’s sure that they discarded the ‘through front door’ option as it was a crazy idea to die fast.

So, every time you get a solution, think once again. There must be an easier way.


- summary of Solving Impossible Puzzles, from The Pragmatic Programmer: from Journeyman to Master

Requirements (Part 2)

Documenting Requirements

Now, we have idea about the requirements of our user. As professionals, we would like to write them down so that we don’t miss anything from our valuable user. We have to publish a document that can be used as a basis for discussions between developers, the end users and the project sponsors. There are different ways for this.

Swedish computer scientist, Ivar Jacobson proposed the concept of use cases to capture the requirements. Use cases help us to describe a particular use of the system in an abstract fashion. But his book was a little vague, which created different opinions among different people.

One way of looking at use cases is to emphasize on their goal driven nature. Templates can be used as a starting place. Use of a formal template ensures that we include all the information needed in a use case.

Over Specifying

Never be too specific, at least in the case of requirements! Good requirement documents remain abstract. They consist of simplest statements which reflect the business need clearly. But also make sure that the requirements are not vague.

Requirements are not architecture, Requirements are not design, nor are they the user interface. Requirements are need.

- summary of Digging for Requirements, from The Pragmatic Programmer: from Journeyman to Master

Requirements (Part 1)

We use the term, 'Requirement Gathering' everyday. This usually implies a group of analysts collecting the requirements from the user. 'Gathering' forces us to think that requirements are already available, we just need to find them.

But in practice, it is not exactly true. Requirements are rarely seen on the surface. They are buried deep beneath the layers, hidden from the eyes of the analysts or even the user. Perhaps this is not the right one, but it's better to remember Steve Job saying:

"People don't know what they want until you show it to them"

Digging for Requirements

The hardest part is to recognize a true requirement while digging through the dirt. For that, we need to know what a requirement is. Let's start with a basic one:

"An employee record may be viewed only by a nominated group of people"

This is an example of a good requirement. Why? Because this implies something that the system requires in a general way. Let's dig in further. Consider the following:

"Only an employee's supervisors and the personnel department may view that employee's records"

Is this statement truly a requirement? Perhaps today. But what if the policy changes tomorrow and somebody else can also view an employee's record? Then we'll have to re-write the requirement. But first example applies even after the policy change. It is always important to know why the users are doing certain things rather than knowing how they are currently doing it.

What's the best way to get inside our user's requirements? The answer is simple: become a user. Sit with the user while she's doing her job. Think and understand from the perspective of the user. We'll see how the requirement collection changes from a mere boring task to something remarkable.

Work with a User to Think Like a User


- summary of Digging for Requirements, from The Pragmatic Programmer: from Journeyman to Master

Evil Wizards

Wizards are always great! From the time when we started using computers, they take care of many things like software installations and removals (‘The wizard will now install your software!!’).

Wizards also help us in coding, especially while working with IDEs. They write us initial code for setting up a project, or depending on the situation at hand. We can use wizards to create server components, implement Java beans and handle network interfaces - all areas where it’s better to have expert help.

But, never let the wizard to be evil! The code wizard generates may not be required for our program, or may be wrong depending on the circumstances. Be aware of what the wizard writes for us. If we don’t understand the code written, then the program control is not in our hand.

Don’t use wizard code you don’t understand.

This code will also eventually become a part of our application. Always understand what’s there in our program at any point of time.


- summary of Evil Wizards, from  The Pragmatic Programmer: from Journeyman to Master

Unit Testing (Part 3)

Writing Unit Tests

Unit tests for a module shouldn’t be located far away from the module. In small projects unit tests can be included in the module itself. But for larger projects, it’s better to move them into a separate directory. But why do we say that it should be closer? Because, anything that’s not easy to find will not be used!

There are two advantages of doing this:

• It gives some examples on how to use all the functionalities of the module.
• A mean to build regression tests for validating the future changes in the code.

But providing unit tests is not only enough. We have to run them often, every time when we make changes to our code. It helps us to confirm that the class passes its tests once in a while.

Test Harness

When we write a lot of test code, it’s always better to have a standard test harness to create a testing environment. This helps to make our life easier.

A test harness can be implemented in the same language used for coding, or using makefiles and scripts. It can handle common operations like logging status, analyzing output for expected results and selecting and running the tests. It can also be GUI driven.

No matter however it is implemented, a test harness should have the following capabilities:

• A standard way to specify setup and cleanup.
• A method to select individual tests or all the tests.
• A method to analyze output for expected or unexpected results.
• A standardized form of failure reporting.

Tests should be always composable. That means, a test can be composed of subtests of subcomponents to any depth. In this way, we can select the complete test, or individual parts of the test depending on the requirement at that time.


- summary of Writing Unit Tests & Test Harness, from The Pragmatic Programmer: from Journeyman to Master

Unit Testing (Part 2)

Testing Against Contract

We like to write test cases for a module to ensure that the modules honors its contract. This will tell us

• whether the code meets the contract.
• whether the contract means what we think it means.

We test the module to check whether it delivers the functionality it promises, over a wide range of test cases and boundary conditions.

How do we define contract? For example, let’s take the factorial of a number problem. It’s contract can be defined as follows:

• Pass in a negative argument and ensure that it is rejected.
• Pass in an argument zero and ensure that it is accepted (boundary condition).
• Pass in an argument between zero and upper limit and ensure that the correct factorial is returned.

We have to test all these scenarios to ensure that our factorial module is working.

Why do we go to all this trouble? We are investing time and effort to write test cases. All we want is to avoid ‘time bombs’ in our code that sits unnoticed and blows up at an unexpected moment later in the project.

Suppose our module uses a linked list and a sort algorithm, we have to test the linked list, sort and the module completely. If the linked list and sort pass and the module test cases fails, we’ll know that the problem is with the module. This helps to reduce the debugging time also.

While designing a module, always design both its contract and the code to test that contract.


- summary of Testing Against Contract, from The Pragmatic Programmer: from Journeyman to Master

Unit Testing (Part 1)

While writing code, we have to build testability into our software from the beginning itself. This helps us to test each piece of our code before integrating them together.

Unit testing helps us to do this. This is done on each module of our application. Unit testing is performed isolating each module from one another so that the functionality of each module can be tested separately.

Unit testing establishes an artificial environment and invokes the module under testing. It checks the result returned by the module and compares it against a list of known values or against the results from previous runs of the same test. Unit test ensures that in a complete system, individual parts work as expected.

- summary of Unit Testing, from The Pragmatic Programmer: from Journeyman to Master

Refactoring (Part 2)

How do we Refactor?

Refactoring can be triggered by many factors such as deeper understandings, changes in requirements and so on. But never try to break into large quantities of code and refactor. We’ll end up in a much worse position than we are.

Refactoring has to be done slowly, deliberately and carefully. Below are some guidelines for refactoring given by Martin Fowler:

• Don’t try to add functionality and refactor at the same time.
• Have good tests before refactoring. It’ll help us to detect if anything breaks.
• Take short steps.

While refactoring, fix the code and everything depends on the code. This will a pain. But it’s going to hurt more later.


- summary of Refactoring, from The Pragmatic Programmer: from Journeyman to Master

Refactoring (Part 1)

As the program develops, we will have to rethink earlier decisions and rework certain portions of the code. Code is not static. It needs to evolve.

Rewriting, reworking, and re-architecting code is collectively known as refactoring.

Refactoring.. When should we do that?

Following are some scenarios where we need to refactor our code:

• Duplication - when we detect a violation of DRY principle.
• Non-orthogonal design - when we discover that the design can be much more orthogonal.
• Outdated knowledge - when the knowledge about the problem domain increases.
• Performance - when we need a much better performance than existing.

Refactoring is not always easy. We have to go through the existing code and modify it without affecting the functionality. Many developers are reluctant to do this because their code is mostly fragile.

Time is another reason for not refactoring. But the reality is that, if we fail to refactor it now, we might have to spend much more time later for fixing bigger problems.

Refactor Early, Refactor Often

- summary of Refactoring, from The Pragmatic Programmer: from Journeyman to Master

Algorithm Speed (Part 3)

Algorithm Speed in Practice

We may not be dealing with sorting or searching algorithms more often in the real life. But there are situations where we need to think of estimation. While encountering a single loop, it’s easy to identify now that we are dealing with a O(n) algorithm. It is O(n×m) if it has a nested loop.

Estimate the Order of Your Algorithms

If we have an algorithm of order O(n2), we can try to bring it down to O(nlog(n)). If we don’t know how long it takes, the easiest way is to test with different set of inputs and plot a graph. With around 3 - 4 points in the graph, we’ll be able to estimate the order of the algorithm.

But this is not always the case. A simple O(n2) algorithm works better than O(nlogn) for smaller values of n. At the end of the day, what really matters is how long our code takes to execute with real data in the production environment. So, always

Test Your Estimates

In some other cases, the fastest is not always the best to do the job. We have to make sure that the algorithm is apt for our problem, before going any further.


- summary of Algorithm Speed, from The Pragmatic Programmer: from Journeyman to Master

Algorithm Speed (Part 2)

The Big O Notation

The O() notation is a mathematical way of dealing with approximations. We say that the worst case time complexity of an algorithm is O(n2). It means, for n records, the time taken for the algorithm to run is in the order of square of n. We consider only higher orders while estimating time complexity. For example:

O(n2 + 2n) = O(n2)

The higher order dominates other values while n varies. Since we are eliminating lower order terms, one O(n2) algorithm can be much faster than another O(n2) algorithm.

- summary of Algorithm Speed, from The Pragmatic Programmer: from Journeyman to Master

Algorithm Speed (Part 1)

We estimate the time taken to complete the project, or the time taken to complete a particular task. There is another kind of estimation: estimating the resources used by an algorithm. This includes time taken to complete the algorithm, processor and memory consumption etc.

This kind of estimation is always important. Resource estimation is used to know how long the program takes to run with a particular set inputs. This also helps us to understand how the program scales for large number of records, thereby letting us know which all parts of the code need optimization.

How do we estimate algorithms? 

That is where we get the help of big O notations.

Estimating Algorithms… What does that mean?

Algorithms work with variable inputs: sorting takes an n element array, matrix operations require an n×m matrix etc. The size of the input affects the running time and amount of memory it takes.

But why do we need to estimate algorithm speed? Because the rate at which the execution speed increases is not always linear. An algorithm which takes one minute to process 10 records may take a lifetime to process 1000!

Big O notations allow us to perform a more detailed analysis.

- summary of Algorithm Speed, from The Pragmatic Programmer: from Journeyman to Master

Programming by Coincidence (Part 2)

There is only one way to avoid all these accidents: Always program deliberately.

How to Program Deliberately

• Always be aware of what you are doing. Never let things go out of your hand.
• Don’t code blindfolded. Chances of coincidence are high when we try to build an application without fully understanding it.
• Always proceed from a plan. It doesn’t matter whether the plan is in paper or in your mind.
• Rely only on reliable things. Do not depend on accidents or assumptions.
• Document your assumptions. This helps to recollect and validate the assumptions at a later point of time.
• Don't just test your code, but test your assumptions as well. Write assertions to test your assumptions. Never guess anything; actually try it.
• Prioritize your effort. Spend time on important aspects.
• Don’t be a slave to history. Always be ready to refactor. Never let what you have already done constrain what you do next.

- summary of Programming by Coincidence, from The Pragmatic Programmer: from Journeyman to Master

Programming by Coincidence (Part 1)

Programming is like working in a minefield! An explosion can happen at any moment. Never take chances. Always be careful.

Programming by coincidence is relying on luck and accidental successes.

But, how do we program by coincidence?

Image that we are working on a programming project. We add some code, run it. It seems to be working. Then we add more code. Still it’s working. Suddenly, after several days/weeks, the program stops working. We spend hours to see what went wrong. But we couldn’t figure anything.

Why?  Because, we didn't know why it worked in the first place!

Sometimes, we rely on coincidences. Here are some examples:

Accidents of Implementation

This happens because of the way the code is currently being written. Suppose we call a routing with some data. The routine responds in a particular way. But the author didn’t indent for the routine to work that way. It can happen either due to code, or due to the routine itself. When the routine is fixed, our code might break.

Accidents of Context

We sometimes rely on the context in which we are currently working. We tend to think in such a way that this is for that particular context only. Accidents of context happens while relying on a thing that isn't guaranteed

Implicit Assumptions

We assume things. Assumptions are always poorly documented. It may vary from developer to developer.

How can we avoid these problems?

Simple...

Don’t program by coincidence


- summary of Programming by Coincidence, from The Pragmatic Programmer: from Journeyman to Master

It’s Just A View (Part 2)

Pushing all the events through a single routine is a violation of object encapsulation. It also increases coupling. That one routine has now knowledge about the interactions among many other objects. Because of the same, we are also violating DRY principle, as well as orthogonality.

Objects should be able to register to receive only the events they need, and should never be sent events they don't need.

Model View Controller

Suppose we have a spreadsheet application with some data. In addition to the sheet, we need to show the data as a graph/pie chart, and also a table with total data. But here we don’t want separate copies of data for each representation.

How do we tackle this problem?

We can create a model with the available data and common operations to manipulate it. Then we can create separate views, which use this model to display the data in various formats such as sheets, pie/bar charts  or tables. Each view also has a controller which takes care of manipulating the model data into the particular format.

Our basic objective is this:

Separate Views from Models

As the coupling between model and view reduces, the flexibility and reversibility of the code increases to a very large extend.


- summary of It’s Just A View, from The Pragmatic Programmer: from Journeyman to Master

It’s Just A View (Part 1)

We always use the ‘divide and conquer’ approach to separate big chunks of program into different modules. A module is supposed to have one, and only one, well-defined responsibility.

But here arises a new problem: At runtime, how will these modules communicate with each other? How do we manage the logical dependencies between them? How do we synchronize changes in states or data values between these modules? These things has to be done in a clean and flexible manner. We need make these happen, but we don’t want modules know too much about each other.

Here we introduce the concept of events. An event is a message which says, “something just happened”. We can use events to signal changes in one module that some other module may be interested in. Events helps to minimize the coupling between modules. The sender doesn’t need to know anything about the receiver. There can be multiple receivers also.

It’s a bad idea to have one routine which receives all the events destined for a particular application.


- summary of It's Just a View, from The Pragmatic Programmer: from Journeyman to Master

Temporal Coupling (Part 4)

Deployment

Once we design our application with the element of concurrency, it becomes easier to think and adapt to many concurrent services. The system becomes more flexible.

Architecting the system with independent services helps us to make our configurations dynamic. By planning for decoupling and concurrency, we can also have independent options where we can choose not to be concurrent.

But trying to add concurrency to a non-concurrent system is much more harder. 

If we design to allow for concurrency, we can more easily meet scalability or performance requirements when the time comes—and if the time never comes, we still have the benefit of a cleaner design.

- summary of Deployment, from The Pragmatic Programmer: From Journeyman to Master

Temporal Coupling (Part 3)

Design for Concurrency

It is always easier to make assumptions with linear programming, which might lead to sloppy programming. But concurrency forces us to think more carefully. Because, there are things that can happen at the same time. We’ll see a lot of time based dependencies.

The Java platform exposed more programmers to Multithreaded programming. But threads impose new design constraints, which help us to decouple our code and avoid programming by coincidence.

Consider a windowing system where widgets are created and displayed in two separate steps. In the first step, widgets are created and in the second step, we display them on the screen. This means, no other objects can access the widgets until we show it on the screen.

But in a concurrent system, this may not be true. An object must be in a valid state whenever it is called. They can be called at the most awkward times. We have to ensure that the object is valid at any time it could possibly be called.

Always Design for Concurrency

- summary of Design for Concurrency, from The Pragmatic Programmer: From Journeyman to Master

Temporal Coupling (Part 2)

Workflow

In all projects, we would like to know what all things can happen at the same time, and what must happen in a strict order. This is why we need to know users’ workflow. One way for recording this is to use UML Activity Diagram.

Activity Diagram consists of rounded boxes, representing actions. An arrow from one action leads to another action, which can start after the first action. It also contains a thick line called Synchronization Bar. Once all the actions leading to a synchronization bar are complete, we can then proceed along any arrows leaving the bar. Also, an action can be started at any time if no arrows are leading into it. Activity diagrams can be used for achieving parallelism.

Analyze Workflow to Improve Concurrency


- summary of Workflow, from The Pragmatic Programmer: From Journeyman to Master

Temporal Coupling (Part 1)

Temporal coupling, as the name indicates, is related to time. In software development, we think about time schedules or deadlines. But time should always be a design element of the software. Usually we don’t approach programming with this aspect in mind. Normally, during development or architectural stage, things are always linear. We have plans like - do this after doing that, or method A should be called before calling method B.

This approach is not very flexible, and not very realistic

We should allow concurrency in our programs. Always think about decoupling and ordering dependencies. This helps us to gain flexibility and reduce any time-based dependencies in many areas of development.

- summary of Temporal Coupling, from The Pragmatic Programmer: From Journeyman to Master

Metaprogramming (Part 2)

Metadata-Driven Applications

Metadata can be used for configuring and driving the applications. Our goal is to create highly dynamic and adaptable programs. For this, we adopt a general rule: program for the general case, and put the specifics somewhere else—outside the compiled code base.

Put Abstractions in Code Details in Metadata

The benefits of this approach are:

1. Helps us to decouple our program, resulting in a flexible and adaptable program.
2. Helps us to create a robust, abstract design by deferring details out of the program.
3. We can customize our applications without recompiling it.
4. Metadata can be expressed in a manner that's much closer to the problem domain than a general-purpose programming language might be.
5. We can implement several different projects using the same application engine, but with different metadata.

The next important question is, when to configure?  Most of the applications read the configuration only during start up. In this case, most of the times, we have to restart the application for applying the configuration.  A more flexible approach is to write programs that can reload their configuration while they're running.


- summary of Metaprogramming, from The Pragmatic Programmer: From Journeyman to Master

Metaprogramming (Part 1)

We want to make our systems highly configurable. It includes not only colors or fonts, but also the choice of algorithms, database products, middle-ware technology and user interface style. These items should be integrated as configuration options, not through integration or engineering.

Configure, Don't Integrate

We can use metadata to describe configuration options. Metadata is data about data. Some examples are data dictionary and database schema. A schema contains data that describes fields (columns) in terms of names, storage lengths, and other attributes. Web browsers and many other applications use metadata for storing configuration options in files.

- summary of Metaprogramming, from The Pragmatic Programmer: From Journeyman to Master

Decoupling and the Law of Demeter (Part 2)

The Law of Demeter for Functions

The law of demeter for functions helps to reduce coupling between modules. The law states that any method of an object should call only methods belonging to:

• itself
• any parameters that were passed into the method
• any objects it created
• any directly held component objects

Writing shy code which obeys this law helps us to achieve our objective:

Minimize Coupling Between Modules

Using the Law of Demeter will make our code more adaptable and robust.


- summary of Decoupling and the Law of Demeter, from The Pragmatic Programmer: From Journeyman to Master

Decoupling and the Law of Demeter (Part 1)

Spies and revolutionaries are often organized into small groups called cells. They might know the people in the same cell, but not from other cells. This ensures that when one cell is discovered, details about other cells will remain protected. This principle can be applied to coding as well. We can organize our code into modules and limit the interactions between them. If one module got replaced or compromised, other modules should continue to work.

Minimize Coupling

Modules knowing each other is not as paranoid as spies or revolutionaries! But we should be careful about how many modules we interact with and how we came to interact with them. 

Imagine that we are building a house. We hire a General Contractor to get the work done. The contractor may or may not do the construction personally. He can hire a number of subcontractors, split the work and assign to them. As clients, we don’t need to interact with these subcontractors. All the headaches related to these subcontractors are isolated from the clients.

Similarly, while writing modules, instead of having one module which depends on many other classes, we can have a hierarchy of classes. In this case, each class need to be concerned about only one other class. This makes it easier when a change happens to some class. We need to modify only the class which uses this changed class and not the entire module. This also helps reduce the dependencies among classes.

Systems with many unnecessary dependencies are very hard to maintain, and they are highly unstable.

- summary of Decoupling and the Law of Demeter, from The Pragmatic Programmer: From Journeyman to Master

When You Can't Balance Resources

There are cases when the basic resource allocation pattern is not applicable. This usually happens in programs which use dynamic data structures. When there are top level structures and inner level structures, some resources are allocated to the top level and some are allocated to inner levels. If there are multiple levels of structures, the resource allocation is done for each level separately.

What happens when we deallocate the top level structure? We have three main options:

1. The top level structure is also responsible for deallocating the substructures. There substructures again recursively deallocates its substructures.
2. The top level structure is simply deallocated, thereby making all its substructures orphans
3. The top level structure refuses to deallocate if it contains any substructures.

The choice depends on each individual data structures and their circumstances. In languages like C, where data structures themselves are not active, we can write a module for each major structure that take care of standard allocation and deallocation.

Checking the Balance

It is always a good idea to build code that checks if the resources are freed properly. For most of the applications this means producing wrappers for each resources that keep track of allocations and deallocations.

At a lower, but no less useful level, you can invest in tools that check your running programs for memory leaks.


- summary of When You Can't Balance Resources, from The Pragmatic Programmer: from Journeyman to Master

How to Balance Resources?

Most of the time, the resource usage in a system follows a specific pattern: we allocate resources, use them and then deallocate it. But many developers have no consistent plan for dealing with resource allocation. So, here is a simple tip:

Finish What You Start…

This means that the routine or object which allocate the resources is also responsible for deallocating it.

Nested Allocation

When the routines require more than one resource, consider the following two suggestions:

1. Deallocate resources in the opposite order in which we allocate them. If one resource contains reference to another, this will ensure that no orphaned resources are there in the system.
2. When we allocate the same set of resources at different places, always allocate them in the same order. This will reduce the possibility of deadlocks happening in the system.

Objects and Exceptions

Classes can be used for allocating and deallocating resources. We can encapsulate resources in a class. When we need a particular resource type, instantiate an object of that class. When an object is created, the constructor allocates the resources. When the object goes out of scope, or is reclaimed by the garbage collector, the object's destructor then deallocates the wrapped resource.

This approach is particularly useful while working with languages like C++.



- summary of How to Balance Resources, from The Pragmatic Programmer: from Journeyman to Master

When to use Exceptions?

Checking for errors at every part may lead to developing ugly code. That is where exceptions come into handy. But the problem with exception is, we need to have a clear idea of when to use them. Exceptions should be reserved only for unexpected events.

For example, consider this scenario. If our code is trying to open a file for reading and that file does not exist, should we raise an exception? It depends. If the file should have been there, but we cannot find them, then it is exceptional. But imagine that we have only the filename. We don’t know where exactly the file is and we cannot find them, then an error return will be more appropriate.

Programs that use exceptions as a part of their normal processing suffer from readability and maintainability problems. These programs break encapsulation as their methods and their callers are tightly coupled via exception handling.

Use Exceptions for Exceptional Problems



- summary of When to use Exceptions, from The Pragmatic Programmer: from Journeyman to Master

The Art of Simplicity (Part 2)

Most of the time, we never realize why some concepts are simple until we think about them. Just consider this example. Have you ever wondered why manholes are round in shape? We could have made them in many other shapes, say squares or real artistic ones.

Not many people had an answer when Venkat asked this question to the audience. For shapes other than round, there is a greater chance for its lid to fall into the hole. But with a round lid, we don’t face this problem. It can also be easily moved by rolling. Also, at the end of a long tiring day, the workman doesn't have to worry in which direction the lid has to be placed to close the hole properly. See how a very simple design helped in solving many issues that could have happened.

Have you ever given a thought why Mona Lisa is considered to be a masterpiece? Along with the fact that it was painted by the famous artist, Leonardo Da Vinci, it is simplicity that always attracts people towards it.

The essence of this keynote was this: Never complicate things too much.

He concluded the session with a very nice story based on Richard Feynman’s quote...

• • • 

Once, I was giving lectures to a bunch of students in the college. I was teaching the same stuff from past one decade. But, for the first time, something strange happened. All the students had nothing to say, after an hour long lecture , other than-

“Sir, we didn't understand anything!”

I was puzzled. I didn't know what had happened. I went back home. I kept  thinking and thinking about it. Then, I went back to read the same topic.

Slowly and surprisingly, I started understanding better. I uncovered many aspects and by the end of  that night, I got a clear picture about the entire concept.

Next day, I went to the same class and started with the same lecture, but with a new insight. Within five to ten minutes, I could make out that every student understood the concept clearly. All students were happy and they exclaimed-

“Sir, why couldn't we understand this yesterday? It is really very easy. ”

“It is simple. I hadn't really understood what I was teaching till yesterday. But now I know.”

• • • 

- summary of the keynote, The Art of Simplicity, by Dr. Venkat Subramaniam (Agile Kerala 2013)


Thanks to +Vaishnavi M K, +Arun Prakash and +Rohit Kumar for your suggestions and proofreading.. :-)