Your code as a Crime Scene by Adam Tornhill

Your Code As a Crime Scene: Use Forensic Techniques to Arrest Defects, Bottlenecks, and Bad Design in Your Programs by Adam Tornhill
My rating: 4 of 5 stars

A very different way of looking at problems in the code. The author suggests various non-traditional means of identifying problems in software development.

Number of times a code has changed, code churn (number of lines added, removed), number of developers working on a single piece of code, code changing together.

The primary requirement for all of this that the version control should be adhered to correctly. Every developer should have a personal login Id and should use it to checkin at regular intervals.

Many of the techniques provided in the book are available in the form the tool from https://github.com/adamtornhill/code-.... These are only starting tools. These need to be coupled with other tools like d3.js https://d3js.org/ to get a good representation of the status of the code.

A must read for all software developers.

View all my reviews

Why Agile is not successful in Asia?

In ​​this​ very thoughtful article the author illustrates why agile way of working has not been very successful in Asia. It is a sad reflection on the state of affairs. While the author has not taught in India and has not quoted this to be an explicit problem it is should not be difficult for us to extend the same reasoning for India.
​The top reasons that the writer quotes are
​1. Hierarchy rules roost in Asia. As a corollary one does not defy one's superior in the hierarchy. This leads to the expectation "People expect to be told what to do and people want to tell other people what to do because that is how the system works". This is apparent. Many projects where the client controls every design and tells us how to code we are OK, but the moment the client says go ahead and give me an application that does this work for me, we are lost as the babes in the wood. This is one of the key reasons many of the projects that we carry out for our Indian clients fail to succeed and if they do so it is only after a lot of struggle and bitterness between the various stakeholders. This is one of the reasons that none of the big IT companies in India have any significant product.
​2. Let's keep things in harmony. Not only are we not open to saying no or suggesting a different way of doing things to our superior, we do not even voice our opinion amongst our peers especially the loud mouthed ones and the ones who are favoured by the superior. This attitude is partly due to our fear of losing our jobs. We do not have the confidence that if we do fit into an organization where we can speak our minds let us look elsewhere. Traditionally too we are brought up to avoid conflicts. The typical reaction of a parent is "Oh! OK that boy bullies you, stay away from him". Also our managers do not wish to hear of any problems, they want to have solutions and further the solution should be executed by the one who has suggested it not matter that it happens to be the duty of somebody else to do it.
​3. The third reason is a damnation of our educational system "Asian education system is all about high grades and ranks, not about experimenting, self-discovery and making mistakes, which is what Agility is all about." How very true. We kill creativity right from the nursery. See the wonderful TED ​talk by Ken Robinson where he criticises the American education system for killing creativity, I shudder to think what he will have to say of our educational system. No rote learning is going to help us become good programmers. We may get all the certifications (and hence my disbelief in any certification) that is required to prove our credentials as programmers (Oracle Certification, Java Certification etc.), as managers (PMP, Prince etc), as architects (TOGAF etc) as IT process organizers (ITIL, Agile, Scrum etc) but none matter as most of us have only learnt the theory by rote and have nary an experience and are absolutely zilch when coming to practising the principles laid out in them.
​4. The fourth reason is the clichéd last straw on the camel's back. It says "outsourcing" is one of the reasons. The work is outsourced to reduce cost. As one pays less and less one gets lesser and lesser experienced, lesser and lesser quality people to work. Again this is out in open for everybody in the IT industry to see, only we do not acknowledge it. This is another reason why we can almost never develop a product of our own. Chetan Bhagat was not wrong when he stated that we are preparing a generation of not so intelligent people by pushing many towards the call centers.

How to Stop Sucking and be Awesome Instead by Jeff Atwood

How to Stop Sucking and Be Awesome Instead by Jeff Atwood
My rating: 4 of 5 stars

Another nice book of blog entries from Jeff Atwood.

First few blogs are about how one should determine at a very early age if one can program or not and should drop out of a programming career if one is not. He speaks about "sheep that can program and goats that cannot program" should be separated out early in the career so that software can become better.

Some of the key observations that I liked are "You have to truly believe, as a company, and as peers, that crucial innovations and improvements can come from everyone at the company at any time, in bottom-up fashion - they aren't delivered from on high at scheduled release intervals in the almighty master plan.

In another blog he speaks about how important it is to persuade others to do something. He refers to a set of dialog from the movie based on Idi Amin. Idi Amin is speaking to his trusted aide a Scottish Doctor.
Idi Amin: I want you to tell me what to do!
Garrigan: "You want me to tell you what to do?
Amin: Yes You are my advisor. You are the only one I can trust here. You should have told me not to throw the Asians out in the first place!
Garrigan: I did!
Amit: But you did not persuade me, Nicholas. You did not persuade me!

Not a very atypical dialog one is likely to have with either one's manager or client. :-)

Another important advice with which I cannot agree more since I have given the same advice to many others who have asked me for my opinion. "Whatever project you are working on, consider it an opportunity to learn and practice your craft. It is worth doing because, well it is worth doing. The journey of the project should be its own reward regardless of whatever happens to lie at the end of that journey."
The corollary is another thing that I keep stressing on; "Never get attached to a project. Execute the project to the best of the abilities, learn along the way and if for some reason beyond your control the project fails or does not see the light of the day, so be it."

Speaking on merit based growth in an organization Jeff has to say the following "Remove barriers that rob people in management and in engineering of their right to pride of workmanship. This means [among other things] abolishment of annual or merit rating and management by objectives. "Even people who think of themselves as Deming-ites have trouble with this one. They are left gasping. What the hell are we supposed to do instead? Deming's point is that MBO and its ilk are copouts. By using simplistic extrinsic motivators to goad performance, managers excuse themselves from harder matters such as investment, direct personal motivation, thoughtful team formation, staff retention, and ongoing analysis and redesign of work procedures. Our point here is somewhat more limited: Any action that rewards team members differentially is likely to foster competition. Managers need to take steps to decrease or counteract this effect.

In one blog Jeff compares F-86 and MIG-15s. The latter was far more superior to the former, but the fighter pilots preferred the former. The difference was that F-86 had Hydraulic flight controller compared to the manual flight controller of MIG-15. This meant that each maneuver increased the fatigue of the MIG-15 pilot even though he might have out-maneuvered the F-86 pilot. The F-86 could maneuver quicker as compare to the MIG-15 as he was less fatigued and this tilted the pilots to favour F-86 despite its limited abilities. Jeff calls this the Boyd's Law of Iteration which states "Speed of iteration beats quality of iteration". Jeff argues the same is true for software development. Although he says in other places that quality cannot be sacrificed beyond a point.

There is a whole set of blogs in User Interface and Usability. One book the author highly recommends is Don't make me think by Steve Krug https://www.amazon.in/Dont-Make-Think... and another one is Rocket Surgery made easy again by Steve Krug https://www.amazon.in/Rocket-Surgery-....
He speaks about the Fitts law which states "Put all commonly accessed UI elements on the edges of screen. Because the cursor automatically stops at the edges, they will be easier to click on. Make clickable areas as large as you can. Larger targets are easier to click on". One should not ignore the corollary of rule which would read "Make all the clicks that the user must be kept safe from as difficult as possible". Jeff refers to this principle as the "seat ejector" button. This button should be easy to find in an emergency, but should not be place such that the pilot ends up turning this on instead of the navigation lights. Buttons like delete all my mails and such should be available, but should be placed such that the no user would click it by mistake.

Speaking on importance of saying not to demands, Jeff says "It is easy to dismiss Just say No as a negative mindset, but I think it is a healthy and natural reaction to observation that optimism is an occupational hazard of programming". Cannot agree with him more as I have been pulled up for saying no many a time in my career.

Speaking about usability Jeff argues that most users of the applications do not progress beyond the intermediary stage. He argues that most move from the novice to Intermediary stage quite quickly or drop off as an user of the system if they find it too difficult to use. Once they reach this stage they remain in this stage for a long time and only a very few move to the expert level. Given this he argues that software should be targeted towards these users rather than at the novices or the experts. He states that most marketing people would advocate making software for the novice as these would be the ones that the marketing people would encounter most of the times, while the software developer would want to address the experts as it is likely that they would be geeks themselves and would want maximum flexibility and most features.

Writing on security and hacking the author says that today to hack into a site one needs social skills and not technical skills. Technology has advanced to a level where hacking into a website has been made difficult enough, but people are not inure to social engineering and that it is much easier way of hacking into a system.

On the whole a very good read.




View all my reviews

Are Processes required for Programming?

When I started programming,
I just talked to the customer,
And coded and everyone was rocking,
Nothing was only bluster.

Then they said we need to do waterfall
Somebody talked to the customer,
Someone else coded leading to downfall,
And the project manager went a fluster.

Now they are saying use Agile,
Everybody talking to the customer,
It is making everything fragile,
And the project manger has lost his luster.

Why one cannot have Software Factories?

A few weeks ago a blog was published asking a rhetoric question "Can software be created in factories?". My good friend pointed me to the wikipedia post on "Software Factory". What I would like to point out is that the statement "Software factory refers to a structured collection of related software assets that aids in producing computer software applications or software components according to specific, externally defined end-user requirements through an assembly process. ^[1] A software factory applies manufacturing techniques and principles to software development to mimic the benefits of traditional manufacturing. Software factories are generally involved with outsourced software creation." from the wikipedia is completely incorrect and fallacious, despite the number of individuals who believe it.

Martin Fowler's bliki, Code as Documentation, has a link to Jack Reeve's famous essay "What is Software Design?".

This article first appeared in 1992 in the C++ Journal. It was written by Jack Reeve who had been in the industry for more than 10 years at the time and the trigger was the fact that C++ had taking the software world by storm. It was being seen as the panacea for all the problems plaguing the software industry during that time.

He summarizes the article as follows:

To summarize:

• Real software runs on computers. It is a sequence of ones and zeros that is stored on some magnetic media. It is not a program listing in C++ (or any other programming language).
• A program listing is a document that represents a software design. Compilers and linkers actually build software designs.
• Real software is incredibly cheap to build, and getting cheaper all the time as computers get faster.
• Real software is incredibly expensive to design. This is true because software is incredibly complex and because practically all the steps of a software project are part of the design process.
• Programming is a design activity—a good software design process recognizes this and does not hesitate to code when coding makes sense.
• Coding actually makes sense more often than believed. Often the process of rendering the design in code will reveal oversights and the need for additional design effort. The earlier this occurs, the better the design will be.
• Since software is so cheap to build, formal engineering validation methods are not of much use in real world software development. It is easier and cheaper to just build the design and test it than to try to prove it.
• Testing and debugging are design activities—they are the software equivalent of the design validation and refinement processes of other engineering disciplines. A good software design process recognizes this and does not try to short change the steps.
• There are other design activities—call them top level design, module design, structural design, architectural design, or whatever. A good software design process recognizes this and deliberately includes the steps.
• All design activities interact. A good software design process recognizes this and allows the design to change, sometimes radically, as various design steps reveal the need.
• Many different software design notations are potentially useful—as auxiliary documentation and as tools to help facilitate the design process. They are not a software design.
• Software development is still more a craft than an engineering discipline. This is primarily because of a lack of rigor in the critical processes of validating and improving a design.
• Ultimately, real advances in software development depend upon advances in programming techniques, which in turn mean advances in programming languages. C++ is such an advance. It has exploded in popularity because it is a mainstream programming language that directly supports better software design.
• C++ is a step in the right direction, but still more advances are needed.

The points to note with respect to the factory aspect of software are highlighted in red. Note that the author states the coding is design and one cannot dispute this fact. An since one does not design in a factory software development cannot be considered to happen in a factory. It may look like splitting hairs, but for somebody who is coding, be it a novice who has started yesterday, or be it somebody who has been doing it for donkey's years it is apparent that this is indeed a fact. One does keep designing practically with every line of code.

Another interesting excerpt is "In software engineering, we desperately need good design at all levels. In particular, we need good top level design. The better the early design, the easier detailed design will be. Designers should use anything that helps. Structure charts, Booch diagrams, state tables, PDL, etc.—if it helps, then use it."


This was the statement of the author in the essay published in 1992. Writing about this in 2005 the author says "Today, I would phrase it differently. I would say we need good architectures (top level design), good abstractions (class design), and good implementations (low level design). I would also say something about using UML diagrams or CRC cards to explore alternatives."
This is what the author is referring to from the earlier article: "We must keep in mind, however, that these tools and notations are not a software design. Eventually, we have to create the real software design, and it will be in some programming language. Therefore, we should not be afraid to code our designs as we derive them."

The author goes on to say "This is fundamental. I am not arguing that we should not “do design.” However you want to approach the process, I simply insist that you have not completed the process until you have written and tested the code."


Note that the author bolsters the argument that software development involves design at all stage. It is not limited to a single design phase.

Another interesting statement in the second essay is "When the document is detailed enough, complete enough, and unambiguous enough that it can be interpreted mechanistically, whether by a computer or by an assembly line worker, then you have a design document. If it still requires creative human interpretation, then you don’t.". Again goes on to prove that software cannot be created in factories.

One final argument to support that Software cannot be created in factory "The problem with software is - design is not just important, it is basically everything. Saying that programmers should not have to design is like saying fish should not have to swim. When I am coding, I am designing. I am creating a software design out of the void."

Update
When this was sent to a few people the reply I got back is "In the Indian IT industry, there is no such thing as a "Less Able Programmer". All donkeys can be "processed" to become a stallion. all crows can become swans..." and all that can be said about this sad fact is that "This belief is exactly the bane of the Indian IT industry and in my, black, cynical, negative opinion is going to lead to the downfall of the what we today consider to be a cash cow."

Cargo Cult in Software Industry

Recently I read the http://yost.com/misc/cargocult.html by Richard Feynman, and I was surprised by how much of this Cargo Cult is being practiced in today's Software Industry.

The managers in IT industry saw the manufacturing companies were benefiting by adopting various standard processes and by getting them to be validated and certified by standard organizations. ISO, Six Sigma are certifications for the processes followed in manufacturing industry. And industries which have these certifications have have manufactured better quality products, they have proved to be more efficient in usage of time and resources and have generated large net profits. The managers in the software thought that aping the processes in the manufacturing industry will help them achieve the same results that have been achieved in the manufacturing industry.

Processes in a Manufacturing Industry
In a manufacturing industry one person or a group of people who understand the manufacturing process in detail can apply their thought generate a set of standard processes for each step of the manufacturing process. If the workers on the shop floor follow these processes the end result will be to achieve the above mentioned results. This is possible because various steps in a manufacturing industry are more or less standardized. A manufacturing plant repeatedly produces the same output again and again following these processes. E.g. a automobile manufacturing unit keeps churning out automobiles of the same kind, if at all with minor variations between any two automobiles. This is one of the key reasons why it has been possible to automate many of the processes in this industry.

Applications in a Software Industry
The software industry works in complete contrast with the way things work in an manufacturing industry. No software company churns out the same product again and again the way it is done in any manufacturing unit. Each application or product that is developed by the software company is vastly different from the earlier applications or products that it would have generated. Although they may use the same framework, same architecture what they need to do will vastly different hence resulting vastly output. And this is the reason why attempts to automate development of software has failed miserably. Some aspects like testing, checking for code quality (within a limited scope) has been automated but the key aspect of writing code which is the heart of software development has not yet been automated. Again certain aspects can and have been automated but the majority of the code in any application has to be hand written by developers.

When it comes to replacing one developer with another it becomes very important that the new developer understand the domain for which application is being developed, understand the standards and processes that are being followed and only they will they be able to become productive.

Additionally each developer comes with a different background and writes code differently even while adhering to same standards of naming and other standard code quality checks making it difficult to create software with uniform structure.

No amount of documentation, training, processes and checks and balances to ensure processes are being followed will guarantee an efficient software. Only use of good quality developers with sufficient experience will ensure good code.

What will happen to the Indian IT Companies

To read what will happen to the big Indian IT companies read What happened at Yahoo?, The same thing that happened to Yahoo is going to happen to the Indian IT companies sooner or later. The time period seems stretched than it was for Yahoo. The only reason for this timeline being so long is that India has been the only company to give people for software development at a cheap rate. Other countries are yet to reach the levels of the rate offered by Indian IT companies and they undercut each other.
The revenue of the Indian IT companies grows because of the quantity of mediocre people that they can supply and not because of the superior quality of the people. That is why most India IT CEOs are talking of linear growth and non-linear growth. They have been in the linear growth mode for so long and company that has tried to shift to the non-linear growth has failed.
The mindset in such companies is such that one should get immediate results and one should get it with inexperienced programs. The concept of promoting, what the article calls, "Hacker Culture" is completely missing in these IT companies. And for the consequences of this, see the paragraph below.
See the paragraph "In technology, once you have bad programmers, you're doomed. I can't think of an instance where a company has sunk into technical mediocrity and recovered. Good programmers want to work with other good programmers. So once the quality of programmers at your company starts to drop, you enter a death spiral from which there is no recovery.". And if any company in the Indian IT industry wants to refute this, it is doomsday for that company.
See the paragraph "So which companies need to have a hacker-centric culture? Which companies are "in the software business" in this respect? As Yahoo discovered, the area covered by this rule is bigger than most people realize. The answer is: any company that needs to have good software."
So can any of Indian IT company claim to be producing good software on their own. The answer is a big NO.
So how do these companies propose to exist a few years down the line. They are only hoping that the trend of the big MNCs giving them work to code will continue long enough for them to survive and probably thrive. Does India progress in any way because of this. Monetarily yes, but knowledge wise it is in no way benefitting.

The last para of the article sums up the entire premise "Why would great programmers want to work for a company that didn't have a hacker-centric culture, as long as there were others that did? I can imagine two reasons: if they were paid a huge amount, or if the domain was interesting and none of the companies in it were hacker-centric. Otherwise you can't attract good programmers to work in a suit-centric culture. And without good programmers you won't get good software, no matter how many people you put on a task, or how many procedures you establish to ensure "quality.""
No matter what ISO certification one has, no matter what CMM level one is at, no matter how many PMP certified Project Managers you have, no matter how many ITIL certified people you have, no matter how many other IT certified people you have one cannot establish Quality.
To survive and thrive, the Indian IT companies need to come out of this complacency and need to take drastic steps to reverse this "Death Spiral" that they are into.