Empirical Studies: On software and sharks

Software has become part of our society and it is found in products ranging from microwaves to space shuttles. It implicates that a vast amount of software has been and is being developed. On the other hand, organizations are continuously trying to improve their software process in order to achieve their goals.

Nevertheless, when the improvement proposal has been identified, it is necessary to determine which to introduce, if applicable. Moreover, it is often not possible just to change the existing software process without obtaining more information about the actual effect of the improvement proposal, i.e., it is necessary to evaluate the proposals before making any changes in order to reduce risks. In this context, empirical studies are crucial since the progress in any discipline depends on our ability to understand the basic units necessary to solve a problem. Additionally, experimentation provides a systematic, disciplined, quantifiable, and controlled way to evaluate new theories. It has been used in many fields, e.g., physics, medicine, manufacturing; however, in the software engineering field, this idea started to be explored in the 70s with the work of Victor Basili from University of Maryland. Currently, we can see conferences, books and other efforts in this direction.

At RiSE Labs, we are working hard in this direction. All of the research before being introduced in an industrial scenario should run an empirical study about it showing the benefits and drawbacks. Sometimes, the students and researchers arguing a lot before seeing and understanding the benefits related to it. However, if we look for other examples in the real world, we can see that it is a standard procedure and is being handled seriously.

Last Sunday, Discovery Channel started the Shark Week, a special program with a bunch of information about sharks. I did not have the opportunity to watch the first day, but on Monday I did it. In the Day of the Shark program, researchers from several places around the world were performing important experiments involving sharks attacks. I will explain some of them here and maybe it can be useful:

• What is the best approach in a shark attack? Stay together in a group (like in cartoons. Everybody praying together) or separate, each one trying to save yourself? The experiments identified that is better to stay together. The researchers used fake humans (toys) to perform the experiments and identified it. As in a software engineering experiment, you have many threats to deal (clothes for the toys, lack of signals that human have and sharks’ sensors can recognize, types of sharks, sharks habits….)

• Can the sharks attack during the day and night in the same way? Yes, the experiment showed that sharks are not concerned about day or night.
  

• Other important experiment was performed in order to design devices to avoid sharks attacks. The researchers analyzed the use of electrical devices and a type of gas and these ones were able to scare the sharks for a while. This way, you have some minutes to find a boat, reef, etc.
  

• In another experiment, they designed a material to be resistant for a shark bite. The material was very resistant, however, it was destroyed by the sharks and they are planning a replication with some adjusts. Can you imagine yourself trying this solution before its experimentation?

The program was very impressive and interesting and the researchers presented important findings, for example, comparing the surfers’ dilemma: stay quiet or try to get out with the surfboard. Nevertheless, the experiment which was more impressive to me involved two researchers rounded by sharks [It can be disgusting to watch].

The goal was try to understand the sharks’ behavior. However, one of the researchers had part of his calf bitten by a shark. That was terrible. Although, after the accident, the same researcher replicated the experiment trying to better control the variables (just him in the water, cameras away from him – the guys were in a rock) and obtain more findings about it. In this replication, he did not have problem.

As you can see, empirical studies are critical to improve the science and the way to understand part of the world and its elements. With luck, in some situations in the software engineering area, we do not have to deal with human in critical conditions to run experiments.

In addition, I agree with David Parnas when he said that we cannot do experiments for everything that we are defining (or we just have to do it for the next years and forget our current activities), however, in some situations they are very important to present evidences about something.