Ganesan ramalingam microsoft research

Nagappan points out two of the most obvious reasons why code coverage alone fails to predict error rates: usage and complexity. Code coverage is not indicative of usage. If 99 percent of the code has been tested, but the 1 percent that did not get tested is what customers use the most, then there is a clear mismatch between usage and testing.

There is also the issue of code complexity; the more complex the code, the harder it is to test. So it becomes a matter of leveraging complexity versus testing. After looking closely at code coverage, Nagappan now can state that, given constraints, it is more beneficial to achieve higher code coverage of more complex code than to test less complex code at an equivalent level. Those are the kinds of tradeoffs that development managers need to keep in mind.

Nagappan and his colleagues then examined development factors that impact quality, another area of software engineering discussed in The Mythical Man Month. One of the recent trends that caught their interest was development practices; specifically, test-driven development TDD versus normal development.

In TDD, programmers first write the test code, then the actual source code, which should pass the test. This is the opposite of conventional development, in which writing the source code comes first, followed by writing unit tests.

Although TDD adherents claim the practice produces better design and higher quality code, no one had carried out an empirical substantiation of this at Microsoft. The study and its results were published in a paper entitled Realizing quality improvement through test driven development: results and experiences of four industrial teams , by Nagappan and research colleagues E. What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams.

They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer. Again, these are decisions that managers have to make—where should they take the hit? But now, they actually have quantified data for making those decisions. Another development practice that came under scrutiny was the use of assertions.

In software development, assertions are contracts or ingredients in code, often written as annotations in the source-code text, describing what the system should do rather than how to do it. But nobody had done the work to quantify just how much assertions improved software quality. One reason why assertions have been difficult to investigate is a lack of access to large commercial programs and bug databases. Also, many large commercial applications contain significant amounts of legacy code in which there is minimal use of assertions.

All of this contributes to lack of conclusive analysis. At Microsoft however, there is systematic use of assertions in some Microsoft components, as well as synchronization between the bug-tracking system and source-code versions; this made it relatively easy to link faults against lines of code and source-code files. The research team managed to find assertion-dense code in which assertions had been used in a uniform manner; they collected the assertion data and correlated assertion density to code defects.

The team observed a definite negative correlation: more assertions and code verifications means fewer bugs. Looking behind the straight statistical evidence, they also found a contextual variable: experience. Software engineers who were able to make productive use of assertions in their code base tended to be well-trained and experienced, a factor that contributed to the end results.

These factors built an empirical body of knowledge that proved the utility of assertions. The work also brings up another issue: What kind of action should development managers take based on these findings? The research team believes that enforcing the use of assertions would not work well; rather, there needs to be a culture of using assertions in order to produce the desired results. Nagappan and his colleagues feel there is an urgent need to promote the use of assertions and plan to collaborate with academics to teach this practice in the classroom.

Having the data makes this easier. Nagappan recognized that although metrics such as code churn, code complexity, code dependencies, and other code-related factors have an impact on software quality, his team had yet to investigate the people factor. The first challenge was to somehow describe the relationships between members of a development group.

The team settled on using organizational structure, taking the entire tree structure of the Windows group as an example. They took into account reporting structure but also degrees of separation between engineers working on the same project, the level to which ownership of a code base rolled up, the number of groups contributing to the project, and other metrics developed for this study. Basili of the University of Maryland, presents startling results: Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent.

This is a significantly higher precision than traditional metrics such as churn, complexity, or coverage that have been used until now to predict failure-proneness. This was probably the most surprising outcome of all the studies.

We thought it would be comparable to other code-based metrics, but these factors were at least 8 percent better in terms of precision and recall than the closest factor we got from the code measures. Eight percent, on a code base the size of Windows, is a huge difference.

Regions can be memory-safe and could be inferred automatically. Papers and proceedings are freely available to everyone once the event begins. Skip to main content. Conferences Students Sign in. Bronze Sponsor. Monday, May 18, - am - am. Ionel Gog, University of Cambridge. Malte Schwarzkopf, University of Cambridge.

Kapil Vaswani, Microsoft Research. Dimitrios Vytiniotis, Microsoft Research. Ganesan Ramalingam, Microsoft Research. Manuel Costa, Microsoft Research.