Requirements traceability is “Necessary but not sufficient”

by with No Comment Observations & Ideas

When asked about “how do you know that your test cases are adequate?”, the typical answer is Requirement Traceability Matrix(RTM) has been generated and that each requirement does indeed have test cases.

Is this logic strong enough? Unfortunately NO! Why? Assume that each requirement had just one test case. This implies that we have good RTM i.e. each requirement has been covered. What we do know is that could there additional test cases for some of the requirements? So RTM is a necessary condition but NOT a sufficient condition.

So, what does it take to be sufficient? If we had a clear notion of types of defects that could affect the customer experience and then mapped these to test cases, we have Fault Traceability Matrix (FTM as proposed by HyBIST). This allows us to be sure that our test cases can indeed detect those defects that will impact customer experience.

Note that in HyBIST potential defects types are mapped to the Cleanliness Criteria derived earlier. Cleanliness criteria are those that have to be met to ensure that customer has a good experience with the system.

This is covered in

Healthy baby at birth!

by with No Comment Learnings & Experiences

A large India Development Center (IDC) of a major consumer electronics& peripherals delivers 3-4 releases of their product every year. They had “birthing” problems – early stage defects were bogging them down. The root cause was identified as ineffective development testing. The team was mature, had good practices and were focused on “unit testing”. The big question that nobody wanted to ask was “what in the name of God is an Unit?”. This resulted in everyone in both early and late stage doing similar tests with poor results.

Applying STEM, we clearly identified the expectations of code from development and listed the types of defects that should not seep from development. Having setup clear cleanliness criteria, we had gotten around the “notion of unit”, and setup a goal focused development test practice. The test cases increased many-fold (this did not increase effort/time) and fault traceability made them purposeful.

The code coverage jumped from 65% to 90% with the remaining 10% identified as exception handling code that were hand-assessed. Now all early stage code were ‘completely assessed’. The RESULT – Defect escapes to QA team dropped by 30-40% and the specialist QA team could focus on their job and releases were made on time.

From premature babies needing incubators, we had transformed the organization to deliver bonny babies!