Posts Tagged ‘Hypothesis-based Testing’

Validation Suite – An innovative way to leveraging test assets & reducing cost of validation

Monday, June 28th, 2010

The test lifecycle produces rich set of assets-strategy, test scenarios/cases, defects, scripts, test data. Other than using these to validate the current release of product, how are these significantly leveraged in the future? How do these assets enable faster ramp-up, de-skilling, optimize testing? Patterns have been used as a method to extract experience and enable a normal person to behave like an experienced person. Sadly, patterns are not commonplace in test engineering.

Validation Suite is a product from STAG that is similar to patterns. Based on HBT (Hypothesis Based Testing) methodology, it enables identifying potential defect types for common features/construction-components along with a common validation strategy consisting of Quality Levels, Test Types and finally a list of common test scenarios/cases/data. The intent is to ensure that we do not start all over again, and leverage the experience typically encoded as tacit knowledge in the individuals.

STAG has applied this to create specific validation suites in the areas of eLearning, ERP, Bluetooth and Mobile applications. These can reduce the cost of validation by 30% and reduce ramp-up time significantly. The key business benefits are (1) Fast ramp-up (2) Readymade test strategy (3) Higher product quality (4) Faster time-to-market and (5) Lower validation cost.

Validation suite for your product can be created by mining your test assets by applying the HBT methodology to create a structured and rich asset base that provides significant leverage subsequently.

Tags: , , ,
Posted in Hypothesis-based Testing | No Comments »

How many “negative test cases” should there be?

Thursday, April 29th, 2010

Before we answer the question let us ensure that we have common understanding of what is positive or negative test cases. In HBT, positive test cases are those whose input values are valid, while a negative test case is one with at least one of the test inputs (i.e. Test data) being incorrect (i.e. out of specification).

The objective of positive test cases is “conformance” while the objective of negative test cases is “robustness”. If a majority of test case are positive, then it implies that we are primarily interested in conformance i.e. ensuring the system handles correct inputs well. This we know is not sufficient, as accidental incorrect inputs should not result in unexpected possibly risky/dangerous behaviour. Hence we need test cases that are indeed “negative”.

So, coming back to the question, what is a good enough distribution of positive and negative test cases? Any quick answer like 75% should not be trusted as they have no basis. So, how do we answer this question? Step back now and look at the number of inputs for a given test case, the clue is there. For example at a lower level of testing, where say we are validating a screen, the number of inputs may be many as the screen may be populated. As we go up the testing level, e.g. Testing a feature that uses a few screens, the test data is not the various individual inputs on the screen, but aggregate data (think like a record) and these may be fewer in number compared to the earlier levels.

Having understood that the number of test data (or inputs) at lower levels is far higher than those at the higher levels, it is only logical to conclude that the number of negative test cases at lower levels. Now how much should that be?  To answer his finally without resorting to magic (!), let us illustrate with simple example. If there are 5 inputs and each input has six possible values (3 positive i.e. valid and 3 negative i.e. invalid) then using simple combinatorial math, we can see that there be (minimally) 3*5=15 negative test cases and (minimally) 5 positive test cases. In this case the 15/(15+5)= (75%) of test cases are negative.

In closing, understanding the number of inputs and clear understanding of what an input at a testing level is (The STEM Core concept “Input granularity principle” of HBT methodology helps in understanding as what an input at a level is) it is possible to quickly estimate the minimal number of negative test cases. This is very useful in quickly ascertaining whether the test cases are conformance and robustness oriented.

This is covered in “HBT.10: Effective review of test cases”, a title in our HBT Series of workshops

Tags: , , , ,
Posted in HBT Series of Workshops, Hypothesis-based Testing, STEM | No Comments »

Ensuring testable non-functional requirements

Thursday, April 29th, 2010

Non-functional requirements are notoriously non-testable! By this, we mean it is more typical that non-functional requirements are fuzzy/less-clear. In a simplistic form “The system should be robust” is non-testable i.e. It is definitely not clear as how to validate this!

Rather than identifying non-functional requirements and describing them, it is suggested that we look at each requirement and partition these into functional and non-functional aspects and probe into the key attributes to be satisfied for the requirement. For attribute, GQM (Goal-Question-Metric) of core concept of STEM enables deriving metric(s) to ensure that each attribute is indeed testable.  Later the various similar attributes across all the requirements can be aggregated to create the system-wide non-functional requirement.

In this manner non-functional requirements are clearer and testable.

This is covered in “HBT.2: Rapid Understanding of customer expectations”, a title in our HBT Series of workshops.

Tags: , ,
Posted in HBT Series of Workshops, STEM | No Comments »

Rapidly understanding the usage profile

Thursday, April 29th, 2010

Understanding the rate and number of transactions probably on a real system is critical to ensure that the system is designed well and later sized and deployed well. Good understanding of the business domain is seen as a key enabler to arrive at the usage profile.

Operational profiling (A STEM core concept) is a scientific way to quickly arrive at a real life profile of usage. Good understanding of this concept alleviates the problem of lack of deep domain knowledge to understand the usage profile. This core concept consists of these key aspects:

  1. Mode – Represents a time period of usage e.g. End of month, where the usage patterns are distinctive and different.
  2. Key operations (features/requirements) used
  3. Types of end users associated with the key features/requirements
  4. Number of end users for each type of users
  5. Rate of arrival of transactions

In short, for a given mode, identify the end users types and their key operations and then identify the number of users for each type of user and then identify the rate of arrival of transaction. Employing this core concept allows us to think better and ask specific questions to understand the marketplace and the usage profile in a typical and worst-case scenario.

This allows us to get a better understanding of the usage and helps in identifying business risks and derive an effective strategy.

This is covered in “HBT.2 : Rapid Understanding of customer expectations”, a title in our HBT Series of workshops.

Tags: , , ,
Posted in HBT Series of Workshops, STEM | No Comments »

Landscaping – A STEM Core concept to understanding system & expectations

Thursday, April 29th, 2010

Understanding a system is non-linear process i.e. we have derive a lot of questions that are interconnected and then seek answers individually and then connect the various answers.

Landscaping is a core concept in STEM (STAG Test Engineering Method) that powers HBT (Hypothesis based testing) that lists the various aspects related to the system, customer, marketplace, technology and forces one to connect these concepts. What emanates is like an interesting web of information (aka Mindmap) that enables you to come up with intelligent questions and there enable rapid understanding.

For example, connecting marketplace, with requirements and end-users, one can arrive at the question: “What are the various markets that we are planning to deploy our system and in these markets, who are the various kinds of end user types and what does each one from the system?

To quote a specific instance, a two-liner requirement spawned 40+ questions rapidly, that when clarified allowed us to understand the requirement in about an hour. In fact, this uncovered issues in the product being built, as certain aspects of the requirements were completely missed by the developers in their implementation. It is to be noted that our ability to identify questions and therefore understand were not due our domain skills, it was purely due to the application of HBT methodology powered by the defect detection technology(STEM) to the problem at hand.

This is covered in “HBT.2: Rapid Understanding of customer expectations”, a title in our HBT Series of workshops.

Tags: ,
Posted in HBT Series of Workshops, Hypothesis-based Testing, STEM | No Comments »

Launching two new HBT series of workshops in MAY 2010

Wednesday, April 21st, 2010

STAG is launching two new workshops in the “HBT Series” in May 2010. We thank our participants of Robust Test Design workshop conducted in Chennai, for suggesting that we come with a new workshop for “How to understand customer expectations rapidly” . Thank you!

The workshops are:

  1. Rapid understanding of customer expectations  on May 27, 2010 at Bangalore and on June 7 at Chennai.
  2. Effective review of test cases  on May 28, 2010 at Bangalore and June 8, 2010 at Chennai.

Workshop details:

Rapid understanding of customer expectations (1-day workshop)

Objective: How to rapidly understand expectations/requirements of the software to be validated in a scientific manner.

Target audience: Test manager, Project manager, Test lead, Test Engineers

To test effectively, estimate correctly, a good understanding of the software/application is very important. The act of understanding is a high maturity skill requiring multiple skills. Domain knowledge is seen as a critical enabler to good understanding. Good documentation of the requirements is also a key ingredient. In real life however, the available documentation of requirements/specifications always lacks the details required for effective testing, and is typically not in sync with the software/system being built. Therefore the test staff with their domain knowledge is expected to come up with good questions and clarify the missing elements and understand the intended behaviors. This is easier said that done, as deep domain knowledge is typically a scarce commodity.

This workshop takes a scientific approach to “act of understanding the intentions or expectations” by identifying key elements required of any requirement/specification and identifying a personal process powered by scientific concepts to ensure that we rapidly understand the intentions and identify the missing elements.

The first two stages of Hypothesis-based testing(HBT) methodology focuses on “Understanding expectations” and “Understand context”, this is powered by the “Business Value Understanding” discipline of STEM, the underlying defect detection technology that powers HBT. This discipline employs seven scientific concepts to enable that the various aspects to understand the expectations can indeed be done in a scientific manner.

This has been successfully used by STAG in its engagements to rapidly identify questions to understand expectations. To quote a specific instance, a two-liner requirement spawned 40+ questions rapidly, that when clarified allowed us to understand the requirement in about hour. In fact, this uncovered issues in the product being built,  as certain aspects of the requirements were completely missed by the developers in their implementation. It is to be noted that our ability to identify questions and therefore understand were not due our domain skills, it was purely due to the application of HBT methodology powered by the defect detection technology(STEM) to the problem at hand.

The topics covered in this workshop are:

  • How to create the big picture and get a good overall view (Landscaping)
  • Identifying end users and their needs
  • Identifying business requirements and  corresponding technical requirements
  • Identifying critical attributes and ensuring that they are testable
  • Understanding the usage patterns/operational profile
  • Identifying business risks and prioritization
  • Understanding intended behaviors for designing test scenarios/cases
  • Formulating cleanliness criteria

The participants will be able to create a User type list, Requirement list, Operational profile, Interaction matrix, Cleanliness criteria  and key questions to understanding expectations at the end of this workshop.

The delivery style will be application oriented, an application will be used to illustrate the concepts and the process of doing.

Each participant will be given a HBT cookbook (NEW!) in addition to the workshop slide set and application case study.

Effective review of test cases (1-day workshop)

Objective: How to assess effectiveness, completeness, consistency and future automation-ability of test cases.

Target audience: Test leads and Test engineers

Test effectiveness is a function of the quantity and quality of test scenarios/cases. The difficult  aspect is assessing if the designed scenarios/cases are indeed adequate. As always, a deep domain and technical knowledge is seen as a critical aspect to effectively review the test scenarios/cases. The challenging part is that deep domain/technical skills is always in short supply.

This workshop teaches a scientific approach to assess the quality of test scenarios/cases by applying a goal centered to testing – “What types of defects should I detect”?. Commencing with identification of potential defect types that will impact the customer experience, the designed scenarios/cases are analyzed for fault coverage in addition to requirements coverage. HBT powered by STEM has a clear structure for effective test scenarios/cases (TS/TC) and this is the basis for assessment of the scenarios/cases. The STEM Test Case Architecture (STEM-TCA) architecture slices the scenarios/cases is multiple ways and allows one to see the gaps in the designed scenarios/cases. For example STEM-TCA requires TS/TC to be segregated by potential defect types and then by quality levels and then by test types, by conformance vs robustness, by importance and other attributes. This approach enables a scientific enquiry process and allows one to assess rapidly and effectively without totally relying on the domain knowledge.

STAG has used this successfully in its boutique service offering of “Test case re-engineering” in its engagements to increase coverage(i.e. defect finding ability)  significantly with its customers. One such interesting work is listed in our blog “Re-architecting  test assets increases test coverage by 250%.” In addition, STAG  has used these concepts to assess completeness of TS/TC for its Japanese customers in their “Diagnostics &  control” solution offering.

At the end of workshop you will able to review the designed scenarios/cases rapidly & effectively enabling you to “produce better bait” to catch the fish! I.e defect defects. The information content of test scenarios/cases plays a vital part of in embarking on successful automation to improve efficiencies. Moving from effectiveness, the workshop will be also enable assessing the efficiency aspect of the testing I.e how can I order and deliver design  assets that will enable faster testing. This is addressed in the assessment of TS/TC on the  automation-fitness aspect.

The topics covered in this workshop are:

  • Understanding the goal of TS/TC i.e. what types of defects should we detect?
  • Assessing basic completeness using RTM(Requirement traceability matrix)
  • Understanding the caveat of RTM i.e. it is necessary but sufficient enough
  • Creating fault traceability matrix
  • Understanding the STEM-TCA
  • Identifying information needed for assessment
  • The personal assessment process for effectiveness and efficiency of TS/TC
  • Understanding the distribution of conformation-oriented(positive) vs robustness (negative)oriented over the various levels of test
  • Limitations of black box techniques
  • What information related to internal aspects of the software do I need to know i.e. how to use white box techniques effectively
  • Metrics that are useful to substantiate the assessment like Test breadth, Test depth and Test granularity

The participants will be able to a create clear assessment report with appropriate metrics   to judge the efficacy and efficiency aspects  at the end of this  workshop.

The delivery style will be application oriented implying test scenarios/case of an real-life application will be used to illustrate the concepts and the process of doing.

Each participant will be given a HBT cookbook (NEW!) in addition to the workshop slide set and application case study.

Both these workshops are limited to a maximum of 25 participants on first-come basis. Email learning at stagsoftware dot com for registration or for more information. We are excited about launching these two unique workshops and look forward to interacting with you.

Click here to download the HBT Series of Workshops brochure.

Tags: , , , , ,
Posted in HBT Series of Workshops, Hypothesis-based Testing, STEM | No Comments »

“The quality race” – STEM wins

Thursday, April 15th, 2010

A large German major with a mature QA practice is seeking new ways to improve its test practice. It all starts with a talk  delivered at this company on “The Science & Engineering of Effective Testing” to its senior management staff and test practitioners in the company. We are amazed at the interest in this topic (75+ people attend the talk) and the enthusiastic response – we are deeply humbled.

A few weeks after this, the management decides to experiment with STEM-based approach to testing. They identify about twenty five people (a small subset of their QA team) to be trained on the new way of testing. We are delighted and conduct a 5-day workshop with intense application orientation, to enable them to understand STEM. The company then decides to conduct a bold experiment- a pilot to evaluate STEM powered approach to testing vis-a-vis their way of testing. They identify a product that is in use for a few years with consumers across the world. They decide to have two five-member identical teams consisting of similar mix of experience levels of people, each given a timeframe of one month to evaluate the new release of this product. These two teams are  kept apart to ensure a controlled experiment and the countdown starts. We wait with bated breath…

The month is slow for us, but it flies for the two teams. Enormous data has been generated and the management analyses them thoroughly to spot the winner. A month later we are called by the senior management. We are sweating, have we won? A few minutes later, it is clear that STEM is a winner. The STEM powered team has designed 3x test cases compared to the non-STEM team and  uncovered 2x number of defects! The icing on the cake is that the couple of defects uncovered by the STEM powered team are “residual defects” i.e. they have been latent in the product for over a year (Remember Minesweeper game on Windows) and one of them corrupts the entire data in the database. Now the discussion steers to effort/time analysis – Does application of STEM require more effort/time? The team has conclusive evidence that it is not significant, implying STEM has enabled them to think better,  not work longer or more.

What enabled the STEM powered team to win the “Race of quality” ? The answers are given by the team itself, and we are delighted, as we have believed in them, and have seen results when we implement them. The top three reasons are: (1) The notion of Potential Defect Types (PDT) is powerful as it forces the team to hypothesize what can go wrong and enable them to setup a purposeful quality goal (2) PDT forces a thorough understanding of the customer expectations and the intended behavior of the product (3) PDT ensures that test design creates adequate test cases, thus eliminating defect escapes and paving the way for robust software.

The STAG team is delighted as their customer acknowledges the effectiveness of the STEM based approach. The team is convinced that STEM powered approach is a winner and is raring to run the marathon, with the customer also cheering them to win!

A heartfelt “Thank you” to the STEM powered team and the innovation-centered Senior Management of the company.

Tags: , , , , ,
Posted in Case-studies, Hypothesis-based Testing, STAG, STEM | No Comments »

10x reduction in post release defects – Finest experience of applying STEM

Thursday, April 15th, 2010

This is about one of the finest experiences that STAG had with a global chip major where the early implementation of STEM yielded significant results. Our engagement with them was to setup an effective validation practice for their porting level API for video decoders. The customer had a great technical team and were involved in both development and QA. The challenge that they faced with their complex product that involved both hardware and software and later system integration on multiple real time OS on various platforms, was that of high defect escapes i.e. post-release field defects.

We spent about a month understanding their domain and the associated technologies. Post this, a detailed analysis yielded interesting data – test cases were primarily conformance oriented, coverage of test cases was suspect, escaped defects seem to propagate from early stages and finally the process of validation was loose.

Having understood the types of defects that were being found and the post-release defects, we figured out the various types of probable defects and the various combinatorial aspects that need to be considered to form a test case. We then staged the validation as consisting of three major levels, the first one at API level, then the next one at a system level, and the last level made up of a customer-centric level that involved using reference applications.

Applying the STEM approach to test design, the test cases were developed, yielding about 6000 test cases at level one and about 800 at the subsequent levels. Also, whereas the ratio of +ve vs –ve test cases was earlier towards the +ve side, after our re-design, the ratio shifted to 60%:40% at the lower level and about 85% :15% at the higher levels. Moreover, the test cases increased in number significantly by a factor of 1000%, allowing for a larger and deeper net to catch many more serious defects. Over the next 9 months, the rate and number of defects detected increased dramatically, resulting in post-release issues reducing by a jaw-dropping 10x times.

Once we solved the test effectiveness problem and increased the yield of defects, the focus shifted to streamlining the process by setting up proper gating in the test process and creating a centralized web based test repository, and finally setting a strong defect analysis system based on Orthogonal Defect Classification (ODC) method. This enabled a strong feedback system, resulting in shifting the defect finding process to earlier stages of SDLC and thereby lowering cycle time. Complementing this, we focused on setting up a custom tooling framework for automating this non-UI based software resulting in a significant cycle time reduction – an entire cycle of tests on a platform took less than 15 hours of time!

This has been one of the finest experiences that we had with STEM, and was a clear winner for STEM implementation. This was only possible because of the very mature engineering management staff of the customer, who were focused on systemic improvement and had systematic improvement goals.

What is to be noted is that our test team was NOT a team with significant depth of experience on the particular product domain. Applying STEM at a personal level, the team was able to understand what was necessary and sufficient for effective validation and complemented the strong technical team with mature defect-oriented thinking. This was an early case study for us to establish that a STEM based approach provided us with the right thinking skills for defect finding, rather than resort to a domain centric approach to defect finding.

Tags: , , , , , ,
Posted in Case-studies, Hypothesis-based Testing, STAG | No Comments »

Re-architecting test assets increases test coverage by 250%

Thursday, April 15th, 2010

This company is an innovative online banking solution provider, having three major products catering to over 100 top financial institutions(FI) of the world including the top five FI in the world. They have a very successful product line, growing rapidly, with major releases approximately every year, incorporating new features to cater to the various needs of the market place. As the code base evolved, the test assets were also modified to reflect the changed product. The challenge faced was that the most of the test cases were passing and the rate of uncovering new defects was low.

The product became huge and the company decided to re-architect the product in order to enable rapid feature addition with low risk. That is when the company decided to re-look at its test assets and re-architect the same to increase the test coverage, improve defect finding ability and ensure that the test assets were future-proof. It had about 8000 test cases then.

We were chartered to analyze the existing test cases for completeness and modifiability and re-architect the same after filling the gaps and to ensure that the future test cases were easily pluggable. Applying STEM, we performed a thorough assessment of the existing test assets and discovered holes in the same. Using the STEM Test Case Architecture (STEM-TCA), we re-engineered the test cases by firstly grouping them into features, then by levels of tests and segregating into various types of tests and then finally by separating into positive and negative test cases. During this process of fitment of existing of test cases into the STEM-TCA, we uncovered quite a few holes. These were filled by STAG by designing 5000 test cases additionally. Not only did the STEM-TCA increase the test coverage by uncovering the missing test cases, it also provided a sharper visibility of the quality as the test cases were well organized by specific defect types. This improved the test coverage by about 250% and the technical management staff were confident about the adequacy of test assets and were also convinced about its future upgradeability and maintainability.

Tags: , , , , ,
Posted in Case-studies, STAG, STEM | No Comments »

Can we guarantee cleanliness of a software?

Wednesday, July 15th, 2009

Couple of weeks ago, I conducted a Thought Leadership workshop on an absolutely new topic “Hypothesis-based testing”. It was conducted in Bangalore at ”The Chancery Hotel” and was well received. Subsequently, I did a 90 minute lecture in Ford Chennai to the QA audience and the feedback that I received was “All the participants thoroughly enjoyed this thought provoking talk to engineer clean software. It was more value adding and was different from the other normal monthly talks”.

So what is “Hypothesis-based testing” and how is it connected to STEM™ 2.0? Let us start from the fundamentals – “Can we guarantee that the released software is clean?” Most often the answer is an emphatic NO! The reason for this is that we do not wish to believe that the final software will have no defects. Hey, wait a minute…. Is this what guarantee means? Not really. As a consumer, we expect the products that we buy to be guaranteed. That is, we expect that it has been well validated, and should ensure a great experience when we use it. In the extreme case when it does fail, we expect fast turnaround via a responsive support system, resulting in a quick fix or replacement. Our customers expect the same too from us as producers of software. Hence guarantee means that we are extremely clear of the probability of the failure and the risk therefore. Once we are sure of this, we should be able to guarantee the cleanliness of software.

What does it take to do this?  If we can guarantee the process of evaluation of software, then we can guarantee that cleanliness of the delivered software. Now what does it take to do this? How can one guarantee the process of evaluation? Most often, we like to believe that experience of the test staff is what will make this possible. But this is simply not enough.  If we can scientifically analyze the process of evaluation, rationally justify why we did the various activities and the outcomes, and yet do not find any anomalies, then we can inclined to think that the process of evaluation is guaranteed to produce clean software. Hence guarantee can be achieved if we can explain the means to achieve the outcome. For example if we can scientifically explain our test strategy, scientifically prove the completeness of test cases etc, we can indeed guarantee cleanliness. The trouble is that the industry does not seem to have rational and scientific form of thinking/method.

STEM™ 2.0 is a method that allows for scientific thinking and disciplined implementation to ensure that the delivered software can indeed be guaranteed to be clean. The central core of STEM™ is establishing a clear goal and then performing activities that will indeed get us to the goal. In STEM™, the goal translates to “Uncover these potential defects”. These potential defects are hypothesized and all the later activities are about proving that the hypothesized defect(s) do/do-not exist.

Hypothesis-based testing is therefore an approach to testing that is about establishing a hypothesis and proving/disproving them. In Hypothesis-based testing, the key tenet is to establish a hypothesis that these potential defects may exist, and then proving/disproving their existence. Hypothesis-based testing is powered by STEM™ 2.0! I hope you see the connection now!

Think – when you go the doctor with ailments, he/she hypothesizes potential problems based on your symptoms, performs diagnostic tests to confirm the hypothesis and then prescribes the treatment regimen. When you go on a holiday, you hypothesize needs and accordingly pack items. Whenever you have a goal-focused, you always hypothesize the future situation(s) and then up with a list of activities. Hypothesis-based testing is the application of this thinking to uncovering defects in software. Hypothesis-based testing is not a normal term, it is invented in STAG! Hypothesis based testing is powered by STEM™ 2.0.

Tags: ,
Posted in Hypothesis-based Testing | No Comments »