A New Zealand based customer in the heath care domain embarked on a journey of migrating their Delphi-based products to Microsoft technologies. The products use specialized GUI controls that are not recognized by the typically popular tools. The company was keen to embark on automation right from the early stage of migration. And the budget to develop automation was tight.

We conducted Proof-Of-Concept (POC) to identify the tool that would support automation for both Delphi and VB.Net. We discovered that most popular tools were indeed not compatible with the developed product. The POC concluded that Test Complete did support both Delphi and VB.Net with a few constraints. It was very cost effective however but not user friendly. We convinced the management of our decision. The project started off with us identifying test cases which could be automated. Seven modules were automated and demonstrated.

We developed reusable Keyword Driven Framework for the client. Both individual test case execution and batch run was possible just by choosing the test cases. STAG provided detailed demo of the framework to the in-house QA team.

However some of the test cases chosen for automation were not complete. We validated the test cases, made the necessary changes and then initiated the scripting. The automation work was divided between the STAG and customers team. As we automated the test cases, we guided and trained the customer’s team to automate.

The result – By automating 326 test scenarios, the testing time was cut down from 80 hours to 12 hours! We saved the customer significant money spent on the tool, but more by enabling them to release the product to market ahead of schedule!

A large petroleum major was rolling out a specialized solution to ensure that fleet tracking solution to ensure zero pilferage during transport. The solution consisted of a plethora of technologies (GPS, GSM, Web, Mapping) and our role was to ensure that the final solution is indeed risk-free for deployment.

With the launch date in the next few weeks, we got cracking on applying HBT to extract the cleanliness criteria from the business and technical specifications outlined in the tender document.The cleanliness criteria consisted of multiple aspects- deployment environment correctness, cleanliness of software, clean working of hardware/software interfaces and finally the ability to support a large load, volume with real-time performance.

We identified the potential types of defects that spanned the entire spectrum of hardware and software. The first step was to understand system development process and identify senior consultants visited the vendor’s facility to assess the people and processes used to develop the system. This provided a clear picture of what to expect and the work that lay ahead of us.

Post our understanding of the development system, we developed a scientific strategy and the evaluation scenarios. A variety of tests were identified – individual feature validation, simulating various business use-cases, understanding load limitations and performance evaluation of the system.

Now we were ready to validate the final system in the data center. The first cut of the solution was used to develop a set of automated scripts for large scale load/stress/performance testing. The system was populated with large data representing a real life system. Vehicles were fitted with the vehicle mounted unit. We were ready to roll now.

The various vehicles were set in motion in various terrains, at various speeds and mapping of the fleet on the India map was validated. We simulated large number of vehicles with data arriving from the simulators at a high rate to ensure that performance was indeed real-time.

In addition, the deployment environment was validated, configurations checked, legality of the software verified. We also verified that the solution integrates with customer’s SAP database as well.

Bugs popped up and were fixed. We recommended changes in the system capacity, pushed the vendor to close all the critical, high and medium priority defects before providing a qualitative feedback on the solution and the potential risks. Once satisfied our customer’s investment was safe, we gave a go-ahead to rollout the solution.

Validating early stage pre-code artifacts like requirement document is challenging. This is typically done by rigorous inspection and requires deep domain knowledge. One of our Japanese customer threw a challenge – “How can you use HBT/STEM to scientifically validate requirements without knowing the domain deeply?” .

The core aspect of HBT is to hypothesize potential defect types that prove that they do not exist. These are identified by keeping in mind the end users and the technology used to construct the system. So how do you apply this to validate a pre-code artifact?

We commenced by identifying the various stakeholders for requirement document and then identified key cleanliness attributes. These cleanliness attributes if met would imply that the requirements was indeed clean. We were excited by this. We then moved and identified potential defect types that would impede these cleanliness attributes/criteria.

Lo behold, the problem was cracked and we then identified the various types and the corresponding evaluation scenarios for validating the requirements/ architecture document. We came up with THIRTY+ defect types that required about 10+ types tests conducted over TEN quality levels with a total of SIXTY FIVE major requirement evaluation scenarios to validate a requirement.

What we came up is not yet-another-inspection-process that is dependent on domain knowledge, but a simple & scientific approach consisting a set of requirement evaluation scenarios that could be applied with low domain skill to ensure that the requirement/architecture can indeed be validated rapidly and effectively. These ensure that the requirement document is useful to the various stakeholders over the software lifecycle and does indeed satisfy the intended application/product attributes.

We now have a unique solution “Clean Requirements Solution” that is an adaptation of HBT to validate requirements.

The problem that stumped them was “how to automate the final-install validation of multi-platform distributed product”. They had automated the testing of the individual components using SilkTest, but they were challenged with “how to unify this and run off a central console on various platforms at the same time”.

Considering each platform-combination took about a day, this required approximate two months of final installation build validation, and by the time they were done with this release, the next release was waiting! This was a relentless exercise, consuming significant QA bandwidth and time, and did not allow the team to do things more interesting or important.

The senior Management wanted single-push-button automation -identify what platform combination to schedule next, allocate machine automatically from the server farm, install and configure automatically, fire the appropriate Silk scripts and monitor progress to significantly reduce time and cost by lowering QA bandwidth involved in this effort. After deep analysis, in-house QA team decided this was a fairly complex automation puzzle and required a specialist! This is when where we were brought in.

After an intense deep-dive lasting about four weeks, we came up with a custom master-slave based test infrastructure architecture that allowed a central console to schedule various jobs onto the slaves, utilizing a custom developed control & monitoring protocol.  The solution was built using Java-Swing, Perl, Expect and adapters to handle Silk scripts. Some parts of the solution where on Windows platform while some on UNIX.  This custom infrastructure allowed for scheduling parallel test runs, automatic allocation of machines from a server farm, installing appropriate components on appropriate machines, configuring them and finally monitoring the progress of validation through a web console.

This test infrastructure enabled a significant reduction of the multi-platform configuration validation. The effort reduced from eight weeks to three weeks. We enjoyed this work simply because it was indeed a boutique work fraught with quite a few challenges. We believe that this was possible because we analyzed the challenging problem from wearing a development hat and not the functional test automation hat.

STAG singularly focused to ensure the defect escapes are minimized. Hence a three-pronged approach was adopted to determine the breadth and depth of testing required -

• Identify what poses high business risks? What has been de-risked already? What remains as risk that is to be assessed?
• How well has the “net” been cast to uncover defects in the lifecycle? Are the methods to uncover defects expansive/complete?
• Are the test cases (i.e. those inputs that have the ‘power’ to detect anomalies) good?  Do the already existing test cases and therefore the tests conducted have the power to uncover high-risk business issues?

Fixing the high impact defects improved the stability of the product– which otherwise could have led to USD 250K support cost in the initial months. The release worthiness certificate lowered the business risk for the customer and newly gained delivery confidence by the customer management powered their successful product launch and on time to market.

Validation Suite is a product from STAG that is similar to patterns. Based on HBT (Hypothesis Based Testing) methodology, it enables identifying potential defect types for common features/construction-components along with a common validation strategy consisting of Quality Levels, Test Types and finally a list of common test scenarios/cases/data. The intent is to ensure that we do not start all over again, and leverage the experience typically encoded as tacit knowledge in the individuals.

STAG has applied this to create specific validation suites in the areas of eLearning, ERP, Bluetooth and Mobile applications. These can reduce the cost of validation by 30% and reduce ramp-up time significantly. The key business benefits are (1) Fast ramp-up (2) Readymade test strategy (3) Higher product quality (4) Faster time-to-market and (5) Lower validation cost.

Validation suite for your product can be created by mining your test assets by applying the HBT methodology to create a structured and rich asset base that provides significant leverage subsequently.

The objective of positive test cases is “conformance” while the objective of negative test cases is “robustness”. If a majority of test case are positive, then it implies that we are primarily interested in conformance i.e. ensuring the system handles correct inputs well. This we know is not sufficient, as accidental incorrect inputs should not result in unexpected possibly risky/dangerous behaviour. Hence we need test cases that are indeed “negative”.

So, coming back to the question, what is a good enough distribution of positive and negative test cases? Any quick answer like 75% should not be trusted as they have no basis. So, how do we answer this question? Step back now and look at the number of inputs for a given test case, the clue is there. For example at a lower level of testing, where say we are validating a screen, the number of inputs may be many as the screen may be populated. As we go up the testing level, e.g. Testing a feature that uses a few screens, the test data is not the various individual inputs on the screen, but aggregate data (think like a record) and these may be fewer in number compared to the earlier levels.

Having understood that the number of test data (or inputs) at lower levels is far higher than those at the higher levels, it is only logical to conclude that the number of negative test cases at lower levels. Now how much should that be?  To answer his finally without resorting to magic (!), let us illustrate with simple example. If there are 5 inputs and each input has six possible values (3 positive i.e. valid and 3 negative i.e. invalid) then using simple combinatorial math, we can see that there be (minimally) 3*5=15 negative test cases and (minimally) 5 positive test cases. In this case the 15/(15+5)= (75%) of test cases are negative.

In closing, understanding the number of inputs and clear understanding of what an input at a testing level is (The STEM Core concept “Input granularity principle” of HBT methodology helps in understanding as what an input at a level is) it is possible to quickly estimate the minimal number of negative test cases. This is very useful in quickly ascertaining whether the test cases are conformance and robustness oriented.

This is covered in “HBT.10: Effective review of test cases”, a title in our HBT Series of workshops

Is this logic strong enough? Unfortunately NO! Why? Assume that each requirement had just one test case. This implies that we have good RTM i.e. each requirement has been covered. What we do know is that could there additional test cases for some of the requirements? So RTM is a necessary condition but NOT a sufficient condition.

So, what does it take to be sufficient? If we had a clear notion of types of defects that could affect the customer experience and then mapped these to test cases, we have Fault Traceability Matrix (FTM as proposed by HBT). This allows us to be sure that our test cases can indeed detect those defects that will impact customer experience.

Note that in HBT potential defects types are mapped to the Cleanliness Criteria derived earlier. Cleanliness criteria are those that have to be met to ensure that customer has a good experience with the system.

This is covered in “HBT.10 : Effective review of test cases”, a title in our HBT Series of workshops.

What is efficiency? It is ensuring that we execute the test cases in as short a time as possible with optimal effort and no more.  Understanding (1) the priority or business importance of test cases, (2) knowing what test cases to execute in what part of the lifecycle (3) clear segregation of test cases by various types of tests and levels enables to optimize testing and become efficient.

This is covered in our HBT Series of workshop “HBT.10 : Effective review of test cases”.

Rather than identifying non-functional requirements and describing them, it is suggested that we look at each requirement and partition these into functional and non-functional aspects and probe into the key attributes to be satisfied for the requirement. For attribute, GQM (Goal-Question-Metric) of core concept of STEM enables deriving metric(s) to ensure that each attribute is indeed testable.  Later the various similar attributes across all the requirements can be aggregated to create the system-wide non-functional requirement.

In this manner non-functional requirements are clearer and testable.

This is covered in “HBT.2: Rapid Understanding of customer expectations”, a title in our HBT Series of workshops.