Archive for the ‘Case-studies’ Category

Low cost automation challenge

Monday, August 9th, 2010
A New Zealand based customer in the heath care domain embarked on a
journey of migrating their Delphi-based products to Microsoft technologies.
The products use specialized GUI controls that are not recognized by the
typically popular tools. The company was keen to embark on automation right
from the early stage of migration. And the budget to develop automation was
tight.
We conducted Proof-Of-Concept (POC) to identify the tool that would support
automation for both Delphi and VB.Net. We discovered that most popular tools
were indeed not compatible with the developed product. The POC concluded
that Test Complete did support both Delphi and VB.Net with a few constraints.
It was very cost effective however but not user friendly. We convinced the
management of our decision. The project started off with us identifying test
cases which could be automated. Seven modules were automated and
demonstrated..
We developed reusable Keyword Driven Framework for the client. Both
individual test case execution and batch run was possible just by choosing the
test cases. STAG provided detailed demo of the framework to the in-house QA
team.
However some of the test cases chosen for automation were not complete. We
validated the test cases, made the necessary changes and then initiated the
scripting. The automation work was divided between the STAG and customers
team. As we automated the test cases, we guided and trained the customer’s
team to automate.
The result – By automating 326 test scenarios, the testing time was cut down
from 80 hours to 12 hours! We saved the customer significant money spent on
the tool, but more by enabling them to release the product to market ahead of
schedule!

A New Zealand based customer in the heath care domain embarked on a journey of migrating their Delphi-based products to Microsoft technologies. The products use specialized GUI controls that are not recognized by the typically popular tools. The company was keen to embark on automation right from the early stage of migration. And the budget to develop automation was tight.

We conducted Proof-Of-Concept (POC) to identify the tool that would support automation for both Delphi and VB.Net. We discovered that most popular tools were indeed not compatible with the developed product. The POC concluded that Test Complete did support both Delphi and VB.Net with a few constraints. It was very cost effective however but not user friendly. We convinced the management of our decision. The project started off with us identifying test cases which could be automated. Seven modules were automated and demonstrated.

We developed reusable Keyword Driven Framework for the client. Both individual test case execution and batch run was possible just by choosing the test cases. STAG provided detailed demo of the framework to the in-house QA team.

However some of the test cases chosen for automation were not complete. We validated the test cases, made the necessary changes and then initiated the scripting. The automation work was divided between the STAG and customers team. As we automated the test cases, we guided and trained the customer’s team to automate.

The result – By automating 326 test scenarios, the testing time was cut down from 80 hours to 12 hours! We saved the customer significant money spent on the tool, but more by enabling them to release the product to market ahead of schedule!

Tags: , , , , , , ,
Posted in Case-studies | No Comments »

It was more than just validation. It was ensuring nation’s pride.

Monday, August 9th, 2010
A large petroleum major was rolling out a specialized solution to ensure that fleet
tracking solution to ensure zero pilferage during transport. The solution consisted of a
plethora of technologies (GPS, GSM, Web, Mapping) and our role was to ensure that the
final solution is indeed risk-free for deployment.
With the launch date in the next few weeks, we got cracking on applying HBT to extract
the cleanliness criteria from the business and technical specifications outlined in the
tender document.The cleanliness criteria consisted of multiple aspects- deployment
environment correctness, cleanliness of software, clean working of hardware/software
interfaces and finally the ability to support a large load, volume with real-time
performance.
We identified the potential types of defects that spanned the entire spectrum of
hardware and software. The first step was to understand system development process
and identify senior consultants visited the vendor’s facility to assess the people and
processes used to develop the system. This provided a clear picture of what to expect
and the work that lay ahead of us.
Post our understanding of the development system, we developed a scientific strategy
and the evaluation scenarios. A variety of tests were identified – individual feature
validation, simulating various business use-cases, understanding load limitations and
performance evaluation of the system.
Now we were ready to validate the final system in the data center. The first cut of the
solution was used to develop a set of automated scripts for large scale load/stress/
performance testing. The system was populated with large data representing a real life
system. Vehicles were fitted with the vehicle mounted unit. We were ready to roll now.
The various vehicles were set in motion in various terrains, at various speeds and
mapping of the fleet on the India map was validated. We simulated large number of
vehicles with data arriving from the simulators at a high rate to ensure that
performance was indeed real-time.
In addition the deployment environment was validated, configurations checked, legality
of the software verified. We also verified that the solution integrates with customer’s
SAP database as well.
Bugs popped up and were fixed. We recommended changes in the system capacity,
pushed the vendor to close all the critical, high and medium priority defects before
providing a qualitative feedback on the solution and the potential risks. Once satisfied
our customer’s investment was safe, we gave a go-ahead to rollout the solution.

A large petroleum major was rolling out a specialized solution to ensure that fleet tracking solution to ensure zero pilferage during transport. The solution consisted of a plethora of technologies (GPS, GSM, Web, Mapping) and our role was to ensure that the final solution is indeed risk-free for deployment.

With the launch date in the next few weeks, we got cracking on applying HBT to extract the cleanliness criteria from the business and technical specifications outlined in the tender document.The cleanliness criteria consisted of multiple aspects- deployment environment correctness, cleanliness of software, clean working of hardware/software interfaces and finally the ability to support a large load, volume with real-time performance.

We identified the potential types of defects that spanned the entire spectrum of hardware and software. The first step was to understand system development process and identify senior consultants visited the vendor’s facility to assess the people and processes used to develop the system. This provided a clear picture of what to expect and the work that lay ahead of us.

Post our understanding of the development system, we developed a scientific strategy and the evaluation scenarios. A variety of tests were identified – individual feature validation, simulating various business use-cases, understanding load limitations and performance evaluation of the system.

Now we were ready to validate the final system in the data center. The first cut of the solution was used to develop a set of automated scripts for large scale load/stress/performance testing. The system was populated with large data representing a real life system. Vehicles were fitted with the vehicle mounted unit. We were ready to roll now.

The various vehicles were set in motion in various terrains, at various speeds and mapping of the fleet on the India map was validated. We simulated large number of vehicles with data arriving from the simulators at a high rate to ensure that performance was indeed real-time.

In addition, the deployment environment was validated, configurations checked, legality of the software verified. We also verified that the solution integrates with customer’s SAP database as well.

Bugs popped up and were fixed. We recommended changes in the system capacity, pushed the vendor to close all the critical, high and medium priority defects before providing a qualitative feedback on the solution and the potential risks. Once satisfied our customer’s investment was safe, we gave a go-ahead to rollout the solution.

Tags: , , , , ,
Posted in Case-studies | No Comments »

We demystified the automation puzzle. Relentless validation tamed!

Monday, June 28th, 2010

A large global provider of BI solutions has a product suite that runs on five platforms supporting thirteen languages with each platform suite requiring multiple machines to deliver the BI solution. The entire multi-platform suite is released on single CD multiple times a year.

The problem that stumped them was “how to automate the final-install validation of multi-platform distributed product”. They had automated the testing of the individual components using SilkTest, but they were challenged with “how to unify this and run off a central console on various platforms at the same time”.

Considering each platform-combination took about a day, this required approximate two months of final installation build validation, and by the time they were done with this release, the next release was waiting! This was a relentless exercise, consuming significant QA bandwidth and time, and did not allow the team to do things more interesting or important.

The senior Management wanted single-push-button automation -identify what platform combination to schedule next, allocate machine automatically from the server farm, install and configure automatically, fire the appropriate Silk scripts and monitor progress to significantly reduce time and cost by lowering QA bandwidth involved in this effort. After deep analysis, in-house QA team decided this was a fairly complex automation puzzle and required a specialist! This is when where we were brought in.

After an intense deep-dive lasting about four weeks, we came up with a custom master-slave based test infrastructure architecture that allowed a central console to schedule various jobs onto the slaves, utilizing a custom developed control & monitoring protocol.  The solution was built using Java-Swing, Perl, Expect and adapters to handle Silk scripts. Some parts of the solution where on Windows platform while some on UNIX.  This custom infrastructure allowed for scheduling parallel test runs, automatic allocation of machines from a server farm, installing appropriate components on appropriate machines, configuring them and finally monitoring the progress of validation through a web console.

This test infrastructure enabled a significant reduction of the multi-platform configuration validation. The effort reduced from eight weeks to three weeks. We enjoyed this work simply because it was indeed a boutique work fraught with quite a few challenges. We believe that this was possible because we analyzed the challenging problem from wearing a development hat and not the functional test automation hat.

Tags: , , ,
Posted in Case-studies, Hypothesis-based Testing | No Comments »

Delivering peace of mind- Assessing release worthiness

Monday, June 28th, 2010

The product helps to detect different types of telecom fraud, be it in wireless or wire line networks. It also helps to detect fraud in roaming, pre-paid and post-paid environments and tailor-made for GSM/CDMA/Fixed/GPRS network. The product team comprised of strong development team ably supported by an in-house QA team. The product was developed using J2EE technologies and had undergone multiple versions of build – currently in version 6.0 – with wide installation base in Asian/US market. The company had an ambitious plan to expand the product reach and move into a new market – European market. The product went through multiple feature upgrade/modifications to meet the needs of the new market. Though the product was tested diligently by the in-house QA team, the management was skeptical about the release worthiness of the product. They preferred to have an independent third party product assessment to enhance their delivery confidence before the formal product launch.

STAG singularly focused to ensure the defect escapes are minimized. Hence a three-pronged approach was adopted to determine the breadth and depth of testing required -

  • Identify what poses high business risks? What has been de-risked already? What remains as risk that is to be assessed?
  • How well has the “net” been cast to uncover defects in the lifecycle? Are the methods to uncover defects expansive/complete?
  • Are the test cases (i.e. those inputs that have the ‘power’ to detect anomalies) good?  Do the already existing test cases and therefore the tests conducted have the power to uncover high-risk business issues?

Fixing the high impact defects improved the stability of the product– which otherwise could have led to USD 250K support cost in the initial months. The release worthiness certificate lowered the business risk for the customer and newly gained delivery confidence by the customer management powered their successful product launch and on time to market.

Tags: , ,
Posted in Case-studies, STAG | No Comments »

Large scale migration of automation suite – “Scaling the peak”

Thursday, April 15th, 2010

The customer in focus provides data integration software and services that empower organizations to access, integrate, and trust all its information assets, giving organizations a competitive advantage in today’s global information economy. As the independent data integration leader, the customer has a proven track record of success helping the world’s leading companies leverage all their information assets to grow revenues, improve profitability, and increase customer loyalty.

The customer had a large base of automated test scripts (1300) in Rational Visual Test (VT) and a single license of the  Visual Test on a dedicated machine – a risky affair considering that the Visual Test is no more supported and relies on a older version of Windows. They decided to de-risk this to support the newer versions of products and also get rid of limitations existing in the existing test script suite. Some of the limitations of existing automation suite were: (1) Manual dependency to get the scripts run in suite against any new build released,(2) Non-existence of tool support for Visual Test (3) Limitations with respect to support on other flavors of windows operating system and support for internationalization (i18N) (4) Limitations in the framework to extend the scripts with new functional changes and (5) Difficulty in building competency on Visual Test. The management decided to migrate these VT scripts to Borland Silk Test Suite.

Now came the challenges that we had to solve. The test suite was large with only the script available and no documentation on the test cases. They were keen that the performance of the new suite in SilkTest be considerably enhanced. Technically the test suite was intensely data-driven with huge data set to drive the test suite. The automation run was l-o-n-g, ranging from 24-36 hours, necessitating that a robust recovery mechanism be in place to ensure uninterrupted runs with minimal baby-sitting. Since it was a new investment, the management was keen that the automation framework is indeed flexible so that new test cases could be added to the suite quickly. Finally, the suite had to cater to the different language packs of the product.

Phew – It was real challenge “scaling the peak”, and we had our share of issues of encountering bad weather, storms, landslides, but heck we made it! As always, the journey was arduous, but the rush of adrenalin after reaching the peak was great. We must confess that the journey would not been possible without the wholesome support and cooperation from the customer – Thank you.

Now the details of the climb! We had to analyze the large VT suite to understand the structure, flow, data inter-relationship and the finer nuances. Remember that we only had the scripts machine to try and understand! The key learning points and the action items were:

1. Build an effective data-driven mechanism to provide the flexibility to add and maintain test data in external SQL tables.

2. Implement robust delivery mechanism to enable the scripts to run uninterrupted for long durations, upwards of 24 hours.

3. Support for internationalization to enable testing of English and Japanese language packs via external language property files.

4. The ability to add new test cases to the existing framework to support new features and application changes with 50% less effort.

We  approached  the problem of scaling the peak, by firstly going through a strong intellectual process of technical problem analysis and  devising a library-based & data-driven framework and subsequently putting together a factory-driven approach to rapidly code the scripts.  Once we architected the custom framework, we  identified with the “good principles of development”,  such as  avoiding global variables, avoiding hardcoded information, level & depth of documentation, language coding conventions, and finally object referencing and de-referencing strategies. The development process was iterative with multiple milestones identified and acceptance criteria clearly identified.

A skeletal team of architects and specialists got cracking on the problem,  making the first move to built a flexible and robust automation framework. They also commenced development of common library components, that will be used to by the larger extended team later. Once the architecture custom framework was in place, coding standards were enforced, and then these activities of coding the framework and the library components were done. At this point, a larger  team was assembled, each of them was assigned a certain set of scripts  to be converted from VT to SilkTest. The act of coding the newer SilkTest scripts was individually done on developer machines, code-reviewed  and then later integrated on a test  machine,  and tested by running this on the target  application. We did encounter a stormy climb, with myriad integration issues  popping up,  each of them was solved and we continued to make good progress.

The D-day came and we were delighted to hoist the flag on the peak!  We had covered good ground,  generating approximately 50,000 LOC with about fifth of that constituting the framework level component code. The cool air at the peak was refreshing and sweet! -  We had  reduced the cycle time by approximately 80%  i.e.  from FIVE days to just ONE day,  were able to long runs of 24 hours without issues, switched language pack with ease and were able to add a new set of 120 test cases quickly enough. In the  subsequent two months of intense usage only about five issues were reported, which were  fixed.

This was a unique project, where  we had  to migrate automation code from one commercial tool to another with constraints of documentation and machine availability. It was indeed a pleasure to work with a demanding customer, who worked closely with us  to help us understand the product and the VT automation, and also making available a dedicated integration machine with the lone VT server machine.

We have always enjoyed challenges, and we thank the customer for giving us the opportunity and reposing trust in us. This is the fun-part of being a test boutique!

Tags: , , , , , ,
Posted in Case-studies, Hypothesis-based Testing, STAG | No Comments »

“The quality race” – STEM wins

Thursday, April 15th, 2010

A large German major with a mature QA practice is seeking new ways to improve its test practice. It all starts with a talk  delivered at this company on “The Science & Engineering of Effective Testing” to its senior management staff and test practitioners in the company. We are amazed at the interest in this topic (75+ people attend the talk) and the enthusiastic response – we are deeply humbled.

A few weeks after this, the management decides to experiment with STEM-based approach to testing. They identify about twenty five people (a small subset of their QA team) to be trained on the new way of testing. We are delighted and conduct a 5-day workshop with intense application orientation, to enable them to understand STEM. The company then decides to conduct a bold experiment- a pilot to evaluate STEM powered approach to testing vis-a-vis their way of testing. They identify a product that is in use for a few years with consumers across the world. They decide to have two five-member identical teams consisting of similar mix of experience levels of people, each given a timeframe of one month to evaluate the new release of this product. These two teams are  kept apart to ensure a controlled experiment and the countdown starts. We wait with bated breath…

The month is slow for us, but it flies for the two teams. Enormous data has been generated and the management analyses them thoroughly to spot the winner. A month later we are called by the senior management. We are sweating, have we won? A few minutes later, it is clear that STEM is a winner. The STEM powered team has designed 3x test cases compared to the non-STEM team and  uncovered 2x number of defects! The icing on the cake is that the couple of defects uncovered by the STEM powered team are “residual defects” i.e. they have been latent in the product for over a year (Remember Minesweeper game on Windows) and one of them corrupts the entire data in the database. Now the discussion steers to effort/time analysis – Does application of STEM require more effort/time? The team has conclusive evidence that it is not significant, implying STEM has enabled them to think better,  not work longer or more.

What enabled the STEM powered team to win the “Race of quality” ? The answers are given by the team itself, and we are delighted, as we have believed in them, and have seen results when we implement them. The top three reasons are: (1) The notion of Potential Defect Types (PDT) is powerful as it forces the team to hypothesize what can go wrong and enable them to setup a purposeful quality goal (2) PDT forces a thorough understanding of the customer expectations and the intended behavior of the product (3) PDT ensures that test design creates adequate test cases, thus eliminating defect escapes and paving the way for robust software.

The STAG team is delighted as their customer acknowledges the effectiveness of the STEM based approach. The team is convinced that STEM powered approach is a winner and is raring to run the marathon, with the customer also cheering them to win!

A heartfelt “Thank you” to the STEM powered team and the innovation-centered Senior Management of the company.

Tags: , , , , ,
Posted in Case-studies, Hypothesis-based Testing, STAG, STEM | No Comments »

10x reduction in post release defects – Finest experience of applying STEM

Thursday, April 15th, 2010

This is about one of the finest experiences that STAG had with a global chip major where the early implementation of STEM yielded significant results. Our engagement with them was to setup an effective validation practice for their porting level API for video decoders. The customer had a great technical team and were involved in both development and QA. The challenge that they faced with their complex product that involved both hardware and software and later system integration on multiple real time OS on various platforms, was that of high defect escapes i.e. post-release field defects.

We spent about a month understanding their domain and the associated technologies. Post this, a detailed analysis yielded interesting data – test cases were primarily conformance oriented, coverage of test cases was suspect, escaped defects seem to propagate from early stages and finally the process of validation was loose.

Having understood the types of defects that were being found and the post-release defects, we figured out the various types of probable defects and the various combinatorial aspects that need to be considered to form a test case. We then staged the validation as consisting of three major levels, the first one at API level, then the next one at a system level, and the last level made up of a customer-centric level that involved using reference applications.

Applying the STEM approach to test design, the test cases were developed, yielding about 6000 test cases at level one and about 800 at the subsequent levels. Also, whereas the ratio of +ve vs –ve test cases was earlier towards the +ve side, after our re-design, the ratio shifted to 60%:40% at the lower level and about 85% :15% at the higher levels. Moreover, the test cases increased in number significantly by a factor of 1000%, allowing for a larger and deeper net to catch many more serious defects. Over the next 9 months, the rate and number of defects detected increased dramatically, resulting in post-release issues reducing by a jaw-dropping 10x times.

Once we solved the test effectiveness problem and increased the yield of defects, the focus shifted to streamlining the process by setting up proper gating in the test process and creating a centralized web based test repository, and finally setting a strong defect analysis system based on Orthogonal Defect Classification (ODC) method. This enabled a strong feedback system, resulting in shifting the defect finding process to earlier stages of SDLC and thereby lowering cycle time. Complementing this, we focused on setting up a custom tooling framework for automating this non-UI based software resulting in a significant cycle time reduction – an entire cycle of tests on a platform took less than 15 hours of time!

This has been one of the finest experiences that we had with STEM, and was a clear winner for STEM implementation. This was only possible because of the very mature engineering management staff of the customer, who were focused on systemic improvement and had systematic improvement goals.

What is to be noted is that our test team was NOT a team with significant depth of experience on the particular product domain. Applying STEM at a personal level, the team was able to understand what was necessary and sufficient for effective validation and complemented the strong technical team with mature defect-oriented thinking. This was an early case study for us to establish that a STEM based approach provided us with the right thinking skills for defect finding, rather than resort to a domain centric approach to defect finding.

Tags: , , , , , ,
Posted in Case-studies, Hypothesis-based Testing, STAG | No Comments »

Re-architecting test assets increases test coverage by 250%

Thursday, April 15th, 2010

This company is an innovative online banking solution provider, having three major products catering to over 100 top financial institutions(FI) of the world including the top five FI in the world. They have a very successful product line, growing rapidly, with major releases approximately every year, incorporating new features to cater to the various needs of the market place. As the code base evolved, the test assets were also modified to reflect the changed product. The challenge faced was that the most of the test cases were passing and the rate of uncovering new defects was low.

The product became huge and the company decided to re-architect the product in order to enable rapid feature addition with low risk. That is when the company decided to re-look at its test assets and re-architect the same to increase the test coverage, improve defect finding ability and ensure that the test assets were future-proof. It had about 8000 test cases then.

We were chartered to analyze the existing test cases for completeness and modifiability and re-architect the same after filling the gaps and to ensure that the future test cases were easily pluggable. Applying STEM, we performed a thorough assessment of the existing test assets and discovered holes in the same. Using the STEM Test Case Architecture (STEM-TCA), we re-engineered the test cases by firstly grouping them into features, then by levels of tests and segregating into various types of tests and then finally by separating into positive and negative test cases. During this process of fitment of existing of test cases into the STEM-TCA, we uncovered quite a few holes. These were filled by STAG by designing 5000 test cases additionally. Not only did the STEM-TCA increase the test coverage by uncovering the missing test cases, it also provided a sharper visibility of the quality as the test cases were well organized by specific defect types. This improved the test coverage by about 250% and the technical management staff were confident about the adequacy of test assets and were also convinced about its future upgradeability and maintainability.

Tags: , , , , ,
Posted in Case-studies, STAG, STEM | No Comments »