Very often in discussions with senior technical folks, the topic of developer testing and early stage quality pops up. And it is always about ‘we do not do good enough developer testing’ and how it has increased post release support. And they are keen on knowing ‘how to make developers test better and diligently’ and outlining their solution approach via automation and stricter process. The philosophy is always about “more early testing” which typically has been harder to implement. Should we really test more?
This article published by T Ashok in Tea-time with Testers Jan-Feb 2015, suggests “frictionless testing” to enable a developer to do less and therefore deliver great early stage code.
An user story is seen as a modern way of communicating the end user’s needs and expectations in a sweet and simple format that can be easily modified. This brevity/simplicity hides information leading to understanding in the small and potentially missing the big picture.
This Tri-webinar series consisting of three webinars commencing with ‘How-to question to understand an user story and identify gaps’, moving onto ‘How-to set clear baseline’ to ensure an effective strategy, and finally culminating with ‘How-to design test scenarios/cases’ using a scientific and disciplined approach.
STAG presented a webinar on Feb 19, 2015 which outlines part#1 of this webinar ‘How-to question to understand an user story and identify gaps’
You can view / download the webinar recording here
You can view the webinar slidedeck below:
You can see the Q&A session of this webinar below. Click on each question to see the answer.
1. Do you have any template for scenario matrix?
I don’t really understand what you mean by Scenario matrix. If you meant by having a test scenario/case template, yes it is there. But it is more than just following a template. It is about a deeper understanding of the user story the conditions that govern its behaviour and its collbaoration with other user. Our intention is to do less documentation. A simple one liner scenario will Ensure that the user story does/does-not …. is more often than not indeed sufficient to spell out the test scenario.
2. How do we make sure that scenarios design do indeed cover the entire user story?
Understand that a user story implements a set of actions. And the actions are given by a business logic which really is a set of conditions. What we are trying to do is ascertain and understand those various conditions that actually form the behavior. Combinations of these conditions become scenarios which you want to evaluate.
So once you understand a user story well, list down the conditions that govern the behavior and combine these to form the test scenarios. Note that if the conditions are clearly listed, then a logical combination of these can be said to be complete, no guess work.
3. What do you mean by Potential Defect Types?
Well, our goal is not to test by just checking each action for correctness. Set a goal of what we are looking for i.e what categories or types of defects are we going to look for. And then come with test scenarios/cases to uncover these 'Potential Defect Types'
4. Hypothesis based testing - how is it different from conventional testing?
In conventional testing, focus is on “Experience to perform Activities” to uncover good issues and to deliver a great software, whereas in HBT, the focus is on “Expertise to reach a Goal”
Typically conventional testing is intensely activity driven, HBT is intensely goal driven. Conventional testing is strongly based on experience whereas HBT is strongly based on expertise in applying scientific problem solving techniques.
5. Would appreciate your views on how HBT is distinct and more effective.
In HBT, our focus is on a good application of scientific thinking to ensure that we have better understanding on clarity and a cleaner appreciation of various behavioral conditions and therefore, having a far superior coverage in terms of test scenarios and test cases. Therefore we are far more effective.
6. Is Information needed for a good understanding is same in a context of user stories (agile) and waterfall dev process?
What we have to appreciate is: the information needed for understanding, the kind of questions to ask is same irrespective of the process, method and styles.
But when you look at a very big system or when you look at a waterfall model for large requirements documents, then this process of questioning becomes far more elongated/big in some sense, whereas the user story being small, it is much more rapid.
But the information required for good understanding as outlined in terms of Landscaping and Viewpoints are absolutely meaningful whether you follow a waterfall development process or Agile. When we write small user stories, it is easier to ask questions to understand what-you-know/what-you-don’t -know rapidly
7. How is end-to-end testing different from testing an user story?
Think of end-to-end testing as testing a collaboration of user stories. The real life usage is not about just using one user story at a time; it is about using a set of user stories to accomplish a bigger task, which is more really like a flow. So it is not only sufficient to test “a user story” well enough but is equally important that test these user stories in a bigger flow. This is what end to end testing is. So end-to-end testing is deferent from user story testing because they ensure that all the user stories are taken together and therefore indeed validated the real time usage flows.
8. Can you please share a real time use story and mapping to these questions?
Love to do it, but in short webinar it’s a bit of challenge.
But will share you a real time experience we have faced in with a client. The user story under consideration was one that generates logs to help a support person for diagnose a system remotely.
The application enables a support staff to login to the system, look up information in the generated log to diagnose the problem and then do the necessary fix.
We discovered during the user story understanding phase, that the support person will have to login and sift through a very large text file using an editor to troubleshoot. And this is hard especially when the system has faulted and the customer is screaming at you! So the implementation of user story though correct in terms of functional behavior, will never satisfy the end user as it could take a long time to find out the issue in the log by walking through the voluminous text!
This is the example of how probing into the user persona and the the use-situation, we uncovered that this would pose a serious challenge! What we build would work correctly but would be terribly useless to the end user because we never understood the situation of the actual end user.
9. How do we handle, if the intended expectation in the early stage changes after the implementation?
What we are trying to do is appreciate and understand the expectations as deeply and clearly as possible, but you see an end user is a human being, so when I see something, I will become greedy and I will change, so quite naturally I have may higher expectations or change expectations after the implemention. And I think it is perfectly ok. I don't think we should look at the notion of ‘freezing spec’ at any point of time. The whole idea is to be responsive in Agile, and that means respond to changes be in the early or late post implemnantion.
10. Can we use the same criteria as a question list for user stories and a check list for standard development?
Yes and No. Please keep in mind, that whenever we have question/check list, sometimes it becomes a stock activity that we do it irrespective of whether it is useful or not. Yes please have certain guideline, certain questions like what, when... but feel free to improvise constantly because we want to be in a responsive agile way of thinking. So the checklist/question list is useful tool, but let that not limit your thinking.
11. How do we track the test cases that involve 2-3 user stories?
We have test cases connected to a user story,and we may have set of test cases which connect various user stories together, that you may call as a flow. You will have two sets of test cases one set map to user stories and second will connect on to the flows. We will talk/discuss about in subsequent webinar.
12. Where will we use HBT in real world?
As an organization we have used HBT across wide variety of softwares which we have tested with various companies. We use it for testing, for implementing automation, for coming with better measures and metrics etc. We can use HBT in any kind of process/ model/ technology/ domain.
13. Is fish bone analysis included in HBT?
By itself HBT does not included fishbone analysis. There are lots of other interesting QC techniques that are available in the industry, use as applicable if you like and HBT complements such techniques.
14. I feel HBT needs practitioners with more expertise compared to conventional testing, what is your view?
Remember we have used words called EXPERIENCE and EXPERTISE. Experience comes from time and mistakes whilst expertise comes from deeper application of scientific thinking.
HBT primarily helps to enable an individual to acquire a deeper expertise and therefore approach in testing is quite different from conventional testing.
15. How much domain knowledge / skill are required for HBT approaches? Nothing? Isn’t it total necessary to check the source codes of the users' programs/application programs?
Yes and No. With respect to source code, it depends. If you are testing really in the small, appreciating source code useful. And if you are testing behavior in the large, then source code may not be necessary. The whole idea of HBT is to provide you the set of techniques, principles and guidelines so that you are not dependent on experience as the whole.
16. Is it advisable to start testing as early as development in Agile methodology?
Absolutely Yes! We should be “lean” and avoid testing. Because the whole idea is not to feel happy about finding bugs after testing, because that goes against the fundamental notion of “Reduce waste” of Lean Thinking. In software context, waste can be treated as issues, so the idea of deeper questioning is to find faults in the specification, in the design that we can prevent, so that we need not have to test it out.
17. User story comes into picture only for black box testing., can it be used for white box testing?
Black box and white box are set of techniques which help us to design test cases. Unfortunately we call them testing and confuse the matter!
We understand the user story both in terms of business logic and behavior, the black (external piece) and the structure of the implementation (possibly white). The intention is evaluating the specification of the design and it is always the meaningful mixture of external specification and internal structure. So can it be used? YES.
Suggestion - please use the phrase Black/White box techniques and not as testing!
18.As fish bone analysis and 6 thinking hats is not included in HBT, does it mean that user story or HBT is not enough for wholesome testing, implementation or to uncover issues but it’s a way to have best start for testing or implementation?
Remember what we are trying to do is to compliment and put together certain test of techniques that could be useful in software or validation.
There are lots of interesting techniques available in quality control, quality assurance tool. So I would not really want to tell that they are exclusions, but think that there are so many things available, feel free to use it. So just because there is something nice outside I don’t want to add it into the baggage of HBT and make it much more complicated than what we are trying to do. They are compliment and benefit as and when you apply. So the focus on HBT is to enable/equip you with good set of techniques from a software quality perspective which allows you to question better, design better and test effectively and allow you to monitor that you are doing good enough job.
19. Why should we focus on non functional attributes in ever user story?
Well, sadly you don't want to find out that after implementing the user story integrating with other stories, it is slow, insecure or doesn’t not handle load well. You have to probably redesign some to fix those kinds of issues and this normally takes more effort in later stage.
You don’t want to be painfully surprised later, hence do focus on appreciate on the attributes and how they matter in the context of the smaller user story too.
An user story fosters sharp focus and by virtue of being small enables rapid implementation. But on the flip side – this extreme focus may lead to missing the big picture. Would we get into the “Seven blind men and elephant” situation?
Hmmm, What do you do? – Well, ask questions to understand the big picture. This is the focus of our first webinar “Question to Understand” on the topic “How to test an user story” on Feb 19, 2015. (To attend, register here)
You may find this article interesting.