The evolution of quality assurance and how Azure Test Plans is driving the future of manual testing
The Deming Prize from Japan is the highest honour in Total Quality Management in the world. Japanese products are known for their superior quality, but it wasn’t always the case. In the 1950s fresh out of defeat in the World War 2, Japan was desperately trying to shift manufacturing from military to civilian products for trade. However the poor quality of the products deterred international buyers. Enter W. Edwards Deming who realised companies should invest in quality assurance (QA) methods that reduce the workload and save costs. This marked the beginning of the iteration development in manufacturing and has now progressed into software development. Emphasis for success is in doing short sprints and controlling the quality at every iteration.
Quality assurance is often seen as a dry topic. The software industry is no different. We understand its importance but wished we never have to deal with it ourselves. Developers love to work on design, write beautiful code that will change the course of humanity; but are also required to spent effort on time consuming tasks such as testing and bug fixing. Microsoft with its mission of empowering every person and every organization on the planet to achieve more, is building tools to improve the experience of developers at every step of the cycle. Azure DevOps (henceforth ADO) covers all major aspects of development lifecycle in a single offering that encapsulates planning, design, documentation, quality assurance to CI/CD release pipelines. Azure Test Plans allows customers to create and run manual and exploratory tests without ever leaving the ADO boundaries.
One solution for all your DevOps needs:
Manual testing remains relevant in today’s software development lifecycle as it is less costly to implement and provides quicker feedback. An important starting point can be user stories which reside in a Kanban board similar to Azure Boards within ADO. Users can create test cases, note defects, and report bugs directly against these user stories hence removing effort duplication and maintaining traceability of the items. Teams can define and track their requirements, deploy the corresponding code and test it within ADO; whereby it can be used as a single source of truth by all major stakeholders.
Business teams create functional requirements and developers link these requirements to user stories and deploy the code into Azure Repos using a CI/CD pipeline in Azure Pipelines. With QA teams manually testing the product as a final step before rollout means all stakeholders use the ADO project as a single source of truth.
Deep Dive into Azure Test Plans:
Azure Test Plans offers incredibly impactful features for the testers that enable them to go beyond marking a test case as pass or failed. With the Run Test option, users can additionally capture screenshots, recordings, and provide comments when a case fails. Managers can then view these results in the form of interactive charts and graphs to gauge the overall performance of their team.
Customers drive the maximum benefit of Azure Test Plans when they use the entire ADO ecosystem; but Azure Test Plans is sold as a separate offering for stand-alone customers as well. Many customers generally maintain their test cases in a spreadsheet and need a simple process to migrate these into Azure Test Plans. The ‘Grid View’ functionality allows users to simply paste their existing test cases from a spreadsheet in Azure Test Plans and bulk create corresponding test cases with a single click. Additionally, Azure Test Plans also provides a Comma-Separated Values (CSV) import functionality.
 |
 |
|---|---|
| View all your Test Plans in one place and sort them by various filters. | Capture details of each Test case and mark them as Pass/Fail. |
Testing becomes a team sport with exploratory testing whereby multiple stakeholders can collaborate on the end product to discover and solve issues faster. Azure Test Plans also offers this capability through extensions available on all major browsers including Microsoft Edge, Google Chrome and Mozilla Firefox. Customers can report bugs directly from these extensions that can later be viewed in a single dashboard along with relevant metrics related to the test session.
Conclusion:
Manual and exploratory testing compliment automated testing features of ADO to encapsulate total quality management. Azure Test Plans along with automated test offerings from ADO such as code coverage runs during pipeline deployments ensure customers can safely build high quality products through iteration development much like the way Japanese modelled their industrial revolution in the second half of the 20th century.
Want to learn more? See our documented best practices, videos, and other learning materials for Azure Test Plans. Or dive deep into Azure Test Plans.
Light
Dark
8 comments
The pricing for Test Plans is out of step with the pricing for other ADO features. In modern software development where testing may be undertaken by non-dedicated QA the price for casual use makes the Test Plans feature value proposition fall well short and inevitably teams will be left using their spreadsheets to manage their test cases
Agreed. $46 (extra) per person per month is expensive for something that is effectively built into the tool already. Not sure but I suspect the artifacts it stores are being stored with all the other artifacts so you also likely have the artifact storage costs if you’ve exceeded the limit (which we have).
I have tried to get our (dedicated) QA team to look into Test Plans since we prefer built in functionality but each time they look at the cost, compared to say Selenium which is free or using Excel spreadsheets (which is already paid for), and it is a no-go. They said that if they are going to pay for a product then they are going to look around at other options first. If it were free then it would be an easy win for MS.
Not debating the license cost, but this may help minimize how many licenses you need: the only people that actually need the Test Plan license are the people that own creating the plans, managing test suites, configurations …etc. So often this may be just your dedicated QA team (or dev leads). Creating test cases and executing test cases that are already part of a plan, can be done by anyone with a Basic license.
Hello Chad,
I’m the Product manager for Azure Test plans and thank you for providing feedback on our pricing model. Azure DevOps tries to simplify the lives of developers and teams using our products at scale. Our Basic license which is provided at a modest price of $6/user/month intends to cover majority of the users in the DevOps spectrum and offers most of Azure DevOps (ADO) products. Azure Test plan is sold as a separate license and comparatively higher cost since it is not something most of ADO customers need for daily use.
I appreciate you bringing up the concern of ‘light-weight’ users who do not use the product regularly. We will review this request internally.
Regards
Ravi
The test hub was updated a while back and the UX not as useful as it used to be.
1. The old hub supported creating stand-alone test suites without first creating a plan so you could group all your test cases first then add them to plans as needed. E.g. I want a test suite that groups all my mobile test cases which I add to future test passes (plans) but I don’t want to have to create a plan first or continually copy from an old plan. The old UX used a tree view which was simple to work with.
2. When creating a plan or suite with sub suites, they do not role up to the top level to see all test cases. This was lost with the new hub.
Hello Tom,
Thank you for providing feedback on our Test Hub experience. The request to create Test plans first and then populate it with Test suites/cases was implemented a few years back with an objective to simplify the user journeys of creating and managing large test plans. Many of our customers today have more than 10,000+ test cases in their organisations spread across thousands of Test suites. Hence, creating Test plans followed by Test suites brings in organised structure to managing such large number of test cases later.
You highlighted the need to create Test cases before assigning them to Test plans. While this cannot be achieved from within the Test hub today, you can always go to Azure Boards and create new Work Items (with type= Test case). These work items can then be assigned to Test plans from the Test hub.
Regards
Ravi
The one thing that really bothers me about Test Mgmt is that you cannot have a good visual display of the details of the user story, and the Test Case Entry screen side-by-side. Seems that this would be natural and something easy to do. Hopefully, someone will point out that I’m wrong and point me in that direction.
Hi Chris,
Thank you for the query. We would be happy to navigate you in the right direction. I would like to make sure we have understood the concern properly: are you looking for a single UX to view Test case definition AND the Test case results?
Or are you looking for a single UX view for Work item (from Azure Boards) and a Test case definition?
Regards
Ravi