Test Management

The Test Management Service consists of  

• Test Scope Specification (What)
• Test Strategy Definition (How and When)
• Test Strategy Execution, Monitoring and Control
• Test Completion (Handover)

Test Management interfaces with the following processes, partly controlling them and executing replanning if changes are required:

• Project or programme management
• System specification and design
• Change management
• System implementation (e.g. by third-party project partners)
• Defect management
• Risk management
• Operative testing processes (by developers, independent testers, users, and operations and support)
• Test automation
• Test environment management and test data management
• Operations and support

Test Case

What's a 'test case'?
A test case describes an input, action, or event and an expected response, to determine if a feature of a software application is working correctly

What is the 'software life cycle'?

"The full business, from initial thinking to final use, is called the product's life cycle."
The life cycle begins when an application is first conceived and ends when it is no longer in use. 
It includes aspects such as 
initial concept, 
requirements analysis, 
functional design,
internal design,
documentation planning, 
test planning, 
coding, 
document preparation, 
integration,
testing,
maintenance, 
updates,
retesting,
phase-out, and other aspects.

CMMI

Level 5 - the focus is on continuous process improvement. 

The impact of new processes and technologies can be
predicted and effectively implemented when required.

software quality and Good code

What is software 'quality'?
Quality software is reasonably bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable. 

What is 'good code'?
'Good code' is code that works, is reasonably bug free, and is readable and maintainable. 

Types or kinds of Testing

What kinds of testing should be considered?

• Black box testing - not based on any knowledge of internal design or code. Tests are based on requirements and functionality.
• White box testing - based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions.
• unit testing - the most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses.
• incremental integration testing - continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers.
• integration testing - testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.
• functional testing - black-box type testing geared to functional requirements of an application; this type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.)
• system testing - black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.
• end-to-end testing - similar to system testing; the 'macro' end of the test scale; involves testing of a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate.
• sanity testing or smoke testing - typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or corrupting databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state.
• regression testing - re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing approaches can be especially useful for this type of testing.
• acceptance testing - final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time.
• load testing - testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails.
• stress testing - term often used interchangeably with 'load' and 'performance' testing. Also used to describe such tests as system functional testing while under unusually heavy loads, heavy repetition of certain actions or inputs, input of large numerical values, large complex queries to a database system, etc.
• performance testing - term often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.
• usability testing - testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.
• install/uninstall testing - testing of full, partial, or upgrade install/uninstall processes.
• recovery testing - testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.
• failover testing - typically used interchangeably with 'recovery testing'
• security testing - testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques.
• compatibility testing - testing how well software performs in a particular hardware/software/operating system/network/etc. environment.
• exploratory testing - often taken to mean a creative, informal software test that is not based on formal test plans or test cases; testers may be learning the software as they test it.
• ad-hoc testing - similar to exploratory testing, but often taken to mean that the testers have significant understanding of the software before testing it.
• context-driven testing - testing driven by an understanding of the environment, culture, and intended use of software. For example, the testing approach for life-critical medical equipment software would be completely different than that for a low-cost computer game.
• user acceptance testing - determining if software is satisfactory to an end-user or customer.
• comparison testing - comparing software weaknesses and strengths to competing products.
• alpha testing - testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.
• beta testing - testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.
• mutation testing - a method for determining if a set of test data or test cases is useful, by deliberately introducing various code changes ('bugs') and retesting with the original test data/cases to determine if the 'bugs' are detected. Proper implementation requires large computational resources.

verification and validation

What is verification? validation?

Verification typically involves reviews and meetings to evaluate documents, plans, code, requirements, and specifications.( This can be done with checklists, issues lists, walkthroughs, and inspection meetings)


Validation typically involves actual testing and takes place after verifications are completed.

STLC

Contrary to popular belief, Software Testing is not a just a single activity. It consists of series of activities carried out methodologically to help certify your software product. These activities (stages) constitute the Software Testing Life Cycle (STLC).


The different stages in Software Test Life Cycle -

Each of these stages have a definite Entry and Exit criteria  , Activities & Deliverables associated with it.
In an Ideal world you will not enter the next stage until the exit criteria for the previous stage is met. But practically this is not always possible. So for this tutorial , we will focus of activities and deliverables for the different stages in STLC. Lets look into them in detail.

Requirement Analysis

During this phase, test team studies the requirements from a testing point of view to identify the testable requirements. The QA team may interact with various stakeholders (Client, Business Analyst, Technical Leads, System Architects etc) to understand the requirements in detail. Requirements could be either Functional (defining what the software must do) or Non Functional (defining system performance /security availability ) .Automation feasibility for the given testing project is also done in this stage.

Activities

• Identify types of tests to be performed. 
• Gather details about testing priorities and focus.
• Prepare Requirement Traceability Matrix (RTM).
• Identify test environment details where testing is supposed to be carried out. 
• Automation feasibility analysis (if required).

Deliverables 

• RTM
• Automation feasibility report. (if applicable)

Test Planning

This phase is also called Test Strategy phase. Typically , in this stage, a Senior QA manager will determine effort and cost estimates for the project and would prepare and finalize the Test Plan.

Activities

• Preparation of test plan/strategy document for various types of testing
• Test tool selection 
• Test effort estimation 
• Resource planning and determining roles and responsibilities.
• Training requirement

Deliverables 

• Test plan /strategy document.
• Effort estimation document.

Test Case Development

This phase involves creation, verification and rework of test cases & test scripts. Test data , is identified/created and is reviewed and then reworked as well.

Activities

• Create test cases, automation scripts (if applicable)
• Review and baseline test cases and scripts 
• Create test data (If Test Environment is available)

Deliverables 

• Test cases/scripts 
• Test data

Test Environment Setup

Test environment decides the software and hardware conditions under which a work product is tested. Test environment set-up is one of the critical aspects of testing process and can be done in parallel with Test Case Development Stage. Test team may not be involved in this activity if the customer/development team provides the test environment in which case the test team is required to do a readiness check (smoke testing) of the given environment.

Activities 

• Understand the required architecture, environment set-up and prepare hardware and software requirement list for the Test Environment. 
• Setup test Environment and test data 
• Perform smoke test on the build

Deliverables 

• Environment ready with test data set up 
• Smoke Test Results.

Test Execution

 During this phase test team will carry out the testing based on the test plans and the test cases prepared. Bugs will be reported back to the development team for correction and retesting will be performed.

Activities 

• Execute tests as per plan
• Document test results, and log defects for failed cases 
• Map defects to test cases in RTM 
• Retest the defect fixes 
• Track the defects to closure

Deliverables 

• Completed RTM with execution status 
• Test cases updated with results 
• Defect reports

Test Cycle Closure

Testing team will meet , discuss and analyze testing artifacts to identify strategies that have to be implemented in future, taking lessons from the current test cycle. The idea is to remove the process bottlenecks for future test cycles and share best practices for any similar projects in future.

Activities

• Evaluate cycle completion criteria based on Time,Test coverage,Cost,Software,Critical Business Objectives , Quality
• Prepare test metrics based on the above parameters. 
• Document the learning out of the project 
• Prepare Test closure report 
• Qualitative and quantitative reporting of quality of the work product to the customer. 
• Test result analysis to find out the defect distribution by type and severity.

Deliverables 

• Test Closure report 
• Test metrics

Finally, summary of STLC along with Entry and Exit Criteria

STLC Stage	Entry Criteria	Activity	Exit Criteria	Deliverables
Requirement Analysis	Requirements Document available (both functional and non functional) Acceptance criteria defined. Application architectural document available.	Analyse business functionality to know the business modules and module specific functionalities. Identify all transactions in the modules. Identify all the user profiles. Gather user interface/authentication, geographic spread requirements. Identify types of tests to be performed. Gather details about testing priorities and focus. Prepare Requirement Traceability Matrix (RTM). Identify test environment details where testing is supposed to be carried out. Automation feasibility analysis (if required).	Signed off RTM Test automation feasibility report signed off by the client	RTM Automation feasibility report (if applicable)
Test Planning	Requirements Documents Requirement Traceability matrix. Test automation feasibility document.	Analyze various testing approaches available Finalize on the best suited approach Preparation of test plan/strategy document for various types of testing Test tool selection Test effort estimation Resource planning and determining roles and responsibilities.	Approved test plan/strategy document. Effort estimation document signed off.	Test plan/strategy document. Effort estimation document.
Test case development	Requirements Documents RTM and test plan Automation analysis report	Create test cases, automation scripts (where applicable) Review and baseline test cases and scripts Create test data	Reviewed and signed test Cases/scripts Reviewed and signed test data	Test cases/scripts Test data
Test Environment setup	System Design and architecture documents are available Environment set-up plan is available	Understand the required architecture, environment set-up Prepare hardware and software requirement list Finalize connectivity requirements Prepare environment setup checklist Setup test Environment and test data Perform smoke test on the build Accept/reject the build depending on smoke test result	Environment setup is working as per the plan and checklist Test data setup is complete Smoke test is successful	Environment ready with test data set up Smoke Test Results.
Test Execution	Baselined RTM, Test Plan , Test case/scripts are available Test environment is ready Test data set up is done Unit/Integration test report for the build to be tested is available	Execute tests as per plan Document test results, and log defects for failed cases Update test plans/test cases, if necessary Map defects to test cases in RTM Retest the defect fixes Regression testing of application Track the defects to closure	All tests planned are executed Defects logged and tracked to closure	Completed RTM with execution status Test cases updated with results Defect reports
Test Cycle closure	Testing has been completed Test results are available Defect logs are available	Evaluate cycle completion criteria based on - Time, Test coverage , Cost , Software Quality , Critical Business Objectives Prepare test metrics based on the above parameters. Document the learning out of the project Prepare Test closure report Qualitative and quantitative reporting of quality of the work product to the customer. Test result analysis to find out the defect distribution by type and severity

Glossary of software testing

http://www.astqb.org/educational-resources/glossary.php

what is Black Box Testing?

Black box testing is a software testing techniques in which
functionality of the software under test (SUT) is tested without looking
at the internal code structure

In Black Box Testing we just focus on inputs and output of the software system without bothering about internal knowledge of the software program.
 

The above Black Box can be any software system you want to test. For example : an operating system like Windows, a website like Google ,a database like Oracle or even your own custom application. Under Black Box Testing , you can test these applications by just focusing on the inputs and outputs without knowing their internal code implementation.

Black Box Testing and Software Development Life Cycle (SDLC)

Black box testing has its own life cycle called Software Test Life Cycle (STLC) and it is relative to every stage of Software Development Life Cycle.

• Requirement – This is the initial stage of SDLC and in this stage requirement is gathered. Software testers also take part in this stage.
• Test Planning  & Analysis – Testing Types applicable to the project are determined. A Test Plan is created which determines possible project risks and their mitigation.
• Design – In this stage Test cases/scripts  are created on the basis of software requirement documents
• Test Execution- In this stage Test Cases prepared are executed. Bugs if any are fixed and re-tested.

What is Software testing and QA?

Software Testing:: Testing is the process of identifying defects, where a defect is any variance between actual and expected results.



Testing means identifying deviations from a specified target at the earliest possible stage.

In simple words, software testing is an activity to check whether the actual results match the expected results and to ensure that the software system is defect free. 

Testing involves operation of a system or application under controlled conditions and evaluating the results ..(eg, 'if the user is in interface A of the application while using hardware B, and does C, then D should happen').

It is oriented to 'detection'

A test engineer typically has the perspective of 'what might go wrong with this functionality, and how can we ensure it meets expectations?'.

testing is important because software bugs could be expensive 

Early Testing - Testing should start as early as possible in the Software Development Life Cycle

Testing is context dependent which basically means that the way you test a e-commerce site will be different from the way you test a commercial off the shelf application

What is 'Software Quality Assurance'?
Software QA involves the entire software development PROCESS - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to 'prevention'.