Overview of Model Driven Testing

Model Driven Testing (MDT) is an emerging second generation test automation technology that creates graphical test models of the behaviour of an application under test. A MDT test generation tool then uses the models to create thousands of test cases which are then automatically sent to the application to be tested.  The response from the application is then automatically analysed for success or failure.

Model Driven Testing creates a significantly large ROI because effort is not spent on the creation and maintenance of expensive and labour-intensive test scripts.  Applications are guaranteed to be tested to the required levels of test coverage and it is also very easy to maintain the test models and to update it to reflect further changes to systems requirements and specifications.

To fully understand the advantages of Model Driven Testing it is important to understand the current approaches to testing and the resultant costs.

Current costs for software defect correction
A major concern is to ensure that all software developed will function according to the systems specifications. As a result, between 30% - 50% of the typical development budget is spent on application and system testing.  One of the key problems of testing is that the costs of detecting and correcting defects are significantly increased, the longer they remain in the development lifecycle, as shown in Figure 2.

It costs about $14,000 to find and correct a software defect once the application is in production whereas it cost only $455 if the defect is detected and fixed during the software design stage.

The failure of current test automation approaches
A significant percentage of current test automation is based on testing GUI interfaces, which is fine if you are only testing software GUI clients.  However the GUI interface is only the visible 20% tip of the testing “iceberg” which is 80% server oriented and cannot be effectively tested by simply testing the GUI.

Currently, the only other approach employed to automate testing is for test engineers to manually create and maintain the test scripts which are then passed to a test automation tool for automated test execution. 

However whilst this approach will reduce some of the cost of testing, it has little effect on the over all cost of defects.    

This is because around 70% of defects are caused by errors during the phase when the requirements are translated into design documents – see Figure 2.  Whereas 60% of defects are detected during user acceptance and 21% are detected during production – the two most costly phases in which to detect and correct defects.  The fundamental testing problem is that there is a miss-match between where defects are created and when they are detected and corrected.

Model Driven Testing - The solution to the testing mismatch
Model Driven Testing eliminates the traditional mismatch between where defects are created and where they are detected, by means of the following process (see Figure 3):

• The test designer starts to create the graphical test model based on the User Requirements documentation.  At the same time the System Designers start work on creating the various design documents from the User Requirements documentation.
• The test designer uses the initial design documents to further refine the test model. However during this process he will detect specification ambiguities, defects and hidden assumptions in both the design and user requirements documentation.
• The next stage is to discuss these problems with the stakeholders, to get clarification so that all of the user requirements, design and test models can be updated.  This process works because a MDT test model is executable and therefore it must have precise specifications free of any hidden assumptions and ambiguities.

Once the test model has been created it can then be used to automatically create test cases for both Unit and System Acceptance testing without the need to manually create and maintain test scripts.

Benefits from Model Driven Testing
Model Driven testing creates significant ROI because:

• The modelling process forces a rigorous examination of the logical consistency of user requirements – the largest root cause of defects
• Automatic design of test scripts directly from the model removes the requirement for manual test case design – removing a means by which errors can be introduced
• The automated test scripts always provide full functional coverage. This cannot be guaranteed when the scripts are designed by individual test analysts with varying levels of experience
• The process significantly shortens the feedback loop from requirements to test. This has a positive impact on time and cost to fix problems

Vanguard’s Model Driven Testing Solutions
Vanguard Technologies has partnered with the leading Model Driven Testing vendors to provide our clients with a cost effective testing solution.   Contact sales@vanguard-technologies.com for additional information.