Getting it right from the beginning

All software is essentially written the same way regardless of the process; no matter if it is waterfall, V model or Agile. That is Idea -> Build -> Test. No matter how you dress it up; until the software or a model is available to run it can’t be tested or validated. While V model and waterfall try to do the whole project in one go and Agile loops round adding functionality on each loop the underlying process is still Idea -> Build -> Test.

CASE or Model-based design tools can do a lot of the checking and verification of the model, but this occurs part of the way though the design stage; after a lot of effort has been expended. Even here the software engineer(s) might be building to incorrect or flawed specifications, even if they ‘look right’. In complex systems, for example aircraft or rail systems, it is often not until testing the actual system in unusual situations, that flaws emerge.
But what if were possible to validate the requirements before even starting the design or the model? This changes the fundamental flow to Idea -> Validate -> Design -> (Test ->) Implement -> Test. Validating each phase eliminates more problems and does so earlier. We are stopping the Garbage In at the initial phase.
For V model this is clearly a ‘Good Idea’ For Agile methods where the loop is repeated far more often it will save smaller amounts of time and effort but far more often. The net result is the same; you know you are building something coherent before you start. For Agile where new requirements are continually added, any conflicts will be seen before starting each sprint, which will greatly enhance the process.
So, how to test or validate requirements? A common approach uses a requirements management program, which orders, sorts and enters requirements into a database. This helps, as organising the requirements can highlight some problems, deficiencies and conflicts. However it requires people to notice them, and, as many of us know to our cost, requirements tend to be less than easy to read. It is difficult to retain accurately multiple requirements in your head, much less juggle all the ‘what ifs’.
Another approach is to use formal methods to validate the requirements. However, there are very few humans who are good with both formal methods and communicating with teams, particularly with the teams who produce ‘requirements’; the translation of text to formal methods has always been a problem. What is needed is something that will let the average system specifier or designer enter requirements in natural language but validate them with the rigour of formal methods.

The solution?
STIMULUS, from Argosim, is a tool that takes natural language functional requirements and, using a formal method technology hidden under the user-friendly front end, validates them. It can actually produce and run a simulation of the requirements. OK so it is not quite that simple; as most engineers will know, especially those that have seen some of the author’s un-edited prose, that apart from spilling and grimmer, writing clear and precise requirements is not quite as easy as sending a txt!
The text from requirements document is entered into the tool in a structured format. This formatting, and use of keywords, will start the process of assessing and appraising the requirement. As you enter the requirements in to the tool, there will be a need for clarification, clarification which otherwise may not have surfaced until unit or system testing. Anomalies often escape into the field because conventional testing only looks to see that the system meets the requirements but doesn't look for holes in those requirements. If it’s difficult to enter the requirements in to the tool, how can someone else be expected to accurately (safely?) model it or write code for it?
It may be possible to start requirements simulation and validation with just a single requirement but it is usually better to have several interacting. In fact it is possible to simulate whole sub-systems, or even complete systems. Variables and constraints can be entered in to the system: variables with predefined limits, steps and random values if required. The can then be executed as a system multiple times with STIMULUS taking care of a range of start points, within the given constraints and with random inputs. The simulation in Figure 2 was of part of the Paris Metro: STIMULUS is not a theoretical tool but is used in the real world.
Wind speed and direction, time and temperature can be changing variables throughout the simulation which can be run multiple times. This ability to automate simulations can show conflicts and deficiencies in the requirements and allow the running of ‘what if’ scenarios. In these scenarios STIMULUS will show missing ‘else’ requirements and highlight grey areas where things ‘mostly’ work, showing where conflicts are appearing. It will show things that are out of range or are in conflict or it will simply show where behaviour is not what is expected.
With the information from the simulation, requirements can be amended, potentially saving a lot of time and effort, not to mention vast amounts of money. When the tool was being demonstrated to an automotive customer, STIMULUS found in two hours a problem that had taken over a month of debugging to identify.
In aerospace, STIMULUS has been able, after only a few hours or a day’s work, to recreate scenarios that reproduced anomalies in flying aircraft - anomalies had taken months of costly testing to find the cause. Of course had STIMULUS been used initially these problems would never have made it into the design let alone implementation. The tool can also output test vectors, in .CSV files, or .sss files for SCADE integration and can generate C language test Oracles.
Research has identified that Formal Design Review and Formal Code Review produce the best Return On Investment, by a long way, in project development. However with Formal Requirements Validation stopping ‘Garbage In’ it will provide an even better RoI, because subsequent errors (that is, coding, as opposed to requirements) are going to be those that conventional testing tools are actually able to pick up.

Figure 1: A formalised requirement

Figure 2: Simulating the requirement from Figure 1

Figure 3: Aircraft take off

Figure 4: Screenshot showing highlighted conflict condition

Phaedrus Systems