Consider this quote from Tony Hoare:
One can construct convincing proofs quite readily of the ultimate futility of exhaustive testing of a program and even of testing by sampling. So how can one proceed?
The role of testing, in theory, is to establish the base propositions of an inductive proof.
You should convince yourself, or other people, as firmly as possible, that if the program works a certain number of times on specified data, then it will always work on any data. This can be done by an inductive approach to the proof.
Everyone remember what an inductive argument is? Yes? Good!
Then I don't need to say that ....
When reasoning inductively you are reasoning from a specific instance to a generalization. The validity of the generalization can only be known probabilistically. (It's probable-- with high confidence -- that any instance will work because a particular instance -- or particular set of instances -- worked)
Thus, there is no absolute certainty that a generalization will work just because a large set of specifics works.
Monte Carlo testing is inductive. There are two inductive conditions in Monte Carlo testing:
- That the test conditions are time-insensitive ... thus, tests done serially in time are valid for any time, and are valid if the tests were done in parallel rather than serial
- That the test results for a finite number of iterations is representative of the next iteration that is not in the test window
And why would one do this?
Economics mostly.
Usually you can't afford to test every instance possible; sometimes, you can't create all the conditions to test every instance possible. The plain fact is, as Mr Hoare says, you should be able to do "just enough" testing to convince yourself and others who observe objectively that, hey: it will work!
Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!
Read my contribution to the Flashblog
