A note from a reader of some of my stuff on slideshare.net/jgoodpas:
"... I just read your slide presentation on using stats in project management and am a bit confused or perhaps disagree ... that using Monte Carlo will improve the accuracy of a schedule or budget based on subjective selections of activity or projectI replied this way about accuracy:
a) optimistic, realistic and pessimistic ($...Time) values,
b) a probability distribution (3 point triangular) and
c) arbitrary probability of occurrence of each value.
If I understand your presentation Central Limit Theory (CLT) and the Law of Large Numbers (LLN) when applied using Monte Carlo simulation states that accuracy is improved and risk more precisely quantified.
It seems to me that this violates a law that I learned a long time ago...garbage in-garbage out. Is this true? ...."
- Re 'accuracy' and MC Sim (Monte Carlo simulation): My point is (or should have been) the opposite. The purpose of MC Sim is to dissuade you that a single point estimate of a schedule is "accurate" or even likely. Indeed, a MC Sim should demonstrate the utility of approximate answers
- Project management operates, in the main, on approximations... ours is the world of 1-sigma (Six Sigma is for the manufacturing guys; it doesn't belong in PM).
- Re the precision/accuracy argument: My favorite expression is: "Measure with a micrometer, mark with chalk, and cut with an axe", implying that there is little utility in our business (PM) for precision, since most of our decisions are made with the accuracy of chalk or axes.
- A MC Sim gives you approximate, practical,and actionable information about the possibilities and the probabilities (all plural) of where the cost or schedule is likely to come out.
- And, to your point, the approximate answers (or data) should be adequate to base decisions under conditions of uncertainty, which is for the most part what PMs do.
- Re your contention: " ... selecting the distribution, optimistic, pessimistic and realistic values must be accurate for Monte Carlo to provide more accurate results." Actually, this contention is quite incorrect and the underlying reason it is incorrect is at the core of why a MC Sim works.
- In a few words, all reasonable distributions for describing a project task or schedule, such as BETA (used in PERT), Normal (aka Bell), Rayleigh (used in many reliability studies, among others), Bi-nominal (used in arrival rate estimates), and many others are all members of the so-called "exponential family" of distributions. You can look up exponential distributions in Wikipedia.
- The most important matter to know is that, in the limit when the number of trials gets large, all exponentials devolve to the Normal (Bell distribution).
- Thus, if the number of trials is large, the choice of distribution makes no difference whatsoever because everything in the limit is Normal.
- If you understand how to do limits in integral calculus, you can prove this for yourself, or you can look it up on Wikipedia
It depends.
- As few as five sometimes gives good results (see image) but usually 10 or more is all you need for the accuracy needed for PM.
- Consequently, most schedule analysts pick a triangular distribution, which does not occur in nature, but is mathematically efficient for simulation. It is similar enough to an exponential that the errors in the results are immaterial for PM decision making purposes.
- Some other picks the uniform distribution like shown in the image; again for mathematical convenience
- Now the next most important matter to know is that a sum of exponentials (BETA, Rayleigh, whatever) -- like would be the case of a sum of finish-to-start tasks -- has the same effect as a number of trials.
- That is, if the project is stationary (does not matter when or where you look at it), then a string of repeated distributions (as would be in a schedule network) has the same characteristics as a single distribution tried many times.
- And, each member of the string not be the same kind of distribution, though for simplicity they are usually assumed to be the same.
- Thus, whether it's one distribution tried many times, or one distribution repeated many times in a string of costs or tasks, the limiting effect on exponentials is that the average outcome will itself be a random variable (thus, uncertain) with a distribution, and the uncertainty will be Normal in distribution.
- The usual statement about GIGO assumes that all garbage has equal effect: Big garbage has big effects; and little garbage has little effects; and too much garbage screws it all up.
- This is certainly the case when doing an arithmetic average. Thus, in projects, my advice: never do an arithmetic average!
- However, in a MC SIM, all garbage is not equal in effect, and a lot of garbage may not screw it up materially. The most egregious garbage (and there will be some) occurs in the long tails, not in the MLV.
- Consequently, this garbage does not carry much weight and contributes only modestly to the results.
- When I say this I assume that the single point estimates, that are so carefully estimated, are one of the three estimates in the MC Sim, and it is the estimate that is given the largest probability; thus the MLV tends to dominate.
- Consequently, the egregious garbage is there, has some effect, but a small effect as given by its probability.
What about independence and correlations?
- If the distributions in the MC Sim are not independent then results will be skewed a bit -- typically, longer tails and a less pronounced central value. Independence means we assume "memoryless" activities, etc so that whether a string of tasks or one task repeated many times, there is no memory from one to the other, and there is no effect on one to the other other than in finish-to-start there will be effects on the successor start time.
- Correlations are a bit more tricky. In all practical project schedules there will be some correlation effects due to dependencies that create some causality.
- Again, like loss of independence, correlation smears the outcome distribution so that tails are longer etc.
- Technically, we move from the Normal to the "T" distribution, but effects are usually quite inconsequential.
- My advice is in the risk management plan for the project, establish a policy with two points:
1. All tasks on the calculated single point critical path will be estimated by experts and their judgment on the task parameters go in the sim
2. All other tasks are handled according to history re complexity factors applied to the estimated MLV (most likely value, commonly the Mode): that is, once the MLV for a non-critical task is estimated, factors set by policy based on history, are applied to get the estimates of the tails. - Thus, a "complex task" might be: optimistic 80% of MLV and pessimistic 200% of MLV, the 80% and 200% being "policy" factors for "complex" tasks.
- This is the way many estimating models work, applying standard factors taken from history. In the scheduling biz, the issue is not so much accuracy -- hitting a date -- as it is forecasting the possibilities so that some action can be taken by the PM in time to make a difference.
- Providing that decision data is what the MC Sim is all about. No other method can do that with economic efficiency
PERT v MC Sim
Now, if the schedule is trivial, there is no real difference in the PERT and MC Sim. Take, for example, a dozen tasks in a tandem string of finish-to-start.
- The PERT answer to the expected value of the overall duration and the MC Sim will be materially identical.
- The sum of the expected values of each task is the expected value of the sum. So, the SIM and the PERT give identical answers insofar as an "accurate" expected value.
- But that's where it ends. PERT can give you no information about the standard deviation of the outcome (the standard deviation contains the reasonable possibilities about which you should be worried), nor the confidence curve that lets you assess the quality of the answer.
- If your sponsor asks you run the project with an 80/20 confidence of finishing on time, where are you going to go for guidance? Not to PERT for sure.
- And, if your sales manager says bid the job at 50/50, you have the same issue... where's the confidence curve going to come from?
- And, most critically: PERT falls apart with any parallelism in the schedule in which there is no slack.
I'm almost out of gas on this one, so I'll end it here.
The image shows the outcome of 5 independent uniform distribution summed, as in a finish-to-start schedule sequence.
Read in the library at Square Peg Consulting about these books I've written
Buy them at any online book retailer!
Read my contribution to the Flashblog
