Sorting big data

---

That sorting thing is a big deal sometimes. We've all sorted data on a spreadsheet -- but that happens at the blink of an eye. Big data takes a bit longer.

Here's a graphic video demonstration of 15 different sorting methodologies all applied to the same data set, one method at a time (and, set to music). The method is in the small type at the upper left of the video screen.

Sorts random shuffles of integers, with both speed and the number of items adapted to each algorithm's complexity.
The algorithms are: selection sort, insertion sort, quick sort, merge sort, heap sort, radix sort (LSD), radix sort (MSD), std::sort (intro sort), std::stable_sort (adaptive merge sort), shell sort, bubble sort, cocktail shaker sort, gnome sort, bitonic sort and bogo sort (30 seconds of it).

It's fun to watch -- but there's information here also. Take note of how fast one method works compared to another, and note the intermediate data patterns that emerge.

After each method completes, it will be obvious; a big green triangle appears -- you have to see it to know what I'm talking about.

Anyway, it's a fun 6 minutes:

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

A few things to know about Weight

---

I get this all the time from my risk management students: "We do a risk register and weight all the risks".

Fair enough.

But when quizzed about this, it turns out that the students aren't talking about weights, they're talking about probabilities.

Weights and probabilities are really not the same idea. They describe different things. One is about strength or influence; the other is about uncertainty. One has a 'par' value; the other is a number between 0 and 1 (can't happen; will happen, respectively) for which all related numbers sum to 1.0

Some illustrations:

• As a portfolio manager, I may overweight (or underweight) my portfolio-- relative to a par value -- with projects of a certain class -- like semiconductor projects or projects with pharmaceutical content
• As a portfolio manager, I assess the probability of a certain risk -- like curtailment of resources -- as an event that would impact my projects

In the first illustration, "par" might be dollars or a count, or some other metric. Thus I can talk about the weighted value of semiconductor projects in the portfolio in terms of their dollar (or count, or some other metric) influence on the portfolio as being above or below the par value.

It should be obvious, but I'll say it anyway:

• Weights need not sum to 1.0; and in general, they do not

In the second illustration, the usual and most common definition of probability (but not the only one) is frequency of occurrence -- in effect a rate -- based on historical evidence or observation. For instance, how many times was the budget curtailed per 10 opportunities to curtail it?

If only once, as an example, then the probability of such an event is estimated to be 10%. And, by correspondence, the non-event has a probability of 90%, thus accounting for the entire event space. (You guessed it: event space = event + non-event)

Bottom line: watch your weights!

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

Requirements and estimates, and other stuff

---

Take a look at this video, and ask:

1. Could my project's requirements process handle such a project?
2. Could this adventure have been done without software estimates?
3. I wonder how Agile, with all the stuff about emergent scope, have been applied?
4. Would I know how to execute a risk management process for such a venture?

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

Earned value and technical performance

---

Paul Solomon, among many others, has been on a worthy campaign to closely integrate earned value accounting and technical performance -- to wit: no performance, no credit!

Some of this thinking is put down in a paper he wrote for Crosstalk -- The Journal of Defense Software Engineering, entitled: "Basing Earned Value on Technical Performance"

The essence of his idea he states this way:

1.Tie the technical baseline to the EV Performance
Measurement Baseline (PMB).
2.Tie technical progress to the Technical Performance
Measures (TPM) of the program, including progress
towards achieving planned functionality

This stuff is not without consequence. Solomon asserts:

If good TPMs are not used, programs could report 100% of earned value (or credit for work performed), even though they are behind schedule in terms of validating require ments, completing the preliminary design, meeting weight targets, or delivering software releases that meet the requirements

Solomon goes to say that he sees four opportunities (his words):

1. Base EV on technical performance
2. Account for deferred functionality
3. Track tasks discreetly
4. Plan rework and track discreetly

I've probably given away the store with this much reporting, so I'll leave it you to finish the paper, a worthy investment of your time.

---

Paul J. Solomon, PMP, is co-author of the
book, Performance-Based Earned Value.®
He supported the B-2, Global Hawk, and
F-35 programs at Northrop Grumman

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

---

This blog is not about politics and public policy, but the current discourse on meta data brings the subject to the fore in many project meetings because meta data is all around us.

I ran across an interesting project management blog posting precisely on this point that I commend to your inspection and reflection. An excerpt:

The great thing about a project is you can describe it however you want. It just needs to have a goal, a beginning and an end. However, if you subscribe to the project assembly line as the main driver of your value portfolio, then the richness of your project descriptions are paramount. A complete taxonomy with multi-faceted meta is the traffic analyst’s dream. All the analyst needs is one class instance to be thrown, and intelligence is to be gained. Decisions can be made based on probabilities. All of the big data is tamed, as it is now corralled into the project taxonomy. Yes, project, the same thing that is driving your value!

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

Thomas Jefferson and Project Management?

The Jefferson Memorial, Washington

---

It does not speak well of the power of God, ... if He needed a human government to prop him up

Jon Meacham,

expressing Jefferson's sentiment on the freedom of religion 

Project translation:

It does not speak well of the beliefs and culture that inform this project if it takes governance from the PMO to prop them up

To be sure, there's a place for PMO governance, but culture shouldn't need a lot of it. Culture should more or less be inherited from the business -- sometimes from the customer if you are a contractor -- and not require constant attention from the PMO.

But... if you find yourself writing a lot of behavior rules, waxing on at PMO meetings about the culture, then you're likely treating symptoms and not root cause. And, the root cause is probably back in the business.

Culture is a lagging indicator -- but of what? It's a lagging indicator of how well beliefs, etc have been internalized rather than just spoken about. And the internalization is usually led from the top, inherited by all below, and has it's root in the C-Suite

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

That agile estimate thing

---

Neil Kellick has a entertaining slideshare about Agile estimating. Of course, no good deed should go uncritized, so check out the "details" at HerdingCats. Glen has a good discussion of things you should know about what Neill has to say.

Alternatives to Agile Estimation - A Team Perspective from Neil Killick

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

Too many cooks in the kitchen!

---

We've all seen situations where there are too many people in the mix all working on the same problem (when not multi-tasking on one of their screens). So, it should come as no surprise that we hear this wisdom from the corner office:

We’re known for having very small meetings, usually three people. There’s a little clicker for counting people that hangs on the main conference room door.

The reason it’s there is to send a message to people that I care about this issue. If there’s a bunch of people in the room, I’ll stick my head in and say, “It takes 10 of you to decide this? There aren’t three of you smart enough to do this?”

I just hate design by consensus. No innovation happens with 10 people in a room. It’s very easy to be a critic and say why something won’t work. I don’t want that because new ideas are like these little precious things that can die very easily.

Two or three people will nurture it, and make it stronger, give it a chance to see life.

Paul English, co-founder and CTO of Kayak,

in an interview by Adam Bryant

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

Three guys and a project

Three guys are working on a project.
A stranger stops to ask: "What are you making?"
The first answers: "I'm making $20 an hour"
The second answers: "I'm making a wall"
The third answers: "I'm making a cathedral"

Anonymous

What are we to make of this?
I suggest:

• The first guy is not personally committed and does not feel accountable for the task, team, or project success;
• The second guy is a technician, very tactical in outlook, and likely dependable and predictable
• The third guy has the vision in mind, even if he is also a technician, but perhaps also an artisan.

Perhaps also:

• The first guy works for money (I wonder if he is also "eats to live")
• The second guy is one among many you want on your project -- these lend credibility to estimates (#Noestimates not withstanding!)
• The third guy may also have his head a little in the blue-sky domain, but every project needs a few of these -- they bring the passion!

---

Agile 103 -- the THREE BIG QUESTIONS

---

I recently addressed the Central Florida Chapter of PMI with a presentation entitled: "Agile 103 -- THE THREE BIG QUESTIONS". You can get it at slideshare.net/jgoodpas.

• Agile 103?  Because it covers material not in your usual Agile 101 course
• The Three Big Questions? After working with hundreds of students who go through PMI's Agile Project Management course, of which I am one of the instructors, these are the questions every student comes with.

Now, I've addressed these questions here and there in these spaces, but this presentation puts them all together.

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

Agile and System Engineering -- Case Study

---

In a recent edition of the online "Journal of Cyber Security and Information Systems" we find a decent article on agile and system engineering set in a project example context.

Of interest to agilists should be the reference to various frameworks that provide services and tools for different aspects of the project. In broad strokes, the authors have divided their world into business, system, and software aspects.

Another way to look at this article is as an example of "agile in the waterfall". The authors posit, as have we here at Musings, that the grand strategy for such embedding is to work through defined and documented interfaces.

The authors go a step further -- further than I would go, but probably not fatal -- by inventing the idea of the "trusted black box" which is more or less their description of the embedded agile iteration-release model we've talked about before.

I am more comfortable with the old cold war strategy: "trust by verify". That is, no one in a project can be allowed to just take resources and go off to do their thing. (See: command and control lingers on).  In any event, trusted black box is their coin; I like it; I'll probably use it again.

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

Project "soft stuff"

---

From time to time, we pick up a few things that are the grist of project "soft stuff" (I'm not talking about squeezing the Charmin!) Here are a few items you may have missed:

• Are you just returning to the project after maternity leave? As a guy person, I've had to manage others doing this but have no personal experience. However, at "A Girl's Guide to Project Management" we learn about "5 Tips for Returning to Work after Maternity Leave"


• And, in a recent article, we get one Chinese version of Silicon Valley diversions: roller skating to relieve stress! 

Planning impacts
All humor aside, the question ought to be: how to reflect stress relief and the usual living issues in the project planning. I put it all in a bucket and call it "Labor Loss"; I routinely plan for a 15% loss over a year's time.

So, 100 hours on the wall clock has a throughput of 85 hours on the project. Or, a 40 hour week (does anybody only work 40 hours in a week?!) yields about 34 hours on the project.

Do the math the other way: if your project demand is 100 hours, then you need about 117 hours on the clock.

Is this tax too high? It might be; but here's what's worse: not planning for labor loss at all. It's far better to set an anchor (100 per 117) than to drift about with no anchor. At the end of the day, those increments of 17 hours add up.... and then what do you do?

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting

Paper project?

"Reading on paper is 10-30% faster than reading online, plus reviewing notes and highlights is significantly more effective"

Could this claim be true? (I read it in a newspaper ad for paper! Well, hello!)

Does your project run online or on paper? Probably both.  My own experience -- probably like yours -- is that reading online tends to encourage scanning the text rather than reading in depth. Are we trained by texts and twitter to avoid the long form?

For me search and recall dominates: if I can't store, scan, search, annotate, and share a document online then I actually don't want the document, including books. 

All of my professional books are now bought as e-books, though I prefer paper if I'm buying for personal use.

Privacy, secrecy, paper, and (gasp!) the typewriter!

 Now we learn this: The Kremlin's New Cyber-Security Strategy: Russia’s Federal Protective Service, a KGB successor agency in charge of protecting President Vladimir Putin and his officials, has placed an order for 20 typewriters and is ready to pay $750 each for them

But, have they thought it all the way through? Remember carbon paper? That was one of the worst security hazards!

---

Revision

... [my boss] approved of revision. Mainly because revision was about thinking, and he figured thinking never hurt anybody"

Jack Reacher
Fictional police officer
"The Panther"

Public sector funding

---

Public sector funding of your project can be hazardous!

Public sector funding is typically incremental, for one year or two if you're lucky, and subject to the seeming whims of the politics of the moment, to say nothing of the economy that is supposed to source the funds.

What's a PMO supposed to do?
The mainstream risk mitigations for incremental funding is planning in waves, and constructing project and product architectures that accommodate boundaries and deliverables that conform to the budget cycle.

Boundaries means respect for interfaces: that is, in order to get some degree of boundary definition, scope is compartmentalized, and entry/exit is through defined interfaces. In the software business, it's common to think of objects, each object's functionality accessed through an "API" -- application programming interface.

Such a strategy then begets some protocol for ICDs: interface control documentation/design/definition (pick your "D" word)

Another possibility is to plan with options. An option strategy requires a small down payment -- usually not refundable -- to protect future decisions. Recall Yogi Berra's strategy: "If you come to a fork in the road, take it!"

---

Bookmark this on Delicious

Check out these books I've written in the library at Square Peg Consulting