Analytical projects are some of the most complex projects around.  It is possible to research buzz-words on the internet and to then cobble an extremely plausible resume.  I have seen analytical “Gods” who at interview were not able to log into to the application.

Businesses with little or no analytical experience can not easily sift the wheat from the chaff and it can take up to 3 months or more to realise the limitations of analytical resources brought in as experts.

The most important of these resources is the Project Manager.  In many cases, the Project Manager may not have the skills required to drive the project through to an assured success.

In between 17% and 25% of projects, success will be achieved and any skill insufficiencies will never surface.  In the remainder of projects however, there will be substantial project failure.  The more intelligent of these nominal project managers may leave the project before failure becomes visible and in some cases, they have made a very successful career simply by identifying the correct time to leave.  “The project was fine when I left!”.

So how do you identify a project manager who can lead your analytical project to success?  What attributes should he or she have?

One of the most important attributes is communication.  Can the Project Manager communicate with a wide variety of business personnel to establish requirements, challenge requirements where necessary and ensure that business analysis is appropriate and pragmatic? Can he or she manage the stakeholders effectively and ensure that expectations are managed and fulfilled? Does the track record reflect this over say a decade?

Closely related is presence or gravitas.  Can the project manager stand in front of a steering or board level group of executives and explain to them that the product salesman has sold them pie in the sky and that their expectations have now to be realigned with reality?

The ability to create a project plan is good; but the ability to create an achievable project plan is far better.  Many project managers do not know how to calculate the probability of achieving a project plan on time and within budget.  There are statistical techniques well known since the 1950s that precisely identify the probability of a project moving according to plan.  However, these techniques also depend on an understanding of which elements of work are the riskiest and should be weighted most heavily; this knowledge depends on a very detailed understanding of the processes involved in an analytical project during the different phases.

Therefore a technical project manager who has ideally been involved in all tiers of the project roles is more strongly correlated with project success.  An important attribute is therefore technical skill.  Can the Project Manager sit down and develop at each and every part of the analytical application stack?  Can he or she write complex sql?  Can he or she run an ETL process?  Has he or she performed full and complex dimensional modelling in the recent past?

Track record can identify this quite easily.  How many analytical projects has the project manager recovered in the last ten years?

Common sense is unfortunately quite rare.  Does the project manager understand Pareto’s principle and can he or she apply it to the project?

Does the project manager understand operational statistics?  What is the difference between inferential and non-inferential statistics?  How many test cases should be undertaken to achieve Sigma 3 confidence levels in testing?  What is GLM and how does it relate to analytical modelling?

Data Security is one of the most important and abused aspects of analytical projects.  Factors such as regulatory, cross-border, location, position, departmental, cost centre and many to many security relationships means that Data Security is a minefield.  Can the Project Manager show at least five years track record in performant data security?

Resource selection.  Can the Project Manager recruit and employ only the best resources? Say Sigma 3 resources?

And finally track record.  There are many further indefinables.  A consistent and reasonable duration track record is really the most important indicator of future project success.

Don’t worry if your project does not have these skills on board.  You still stand a 20% chance of success.

Given that in the vast majority of instances, the business resources are both competent and passionately interested in project success; and also given that the vast majority of consultancies have competent resources who are passionately interested in project success, particularly for the larger and more visible clients: the number of project failures are therefore apparently inexplicable.

If we look closer, however, the seeds of failure start very early on in the project. So here is a model anatomy of a failed project. This is not a case study, because it is actually an amalgam of experiences from multiple projects; consider this as the lowest common denominator of failure.

Day 0

The organisation identifies its analytical needs, elicits and achieves serious business sponsorship, defines and agrees terms of reference, obtains budgets, begins to investigate products on the market and then invites one or more prestigious consultancies to tender on a fixed price contract for the project.

At this point, there is excitement, passion, fervour and a lot of competent business people convened to make sure that the project is successful.

The consultancy eventually chosen arrives in and if the business has done its homework, the consultancy selected will be much in demand and consequently over-stretched. At best, there will be one or two senior resources and a distribution of other resources ranging from previous week boot-camp completion to less than a year’s experience.

The Project Manager will often have no experience of the product and if challenged will state something along the lines of “Project management is application-agnostic. A good project manager does not have to be technical, does not have to understand the application; he/she just has to understand the principles and practice of good project management”.  This is an actual verbatim quote!

None of the resources will have previous experience with the client and in a surprising number of cases, none of the resources will have experience with the client business domain. “Yes we understand HR. No we do not understand Retail banking/Public Sector/Retail HR but HR is HR”.

The business has no previous understanding of analytical approaches and assuming they firmly avoid buying into the “application-agnostic requirements” white elephant, their requirements will still not be suitable for implementation of the chosen application.

The Consultancy will then make a translation of the business requirements for a client business model they do not understand and possibly for a business domain in which their onsite resources have no previous expertise.

Day 45

A set of requirements will be formalised in classical waterfall SDLC approach and presented to the client. After a number of reviews and workshops, the requirements will be signed off by the client. The requirements will not add anything new to the business understanding, in most cases, they are simply reflections of the business requirements already provided to the Consultancy by the business.

A stage payment will be taken at this point.

The Consultancy will then begin to request hardware. Unfortunately, there is a long lead time on hardware and there are some substantial challenges to taking the data onto the consultancy hardware. Metadata is then provided to the consultancy and the design work begins while the hardware is ordered.

Day 75

The designs are now finished and presented to the client for sign-off. The business has no previous knowledge of the analytical application and after a number of reviews and workshops will be signing off a design they do not fully understand based on the interpretation by the consultancy of a business model they, in turn, do not fully understand.

Another stage payment plus Cost Variations for business-source project delays are invoiced by the Consultancy at this stage.

Day 90

The hardware is now available and a number of challenges will be experienced in preparing and configuring the hardware. At this stage, it is common to hear “Yes, you ordered the servers, but you did not order the server room capacity, the redundant power supply and the air conditioning capability”. Further, process challenges will now be experienced. “No, your DBA cannot have system access to the database! Your administrator cannot have access to the OS root password, even temporarily. No, your people cannot have access to the data, it has to be scrambled.”

Day 110

By this stage, the hardware is now available and ready for use, the software is configured and the first daily ETL takes place. It takes 100 hours to complete and fails miserably. After many ever terser conversations, it transpires that the hardware is not sufficiently performant, the network bandwidth has not been requested and a lot of the required network ports have not been opened.

Day 120

After a number of acerbic and progressively senior conversations, the hardware is now really available and ready for use.

The first ETL takes place and now takes 30 hours to run. This is a daily ETL and immediately the project DBA’s will swing into action and look at both the source and target databases with a view to dramatically reducing the ETL duration.

Day 130

The daily incremental ETL is now down to 11 hours and the despondency of the business is beginning to lift. There is a new confidence and new spirit of optimism.

Day 160

Development, Integration and System testing have now been completed and User acceptance testing (UAT) is convened. Because of the delays, a number of business owners are now on holiday and there is a choice of delaying UAT or continuing on UAT without the business subject matter experts. The decision is taken to proceed with UAT and a number of stringent exit gates are established and communicated.

This is the first test against real rather than scrambled data and there are a number of significant and substantial defects now appearing. Frantic work begins to fix the various application tier insufficiencies to accommodate the real data. Finally, the processes appear to work and UAT resumes.  The extent of testing of ad-hoc fixes is limited.

The Business owners begin to test reports which they have never previous seen in an application which they have not previously experienced dealing with interpretation of their business needs by a consultancy that has limited understanding. Any challenges by the business is immediately referred to the signed off requirements and design documentation by the Consultancy. A number of defects are logged with variable and inconsistent severity. “The font is wrong = show stopper”, “The data is wrong = minor defect”.  Contractual terms and conditions form the basis of many arguments at this stage.

The UAT time-period are used up and there is no more time available to test any fixes.

The Consultancy and Senior management now work together to manage this process and eventually UAT is declared a success because time has run out, there has already been too much delay, the project costs have significantly over-run and board level management are now expressing their impatience with the delays. “If there are problems, we can always fix it later” is often heard at this point.

A further stage payment now takes place

Day 180

Go live takes place and no further UAT is allowed. The only testing is to ensure that the go-live deliverables are the same as the UAT deliverables. Go live acceptance testing takes place and is eventually declared a success.  This success is communicated extensively to internal and external interested parties.  People are patted on the back.  The Consultancy leaves and discussions are concluded concerning warranty/support/etc.

Day 190

Access to the application is rolled out to the wider business community.  A gradual upwelling of business dissatisfaction reaches critical mass. Common sound-bites are:

“The data we need is not there”
“The system is really slow”
“The metrics are just wrong”
“The system is unusable”

Business dissatisfaction climbs and despite any mitigation communications, eventually too many senior business managers decide that the new system is not fit for purpose. A number of internal archetypal statements appear at this time.

“The product is too complex”
“We want to use an alternative product”
“My department has an existing home-grown solution using MS Access and a php website that we should use while we are fixing this catastrophe, and possibly for the longer term”
“You have not managed this project well!”
“You are responsible for this failure”

Any approach to the consultancy is met by reference to the various signed-off documentation and contracts. The Consultancy may suggest a new project with a 5% reduction off the original project cost as a gesture of good faith. In many cases the business will refuse.

The project has failed!!

Clearly a simplified, over-generalised template of specific project failure?  Right?  Wrong!  This actually happens a great deal of the time.

Post-mortem sound-bites:

“But we picked the best product from Gartner”
“But we picked the product owner consultancy”
“But we really did our homework on the requirements”
“But we really spent so much time inspecting the requirements”
“But we set up our TOR and project gates by the book”
“But our waterfall SDLC was perfect”

How could this have gone so wrong?