There have been a few high-profile project failures recently that highlight the danger in status reporting. It seems many projects are reported as low risk and on schedule all the way up until the time they are catastrophic failures. There is no progression from good to bad just a very sudden failure. In fact, many (most) projects fail incrementally. Past performance IS a predictor of future performance for projects. Projects that fall behind early rarely recover and finish on time.

The first example is from the federal government’s project to develop the Healthcare.gov website to satisfy the requirements in the Affordable Care Act (aka “Obamacare”). Major government IT projects are tracked on www.itdashboard.gov. The dashboard from August 2013 (three months before launch) shows the status as nearly all green (good). The overall project rating is green and it is rated a 5 (out of 5, with 5 being the best). The dashboard includes six cost and schedule variance assessments – five are green and one is yellow. It also includes operational performance and other “investment” metrics – most are “met.”

Now, I am sure this is not the main status reporting mechanism used by the Healthcare.gov project team but this one sure is misleading – and it’s on a public website for everyone to see (this dashboard exists even today – nearly six months after the failed launch).

The second example is California’s recent payroll implementation project. The status report before its pilot program launch shows 15 areas – 14 are assessed as green and one is yellow. The pilot program launch went poorly just three months later and the entire project was suspended 11 months later.

In both of these cases, what went wrong and how did it go wrong so quickly? The fact is that neither of these projects failed suddenly – both were high risk when these status reports were published. But the status reports painted a rosy picture that may have led leaders astray as to the true status.

The Sacramento Bee has a(nother) article on the San Francisco – Oakland Bay Bridge project. It details how a Chinese construction company did not weld steel girders to the requirements, potentially jeopardizing the structural integrity of the bridge. Some of the eye-opening facts:

• ZPMC, the Chinese construction company hired to build the bridge girders had never built bridge girders before! They were awarded the contract because their bid was $250M below the next closest bid. Naturally the final cost was well over $250M higher.
• Caltrans, the government agency in charge, allowed ZPMC to ignore both welding codes as well as contractual requirements
• ZPMC was not penalized for failure, in fact, they were paid extra to work faster
• Several government workers were re-assigned after pointing out quality issues
• One lead executive for the government quit and joined a company involved in the project

The article implies government waste as government people lived quite well when traveling to China to check on manufacturing. It also implies criminal wrong-doing when it states the law enforcement officials are looking into things.

This project reminds me of the Maryland health exchange project in that an unqualified company won the contract (see here). I don’t know how government procurement officials could get this so wrong.

It also highlights one of the pitfalls in project status reporting. As noted here, even when risks are identified, leaders frequently fail to act.

I’ve written about the failed U.S., Maryland , Oregon and Vermont ACA health exchange website projects. I’ve collected the biggest failures from each of the projects. Most of these findings come from the QA auditors hired for each project. It’s a litany of woe and there are similar issues on each project.

Vermont

1. Project controls not consistently applied.
2. Change Management: Appears to be very little control over changes in schedule, deliverables or scope. Impact to other project areas are not analyzed or alternatives presented.
3. Current weekly status reports have little detail on dependencies, risks or issues.
4. Schedule Delays: deliverables shifted to the future without going thru change management process; potential impacts not known or agreed to.
5. Plan continues to be reworked and updated, there is no way to report on or assess progress. It is not clear what components will be delivered when. The dates seem to be unrealistic and unachievable. Some dates violate CMS schedule requirements.
6. Development (and, later) test, (and, later) prod, (and, later) pre-prod, training and DR environments are not ready as planned.
7. Requirements: lack of granularity in scope, definition and traceability; Insufficient detail; Amiguity; requirements not tied to work flows or use cases.
8. Vendor staffing not as scheduled (20 unfilled positons) with no staffing plan.
9. It is likely that functionality will not be tested until it is too late.

Oregon

1. Lack of universally accepted PM processes
2. Ineffective communication and lack of transparency
3. Poorly defined roles and responsibilities
4. Lack of discipline in the change management process

Maryland

1. Lack of a schedule
2. No single person as the overall PM
3. Unclear oversight
4. No documented list of features at go-live
5. Code moved directly into production
6. Lack of integration and end-to-end testing
7. Insufficient contingency planning

 

I’ve written about the failed U.S., Maryland and Oregon ACA health exchange website projects. Here is Vermont.

Vermont’s contractor was CGI, who was also responsible for the U.S. project. The QA contractor was Gartner Consulting. Nearly all of the released findings stem from Gartner’s QA reports.

Similar to Oregon’s health exchange project, at no point in time did the QA contractor assess risk as anything but RED – high risk. For Vermont, it was 12 straight high risk bi-weekly status reports, the last one being after the October 1 go-live when the probability of most risks occurring hit 100%. The most critical risks throughout the project were:

• Planning. Gartner notes a lack of planning around integration – probably the hardest part of the project and testing – the most important part of the project.
• Project Management.  A laundry list of project management issues: lack of project controls, appears to be very little control over changes in schedule, deliverables or scope. Impact to other project areas are not analyzed or alternatives presented, plan continues to be reworked and updated, there is no way to report on or assess progress. It is not clear what components will be delivered when. The dates seem to be unrealistic and unachievable. Some dates violate CMS schedule requirements.
• Scheduling. Numerous schedule issues (“missed delivery dates and milestones” is noted in 8 of 14 released QA reports) including one stunning problem probably caused by technical issues or poor management: Development (and, later) test, (and, later) prod, (and, later) pre-prod, training and DR environments are not ready as planned. Essentially no environment was ready as planned – NONE. Also: “Plan continues to be reworked and updated, there is no way to report on or assess progress. It is not clear what components will be delivered when. The dates seem to be unrealistic and unachievable. Some dates violate CMS schedule requirements.”
• Staffing. Within five months of go-live, Vermont had four unfilled positions, including Test and Training Leads and CGI had 20 (!) unfilled positions. Worse is that there was no staffing plan to address this gap.

In some cases, the QA reports include Vermont’s risk management strategy. One strategy is especially damning.

• Risk Rating: HIGH
• Risk: It is likely that functionality will not be tested until it is too late.
• Vermont’s Risk Response: Accept

My response: D’oh! This shows that Vermont was fully prepared to implement features that it had not tested. Unacceptable.

This audit is not as comprehensive as Oregon’s audit as it is almost entirely based on Gartner’s QA reports. And some of them have redactions that, of course, makes you wonder what is hidden – could it be worse? Nonetheless, more lessons learned when undertaking a large IT project.

 

Following my post on the Oregon ACA website and the Maryland ACA website project failures, I am following up with the federal ACA website project. Unlike Oregon, there has not been a (publicly-available) thorough audit of the failure. Some data points have been widely reported: 70% of users could not even log in at launch and only eight people were able to sign up on day 1. I’ve read where stress testing before launch indicated the site could only handle 1,500 users – even though over 30 million Americans are uninsured. I also understand the team was not co-located (White House in D.C., CMS in Baltimore, contractors in Columbia, MD) and decision-making was difficult.

Steven Brill has written a story at Time (gated, here). It’s mostly about the team sent in to rescue the project. The first thing the rescue team was to develop a dashboard. Brill quotes one saying it was “jaw-dropping” that there was no “dashboard  – a quick way to measure what was going on at the website…how many people were using it, what the response times were…and where traffic was getting tied up.” Implementing a dashboard was job 1 for the rescue team.

Brill notes “what saved it were …stand-ups.” He further says  that “stand-ups…are Silicon Valley-style meetings where everyone…stands-up rather than sits….” Brill implies that California software companies invented stand-ups. While I have no doubt that stand-ups are popular in Silicon Valley, I doubt they were invented there. I worked for an admiral (Craig Steidle) in the ’90s that did daily stand-ups and stand-ups are a key component of the Scrum methodology. Many, many companies use stand-ups. I guess journalists are unfamiliar with them.

I look forward to a real audit of the project. Brill’s article is entertaining but not illuminating as to the problems that plagued the project. The federal ACA website project has common characteristics with state-led ACA failures – poor choice in contractors, poor project management, no risk management, lack of a single point-of-authority and poor oversight. Perhaps we can have the GAO review the effort – we must learn form these large IT project failures, otherwise we are doomed to repeat them.

I recently documented the failings of the Oregon ACA website. Now its Maryland’s turn. Maryland has commissioned an audit so we should get a good inside look. But the audit is not scheduled to be done until the summertime. We do know a lot about this failure already. According to documents from the IV&V contractor obtained by the Washington Post, these problems existed early in the project:

• Insufficient State staffing
• Insufficient time for software development
• Lack of a detailed project plan
• Inefficient communications
• Lack of sufficient time for regression and system testing
• Lack of a comprehensive QA and testing plan

The prime contractor, Noridian, has been terminated. A substantial amount of their fees has been withheld by the state.

There was also significant issues between Noridian and one of their subcontractors, EngagePoint. Interestingly, I read where Noridian hired EngagePoint since it had no real ability to develop the exchange. I don’t know why Maryland awarded a contract to a company that had to quickly hire a subcontractor to do the work.

Most recently, Maryland decided to scrap the $125M effort and adopt the same technology used in Connecticut. Adopting the CT exchange technology is expected to cost an additional $50M. It makes me wonder why some states didn’t just do join together in the first place. Or staged a phased implementation so mistakes could be corrected before a nationwide launch.

There is a fantastic audit of Oregon’s attempt to implement the Affordable Care Act (ACA) (“Obamacare”). Highlights include:

• Oregon combined ACA implementation with a complex project that determines if people eligible for healthcare subsidies are also eligible for other government programs. To state the obvious: you NEVER combine a high risk project with another high risk project.
• Oregon chose Oracle as the software provider and paid them on a time and materials basis. Contractors are usually paid on a deliverable or milestone basis so the risk is shared.
• Oregon paid Oracle through a contract vehicle called the Dell Price Agreement. The name alone indicates this may not be the proper way to contract with Oracle. In total, Oregon signed 43 purchase orders worth $132M to Oracle.
• Oregon did not hire a systems integrator, a common practice in these types of projects.
• Oregon hired an independent QA contractor, Maximus. Maximus raised numerous concerns throughout the project. They never rated the project anything but High Risk.
• Lack of scope control, a delay in requirements definition, and unrealistic delivery expectations were cited as main issues.
• As late as two weeks before the October 1 launch, the Government PM was positive about the project: “Bottom Line: We are on track to launch.”

Who Keeps Moving the Goal Post?

One interesting point in the audit is the difficulty in accurately estimating how much effort it takes to do software development. As they were progressing, they were developing estimate-to-completion figures. As shown in the chart below, the more work they did, the higher the estimate-to-completion got. As they got closer to the end, the end got farther away. D’oh! In general, this shouldn’t happen. The more “actuals” you have, the better your estimates should be.

Who Knew this Project Would Fail? These Guys Did.

The audit quotes the QA contractor, Maximus, throughout the report. They appeared to have a good grasp on what was happening, pointing out risks throughout the project. A summary of their risk assessments is shown below. The top row, which is Overall Health, is red (high risk) from Day 1. Unfortunately, the report says the project team became “de-sensitized” by the risk reports since they were always bad. Perhpas Maximus should have started with a green (low risk) status.

If One Oversight Group is Good, More Must Be Better

There were several agencies that had oversight responsibilities. Naturally this caused confusion and blurred lines of responsibility. A common refrain was the lack of a single point-of-authority. The audit doesn’t make a recommendation but I will: Oversight should be the responsibility of a single group comprised of all required stakeholders. I have seen many public sector projects that believe additional levels of oversight are helpful. It is not. They serve to absolve members of responsibility. They can always say they thought someone else was responsible for whatever went wrong. If there is only one oversight group, then they can’t point fingers anywhere else and they are more likely to do their job well.

Like Hitting a Bullet with Another Bullet

As noted above, Oregon combined ACA implementation with a complex project that determines if people eligible for healthcare subsidies are also eligible for other government programs.

The Standish Group produces a famous report (the CHAOS Report) that documents project success. The single most significant factor in project success is size. Small projects, defined as needing less than $1M in people cost, are successful 80% of the time while large projects, over $10M, are successful only 10% of the time. (Success is defined as meeting scope, schedule and budget goals.)

So Oregon decided to combine the high risk Obamacare website project, with a projected success rate of 10%, with another high risk project with success rate of 10%. That’s like hitting a bullet with another bullet.

Everything was On Track, Until It Wasn’t

As late as two weeks before launch, the Government PM reported that the launch was on track. The audit notes the system failed a systems test three days before launch and the launch was delayed the day before launch. (It appears the system test conducted three days before launch was the first system test performed.) Even while announcing the delay, Oregon said the site would launch in two weeks (mid-October). By November, the launch consisted of a fillable PDF form that was manually processed. The site had yet to launch six months later (March 2014).

There is a common story told in project management: “how did the project get to be a year late!” “One day at a time.” By April before the October launch, the project was months behind. As the spring and summer progressed, the project fell further behind. And yet, the PM continued to believe they would “catch up” and finish on time. I don’t know if its ignorance or malfeasance at work here. But it is virtually impossible to “catch up.”

One funny (in a sad way) part. The development effort was planned for 17 iterations. So what happened when they completed iteration 17 and still weren’t done? Iteration 17a, 17b, and 17c. Ugh. Also, the use of the word iteration implies an Agile methodology but this isn’t indicated in the audit. It wouldn’t surprise me that Agile was abused misused used on this project.

The audit has many lessons learned. I encourage you to read it all, especially if you are undertaking a large system implementation in the public sector.