Four elements of measuring program success

By Rick Freedman

March 14, 2012, 8:23 AM PDT

Takeaway: Rick Freedman explores a program measurement theory that consists of measuring success in terms of delivery, organizational, marketing, and innovation capabilities.

In our last post, we talked about the idea of an “achievement pyramid,” a structure that has projects at the foundation of the pyramid, with programs above them and a portfolio (or multiple portfolios, depending on your organization’s governance structure) at the top of the pyramid.
The disciplines associated with project management are well documented and understood, and there are cadres of trained professionals with deep comprehension of the strategies and tactics of project delivery. From Project Management Professionals, proficient in Project Management Institute concepts, to PRINCE2 adherents and agilists like scrum masters and lean project advocates, the foundation layer of the achievement pyramid is well covered. This is not to say that we’ve solved every problem in the PM world and all projects automatically go smoothly; as long as the human element exists, mere tactics and expertise can be no panacea. But we now have a common language, a common set of disciplines, and a series of accepted certifications for PMs agile or traditional.
At the top of the pyramid is the portfolio theme, and this, too, is well documented and has a discrete set of disciplines. Looking back 15 or 20 years, even the idea of managing an organization’s entire suite of projects as a portfolio, and applying governance, financial discipline, and steering committee oversight, was a remote dream. Now, with the sterling efforts of organizations like the IT Governance Institute, and the research of firms like Gartner and the Center for Information Systems Research, many organizations have adopted a strict and disciplined prioritization and return on investment model that makes project portfolio management similar to financial portfolio management in the investment world.
When we discuss project success, we’re typically measuring the tactical elements of delivery: conformance to specification, on time, on budget, with the expected level of quality. When we talk about successful portfolio management, we’re more focused on the strategic level of achievement: did we satisfy the organization’s business goals, did we contribute to competitive advantage or financial results, did we make the right decisions regarding the allocation of scarce resources, from dollars to resources and talents? But what are we measuring when we measure program success?
In a recent article in the Project Management Journal titled “Measuring Project Success” (PDF), authors Shao, Muller, and Turner present a series of Program Management metrics based on a web-based questionnaire to which 172 program managers responded. By conducting a review of the existing literature, and then polling the program management community for input, the authors developed a program measurement theory that consists of four major elements:

1. Delivery capability: Can the organization successfully deliver what it commits to deliver, including the business results as well as the technical deliverables?
2. Organizational capability: Did the program have a positive effect on the capabilities of the organization? Did it improve processes, initiate new ways of collaborating and communicating, or otherwise influence the culture positively?
3. Marketing capability: Does the program organization have the ability to market the new initiatives it delivers, to generate enthusiasm and understanding, to help associates understand what they got and why they should care?
4. Innovation capability: Did the experience of delivering the program enhance the technological or business model innovation? Due to the delivery of this initiative, did the organization uncover new ways of encouraging and developing its own innovativeness?

The article, in the scholarly vein in which it was developed, is more of a statistical and historical reference than a roadmap for practitioners, but I found the basic ideas presented fascinating.

Delivery capability

Clearly the ability to deliver as promised is the foundation of all success on the achievement pyramid; without that, there’s nothing to talk about. The other elements, though, offer an opportunity for project offices and PMs to have a fascinating conversation about the elements that make their work worthwhile.

Organizational capability

The idea of measuring a program by the improvements it leaves behind in an enterprise’s processes, efficiency, and collaborative competence is, to me, a striking way of thinking about our responsibility to do more than just deliver projects, but to also be coaches and catalysts for internal improvement.
One of the key observations of my career as a project and program manager is that technical PMs focus on the technology, and technical organizations measure success through technical achievement. Yet, I’ve also observed that, when projects fail and when they succeed, it’s usually much more a function of the organization’s ability to change the way it works, to adapt to new methods and techniques, to rethink organizational boundaries and roles, and to change process, and make those changes stick, than it is about the implementation of new technology.
This new model proposed in Shao’s article suggests that it’s okay for project managers to focus on the tactical delivery of the technical elements, as long as, at the program level, we’re also measuring the process and cultural readiness of the organization to actually adopt the new methods.

Marketing capability

This leads us to the marketing element. Any experienced project manager has observed the organization that thinks a proper rollout of a sweeping new project is the email announcement that states “starting on May 1, everyone will begin using XYZ Software, and ABC Software will be retired.” We all know the likely success rate of this approach, and yet we still encounter the “they’ll use it because I said so” mentality far too frequently.
When we think about the skills and capabilities that differentiate the roles of project and program managers, the ability to contribute to the marketing, to the creation of understanding and enthusiasm for the new product, and the ability to apply the arts of stakeholder analysis and participation, stand out in my mind like shining lights. In practical terms, when a project manager asks me “what do I have to do to move to the next level?”, these are the capabilities that pop into my mind first and foremost.

Innovation capability

The innovation capability also generates an intriguing line of thought. Agile proponents often promote the use of agile methods in “never-been-done-before” project scenarios, where the ability to explore, speculate, and innovate are central to the success of the initiative. Yet innovation is challenging to instill in a team or organization, and tough to measure when it occurs. Is there such a thing as “50% more innovative” to go with standard metrics like 30% below budget? Maybe not, but the idea that organizations should be thinking at the program level about how a particular initiative encouraged, displayed, rewarded, and institutionalized innovative practices is clearly a noble objective for project managers and executives.
The suggestion that innovation be a formal element of program measurement, while still early in its acceptance, is an outstanding example of the sort of pioneering thinking that keeps our discipline alive and enables us to keep looking for new ways to enhance our contribution as project, or program, managers.

Source:  http://www.techrepublic.com/blog/project-management/four-elements-of-measuring-program-success/4332?tag=nl.e053