Management Training:
Using ROI to Evaluate Project Management Training
Return on Investment (ROI) is a monetary measurement that is used to evaluate the efficiency and effectiveness of an investment made by an organization. Investments take many forms - financial, human capital, equipment, and training programs - to name just a few. This paper will focus on the use of ROI and the Phillips ROI Methodology(TM) to measure the effectiveness of a project management training program completed within XYZ Law Firm. The 5 levels of evaluation which will be reviewed within the case study include:
Level 1: Reactions (the "smiley" sheet): Did participants like the training they received?
Level 2: Learning: Are participants confident that they have learned something from the training program?
Level 3: Behavior/Application: Are participants able to apply what they learned in the training program back on the job?
Level 4: Results/Business Impact: Did the training show improvement in efficiencies, productivity, profits, costs, reduced turnover?
Level 5: ROI: Did the training program show a positive ROI?
Why the Interest in ROI for Project Management Training Programs?
The challenges surrounding training and its effectiveness within the organization have become more complex over the years. Today, the challenge is even more significant for learning and development professionals. Return on Investment (ROI) as a tool for evaluating project management training is becoming an expectation of senior executives within organizations. In today's tight economy with reduced resources and tighter budgets, learning and development professionals are finding it increasingly necessary to show the monetary value of the project management programs they are bringing to the organization. Today, the success of project management training programs is measured by the financial contribution of the program to the organization. It is not surprising then that ROI measurements for project management training programs are often requested. With such a large focus on project management in all kinds of organizations, there is an increasing desire to show the monetary benefit of investing in project management training programs which often reach a wide variety of employees and represent a significant investment for organizations.
The case study example below will follow the Phillips ROI Methodology(TM) and encompass the four major phases of the methodology:
Evaluation Planning
Data Collection
Data Analysis
Reporting
Case Study Background: XYZ Law Firm
XYZ Law Firm was interested in developing a project management mindset within the firm. There was support from the managing partners and funding was expected to be approved. The goal of the training was to ensure that completing projects at the law firm was not a severe drain on resources. There was much anxiety behind any project which needed to be completed and the champions of the training program - which included a few of the partners, attorneys and the paralegals - were confident that with some basic project management training, this anxiety could be controlled and projects done within the firm would be more easily managed within the law firm. The whole concept of "project management" scared individuals.
The managing partners of the law firm specifically wanted to be updated on project status and they often felt that they were unsure what projects were being worked on and how they were progressing. A significant goal was to ensure implementation of best practices and standard processes and increase knowledge sharing.
Overall the goals of the training program included:
Increased likelihood of successful projects
Ability to implement strategic plans into action
Improved monitoring and controlling of projects
Proactive risk management
Improved time management and teamwork
Efficient utilization and tracking of resources
Standards around status reporting
Implementation of best practices and standard processes
Some of the projects that the law firm worked on included:
IT/infrastructure projects
A variety of projects for their corporate clients, such as mergers and acquisitions and major contract negotiations
Annual recruiting projects (which included recruiting from law schools worldwide)
Long range capital strategic projects
Many such projects always took longer than originally planned for, went out of scope and went over budget and resource commitment. This was certainly a problem for the client projects the law firm worked on as such projects were under fixed price agreements with the client. If the project went off track, it would impact the profitability for the law firm for that particular engagement.
Phase 1: Evaluation Planning
Based on the goals of the law firm, a three day customized basics of project management training program was developed. Prior to the start of the program, data was collected on past project initiatives, specifically around:
Budget
Time
Scope
Resource utilization
Quality
Much of this data was tracked, although it was not in one central location and required conversations with many members of the law firm - from managing partners through to paralegals and administrative assistants.
In the first phase of the project, we developed a plan around the data we would be collecting. The plan included the following components for Level 1 - 4:
Level 1: Satisfaction survey to participants
Level 2: A test administered to participants to measure the learning from the 3-day program.
Level 3: Action Plan follow-up and questionnaires including information on how the participants were applying the skills they learned. Level 3 questionnaires also went to managing partners and senior attorneys to get their feedback and perception of the skills being applied.
Level 4: Collection of all data for business impact
We also developed an ROI Analysis Plan to include the items we intended to measure against to show improvement based on the 3-day training program. The data items to be collected at Level 4, which are the impact measures targeted for improvement included:
Improved performance on projects - including improved quality of end result
Efficient utilization and tracking of resources
Increased percentage of successful projects (on time, within budget, within scope)
Implementation of best practices and standard processes
We set the ROI (hurdle) rate at 20%.
Phase 2: Data Collection
A pilot group of 20 junior attorneys, paralegals and administrative assistants were selected to go through the program first and measure the benefits of the training. In this way, an isolation factor - an important component of doing an ROI study - was available to compare against a similar group of attorneys, paralegals and administrative assistants who would work on projects also but would not initially attend the 3 day program.
The following data was collected from the program:
Level 1: 95% of the participants were satisfied overall with the 3 day customized program.
Level 2: 100% pass rate on the exam (85% or higher was considered passing). It should be noted that the exam was not an overly difficult exam and as long as the attendees participated and paid attention in the 3 day class they would be able to pass the exam. Frankly, the real measurement was in the Action Plans that were produced.
Level 3: 100% of the participants submitted Action Plans and completed follow up questionnaires. Action plans were completed after the class and followed up on within 6 months. The action plan included information on:
Participant's goals (individual improvement efforts)
How participants intended to meet those goals
Action steps to be taken
Support and/or resources needed
Timeline for completion
How participants will know if they are successful in meeting those goals
How participants will evaluate that success
After the 6 month time period, participants were asked to complete a Continuous Improvement Plan for moving forward. This follow up to the Action Plan included:
Results/accomplishments to date
Next steps to accomplish
Timeline for doing so and support needed
Level 4: Data collected from participants and managing partners and senior attorneys on the impact to the business. The data collected from Level 4 showed a 40% increase in the number of successful projects. Successful projects went up from 70 successful out of every 100 (smaller projects included in the 100 along with more strategic projects) to 98 successful out of 100.
40% increase in the number of successful projects
On time and within budget and scope
Projects met set quality standards
Resources tracked and effectively allocated
Standard processes implemented for all projects
Lessons learned captured and best practices shared
Phase 3: Data Analysis
In converting the data collected to a monetary value, we found that cost savings (program benefits) were as follows:
More efficient use of resources: $285,000.
Projects on time and within budget and scope: $250,000
Increase in better tracking of client projects: $450,000
Program benefits are measured in Level 4: Business Impact. To convert to a monetary value, a dollar amount was put to every resource based on salary/time/fringe, etc. If projects did not go out of scope, no additional resources were needed, which was a cost savings. More efficient use of resources also meant that projects had fewer people assigned to them. The $450,000 in increase from better tracking of client projects was the amount that would have been lost from the bottom line had those projects that were measured not been completed successfully.
The total cost of the program was $320,000. This included planning for the ROI study, food and beverage for participants in the program, facilitator costs, development of the customized program, time out of the office for training, salary/fringe of participants, etc. Any cost associated with the program or planning for the program was included in the total costs. The biggest component of the cost was salary/fringe of participants - especially since the attorneys and paralegals are billable resources.
Intangible benefits, those benefits that did not have a dollar value associated with them, were also considered. They included:
Reduced anxiety over involvement in and managing projects.
Improved teamwork and reduced conflicts
Satisfaction of managing partners and senior attorneys over status reporting on projects
These intangible benefits were able to be tied back to the program and certainly contributed to the success of the projects.
During phase 3, we calculate the ROI of the project management training program, using the ROI formula:
ROI = Net Program Benefits / Program Costs x 100
ROI = $665,000 / $320,000 x 100
ROI = 2.08 x 100
ROI = 208%
For every $1.00 invested, $2.08 is returned after the costs are recovered. This was significantly over the 20% ROI set at the beginning of the project during the planning of the study.
Phase 4: Reporting
A report of the results of the study was presented to the managing partners of the law firm. Components of the study were presented to the participants and others in the law firm. The project was considered a success and it was decided that the 3 day customized project management training program would be rolled out firm-wide. Additionally, a 1/2 day program was created for the managing partners of the firm to provide them training around how to support project initiatives. A future focus was to develop a portal to house all project information. That was a "to do" for the IT/Application Development department and would b e a key strategic project for them.
Summary
For training and learning departments, the PMO, or lines of business, showing the ROI of the project management training programs conducted within the organization, whether in-house developed programs or vendor-provided programs, enables you to show the value the programs have on the business in monetary terms that executives can understand, respect, and champion.
The six types of data collected during the Phillips ROI Methodology (TM) process (see list below) enable for consistency in the measurement of project management training programs, thereby bringing credibility to the process.
Reaction and value of program
Learning and confidence
Application and implementation
Business impact
ROI
Intangible benefits
To begin measuring your project management training programs, you might start with a brand new program or review the current programs that are in place - following the criteria for measuring project management training programs discussed earlier in this paper. If you review a current program, gather data on the initial purpose of the program when it was first developed and implemented. Does it still meet that purpose? What value do you perceive the individuals taking the project management training program are achieving? How much has been invested in the program since the beginning? Should this program remain in place "as is," or should it be updated, or possibly be eliminated from your offerings in its entirety or in part to accommodate new needs among your project managers? An ROI study will enable you to look at the value of your current programs and enable you to better plan new programs.
Gina Abudi:
link
Subject:
Management Training
More Management Training Tips
Our Management Training
Classes
