The PACE Intensive Services (IS) Program conducts capacity-building exclusively with small and mid-sized organizations that primarily work to help minority members of our community and the disadvantaged. We pride ourselves for our use of innovative and dynamic strategies to increase the efficiency and effectiveness of the organizations in the program.
In the fall of 2013, with funding from the United Way of Allegheny County, PACE began an evaluation process to determine the degree to which the IS Program’s capacity-building outcomes were being achieved. Called the CCAT, this assessment tool looks specifically at Leadership, Adaptive, Management and Technical capacities, and additionally at Organizational Culture and Lifecycle Position and provides relevant comparisons on both an individual and group basis.
On Wednesday, April 2, 2014, globally recognized external evaluators, TCC Group and Algorithm presented the findings of their independent evaluation of the PACE Intensive Services Capacity Building Program to United Way senior staff, PACE Leadership and staff. The evaluators also conducted a workshop for IS participants that provided an in-depth review of the assessment tool results as well as a review on how to use the results to build on their successes. The TCC Executive Summary can be viewed below.
The Evaluation of PACE’s Intensive Services Program
In its Intensive Services (IS) program, Program to Aid Citizen Enterprise (PACE) conducts capacity-building exclusively with small and mid-sized organizations, specifically organizations working to help minority and disadvantaged members of the greater Pittsburgh community. In the fall of 2013, PACE contracted TCC Group to determine the degree to which the Program’s capacity-building outcomes were being achieved. TCC Group and Algorithm collaborated in this evaluation.
II. Evaluation Design
TCC employed a mixed methods design for this evaluation including a bench review of internal PACE documents, interviews with PACE participants, and administration of TCC’s Core Capacity Assessment Tool (CCAT) to PACE participants. CCAT scores of PACE participants were compared to a national database of comparable nonprofits.
III. Program Participants and TCC’s Four Core Capacities Model
Findings were presented using the lens of TCC’s Four Core Capacity Model, which is comprised of the following capacities:
- Leadership: The ability of all organizational leaders to create and sustain the organization’s vision, inspire, model, prioritize, make decisions, provide direction and innovate, all in an effort to achieve the organizational mission.
- Adaptive: The ability of a nonprofit organization to monitor, assess and respond to internal and external changes.
- Management: The ability of a nonprofit organization to ensure the effective and efficient use of organizational resources.
- Technical: The ability of a nonprofit organization to implement all of the key organizational and programmatic functions.
CCAT scores fall on a scale of 0-300 with the top scores indicating a “Strong” level of capacity, middle scores indicating a “Satisfactory” level, and lowest scores representing a “Challenging” level. Findings on Four Core Capacity outcomes include the following:
- High-scoring PACE participants improved the capacities they set out to improve. TCC compared interview findings to a list of capacities that IS program participants set out to achieve. In a large majority of cases, high-scoring interviewees improved the capacities they initially intended to improve. Low-scoring PACE participants achieved many of their capacity-building goals, but they often qualified their answers in interviews.
- The capacity with the highest number of “Challenging” scores was Technical. Nearly half of participants have indicated they are struggling with having the necessary resources to implement their programs, which is extremely common among CCAT respondents. PACE organizations particularly struggle with having sufficient facilities to do their work.
- High-dosage organizations tend to do better on the CCAT. TCC analyzed CCAT scores of all 11 organizations who took the CCAT alongside their dosage (the number of IS program grant cycles high-scoring organizations have completed). CCAT scores of participant organizations tend to be higher if organizations have experienced more PACE programming. Of the seven organizations with a dosage of three or higher, five received “Strong” CCAT scores. No organizations with a dosage of one received a “Strong” score.
- Organizations score well on Adaptive Capacity, particularly the Organizational Learning Subcapacity. In PACE’s previous capacity-building engagements, participant organizations experienced important increases in their ability to plan. Data from this evaluation back that finding up – IS program participant organizations are doing comparatively well on Adaptive Capacity. Participants have average scores that are higher than comparison organizations in six of seven capacities/subcapacities.
- Organizations scored well on many areas of Management Capacity, notably Financial Management and Managing Program Staff. Compared to other capacities, participants seem to be doing especially well on the Management Capacity – five of the participants have “strong” scores. They are doing particularly well in specific areas such as Financial Management, where PACE participant organizations scored 15 points higher than comparison organizations. For some subcapacities, participants’ scores were slightly lower than comparison organizations’ scores (for example, Volunteer Management and Staff Development). They also did less well in areas such as Conveying Value to Staff and Supporting Staff Resource Needs. This finding may point to areas where the Program can increase its capacity-building efforts (example: middle management).
Finally, on average, organizations received scores of at least “Satisfactory” on Organizational Culture. Two interviewees mentioned their culture had improved since the PACE grants.
IV. PACE Intensive Services Capacity-Building Program Components
According to TCC’s review of best practices in the field of capacity-building, capacity-building programs should provide responsive, relevant, and learning-focused support. PACE’s activities appear to be in-line with those best practices. Interview data led to the following findings:
- The IS program promotes positive and open relationships between funders and participants. Interviewees indicated that the tone of the funder-participant relationship was positive and conducive to growth. Additionally, PACE implementers made themselves available to help participants. This evaluation finding aligns with the 2009 survey findings that PACE staff is “hands-on,” “collaborative,” and easily accessible.”
- Convenings/learning opportunities had mixed reception among participants. Some interviewees felt it was difficult to bring board members to convenings but acknowledged the importance of board engagement for sustainability of capacity building.
- Organizations appreciate the opportunity to receive multi-year funding. Across the sector, nonprofits indicate that multi-year funding is important to them. Two of the six interviewees mentioned they appreciate this about PACE support, and they believe it allows the funder to view capacity changes over time.
- Many participants had not previously worked with other capacity-building programs. Some organizations said they had had little or no experience with other capacity-building programs. This evaluation finding was similar to the finding on the 2009 PACE survey that many participants had not used capacity-building programs other than PACE.
- PACE’s IS program fills a unique niche in local capacity-building. Interviewees were asked to describe the characteristics of the IS program that distinguished it from other programs. In many cases, interviewees did not have the ability to speak to that question because they did not have much experience as consumers of capacity-building services. They also indicated that they are not aware of other organizations in the area that provide similar support to PACE’s capacity-building. This data points to the finding that PACE is filling a unique niche in the Pittsburgh capacity-building arena.
Since PACE’s support is in line with best practices, it should continue the following practices:
- Providing multi-year grants;
- Offering high-quality consultants;
- Tracking standardized metrics over time;
- Regularly reviewing programs to ensure you are leveraging the peer-cohort model to its full potential;
- Engaging in efforts to prepare participants for capacity building programs; and
- Monitoring whether PACE’s financial support is in line with the cost to complete the project.
Additionally, PACE implement the following practices for program effectiveness and efficiency.
- Target capacity-building to the areas of the most need. CCAT data collected in this engagement provide information on which capacities and subcapacities need improvement. PACE should pay attention to findings in this attached presentation as well as capacities and subcapacities that are slowest to change over time. Target capacity-building support to organizations and capacities that are slowest to improve.
- “Match up” organizations that do well in areas with those who are not doing as well. Four participant organizations scored “Strong” in Technical Capacity, but many do not. Those that scored well may have additional insight to share on ways to effectively and efficiently increase certain capacities. Utilize CCAT data to identify which organizations should be matched.
- Conduct outreach to communicate that PACE is filling a niche and effectively building capacity. Evaluation findings indicate that PACE participants are experiencing capacity improvements where we would expect to see them. Findings also indicate that the PACE IS program occupies a distinct niche, but that some nonprofits are unaware of the efforts of capacity-building organizations in the area. Conduct outreach to nonprofits and funders to communicate the PACE niche and the metrics on PACE’s effectiveness.