CDC Logo Tuberculosis Information CD-ROM   Image of people
     
jump over main navigation bar to content area
Home
TB Guidelines
Surveillance Reports
Slide Sets
TB-Related MMWRs and Reports
Education/Training Materials
Newsletters
Ordering Information
Help

 

U.S. Department of Health and Human Services

  

This is an archived document. The links are no longer being updated.

TB Notes 4, 2004

No. 4, 2004

Evaluation Capacity Building:

The Case of the TB Evaluation Working Group

In March 2003, DTBE formed an Evaluation Working Group (EWG). By May of the same year, EWG membership included program staff from several of the 68 sites funded by DTBE, representatives from the National TB Controllers Association (NTCA), and DTBE headquarters staff. Spearheaded by Dr. Mark Lobato and Maureen Wilce of DTBE, this group of over 20 active members was charged with developing a national 5-year strategic plan to improve program evaluation capacity at the state and local level. The group plans to ensure implementation of a comprehensive approach to building capacity through participatory evaluation of program performance.

Evaluation capacity building is different from, yet contributes greatly to, program evaluation. Program evaluation is “the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development.”1 As defined by Compton, Baizerman, and Stockdill, evaluation capacity building is “the intentional work to continuously create and sustain overall organizational processes that make quality evaluation and its uses routine.”2 Based on this definition, evaluation capacity building can be described as both a mandatory and supportive process needed to create an environment in which quality program evaluations can be performed. Of the frameworks for describing ECB that have been proposed in the past decade, Milstein’s and Cotton’s3 includes five elements that are demonstrated through the work of the EWG:

  1. Motivational forces
  2. Organizational environment
  3. Workforce and professional development
  4. Resources and supports
  5. Learning from experience

Motivational Forces

Milstein and Cotton simply define motivational forces as the reasons why evaluation happens. In 1997, CDC charged an agency-wide Evaluation Working Group with identifying and organizing the essential elements of program evaluation and with promoting program evaluation both within the agency and throughout the public health system. Two major products of this working group were released in 1999: the CDC Framework for Program Evaluation in Public Health1 and a set of recommendations for engaging in program evaluation at CDC and among programs funded by CDC. In addition to this new agency-wide emphasis on program evaluation, the Institute of Medicine recommended in 2000 that all TB programs regularly evaluate their performance.4

As a result of these recommendations, DTBE has included program evaluation as a required recipient activity in its 2005 cooperative agreements. In response, the DTBE EWG is working to prepare state and local programs for this requirement by developing an evaluation toolkit and evaluation expertise at the federal, state, and local levels. Already, TB programs assess their progress toward meeting national objectives with each funding cycle. They also routinely monitor their activities and submit surveillance information to CDC using the Report of Verified Case of Tuberculosis (RVCT) form. The evaluation toolkit is designed to improve evaluation capacity at the program level in order to turn those data into justified recommendations for program improvement.

Organizational Environment

The organizational environment describes the characteristics and setting in which capacity building is undertaken. Institutional support for program evaluation is not only demonstrated by the agency-wide Evaluation Working Group, but also at all levels of TB leadership. With assistance from NTCA, the DTBE EWG issued a web-based evaluation needs assessment to 59 state and local programs in September 2003 with a 92% response rate; 72% of respondents ranked evaluation as being “very beneficial” and only 17% listed “staff don’t value evaluation” as being a barrier to evaluation. These findings suggest that state and local programs are receptive to program evaluation. In addition, the presence of state and local TB program staff on the EWG implies that their respective managers are also supportive of their participation. This support was further evidenced at the 2004 NTCA meeting, during which an evening session to share the progress of the EWG and solicit additional participation drew a capacity crowd and several commendations for the EWG efforts.

As encouraging as the NTCA meeting was, concerted efforts need to be taken to ensure that a receptive organizational environment is maintained at both local or state and federal levels. To accomplish this, “evaluation champions” must emerge who consistently promote evaluation and ECB.  EWG members from DTBE, state, and local TB programs have served as champions in their respective communities, sharing the progress of the EWG, encouraging others to join, and reassessing evaluation activities at their sites. The membership of the EWG is held open so that as people in programs around the country become interested and available, they are encouraged to contribute. These actions help to ensure commitment to ECB from senior management all the way through local program staff. A logic model created by EWG members details the efforts that will be taken to generate a commitment to evaluation from state and local leadership.

Workforce and Professional Development

Workforce and professional development refers to the skills of professional evaluators and of other stakeholders who participate in the evaluation. The EWG has several members who are professionally trained evaluators. In addition, an outside contractor with expertise in ECB was hired to guide each phase of the 5-year plan and to support development of evaluation skills in the employees who will work with the EWG.

Among the capacity-building objectives of the EWG is for evaluation training to be offered to TB program staff through multiple venues. The EWG plans to compile existing training materials, supplement them as needed, and design a training course that can be operated nationwide. The final program evaluation training will include guidance on using the evaluation toolkit and will be offered at DTBE program manager courses, regional TB meetings, and other appropriate venues. Furthermore, the EWG encourages an assortment of TB staff to be trained in evaluation so that evaluation expertise is not limited to one person. Instead, multiple program staff should be trained to perform program evaluations in order to maximize the program’s capacity.

In addition to evaluation training, technical assistance will be available to all states and locales as well as stakeholders through a peer network by summer 2006. The members of the peer network will share TB-specific evaluation guidance so as to provide support for programs as they plan and perform evaluations. In establishing the peer network, the EWG expects to improve the evaluation capacity at the state and local levels.

Resources and Supports

Resources and supports are the tools evaluators use to do their work. In order to build evaluation capacity at the state and local level, the EWG is developing an evaluation toolkit that will include references, guidance, and evaluation tools and indicators that programs can use or adapt. Also included in the toolkit will be instructions and sources of technical assistance so that programs will be able to plan and implement their own evaluations.

As the toolkit is being developed, support for EWG members is available in the form of an eRoom. The eRoom is a password-secure, interactive website that stores and posts documents, provides a bulletin board for announcements, and offers meeting planning features. Through the eRoom, the EWG is exploring ways to use Web-based technology to engage people interested in evaluation, regardless of home base. For example, EWG Tools Team members located around the country have come together in the eRoom to assign tasks, report progress, and post findings. Through this interactive forum, the EWG is more likely to not only attract and retain members, but also to share experiences and “evaluation wisdom,” thereby helping to build capacity within the EWG.

Learning from Experience

Sharing lessons learned from the evaluation process is the final element in this framework for ECB. As mentioned earlier, the EWG wants to ensure that TB program evaluators share lessons learned through a peer network. Since many programs’ evaluation questions will be specific to their site (for example, How successful was the peer education program in reducing the incidence of TB in the US-born African-American community in Fulton County?), it is important for evaluation skills to be shared within programs as well. It is for this reason that the intended audience for the evaluation toolkit and evaluation trainings include program staff at all levels.

Ensuring Participation and Sustainability

Though not explicitly mentioned in the five-element framework presented by Milstein and Cotton,3 evaluation capacity building is also critical for ensuring participation and sustainability in evaluation efforts. All stakeholders in the TB program should be involved in the evaluation and should be the beneficiaries of ECB efforts. In order to accomplish this, evaluation skill, knowledge, and dedication should not rest with one person.  What if that person quits?  What if that person is an outside evaluator?  What if that person is unable to make changes based on the results of the evaluation in order to improve the program?  By continually building capacity at all levels within an organization, or in the case of the TB EWG, between organizations (CDC, state programs, and local programs), programs become empowered to perform tailored evaluations through a participatory process.

Conclusions and Next Steps

Despite being described separately, the five elements of Milstein’s and Cotton’s framework and the need to ensure participation and sustainability are interrelated; the level of development of any one of the elements impacts the others. For example, the level of institutional support as described by the organizational environment determines the allocation of resources and support offered by management, which in turn impacts the participation and professional development generated. State and local programs are encouraged to consider each of the elements of evaluation capacity building and how they impact each other when planning and conducting program evaluations.

As the Evaluation Working Group enters Phase II of its 5-year plan, during which time the EWG expects to complete development of the toolkit and proceed to pilot testing, membership in the EWG remains open. You are encouraged to contact Maureen Wilce if you would like to be involved. All levels of participation are welcome!

—Submitted by Carrie Bridges, MPH
Public Health Prevention Service Fellow
Div of TB Elimination

References

  1. CDC. Framework for program evaluation in public health. MMWR 1999; 48(RR-11): 1-40.
  2. Compton D, Baizerman M, and Stockdill S, Eds. The Art, Craft, and Science of Evaluation Capacity Building. New Directions for Evaluation.  San Francisco: Jossey-Bass; Spr 2002: 93.
  3. Milstein B and Cotton D. Defining Concepts for the Presidential Strand on Building Evaluation Capacity [online]. 2000. Evaluation 2000 conference information. Available from www.eval.org/eval2000/public/presstrand.pdf
  4. Institute of Medicine. Ending Neglect: The Elimination of Tuberculosis in the United States. Washington, DC: National Academy Press; 2000.

 


Released October 2008
Centers for Disease Control and Prevention
National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention
Division of Tuberculosis Elimination - http://www.cdc.gov/tb

Please send comments/suggestions/requests to: hsttbwebteam@cdc.gov, or to
CDC/Division of Tuberculosis Elimination
Communications, Education, and Behavioral Studies Branch
1600 Clifton Rd., NE - Mailstop E-10, Atlanta, GA 30333