This is an archived document. The links are no longer being updated.
TB Notes 4, 2004
No. 4, 2004
Evaluation Capacity Building:
The Case of the TB Evaluation Working Group
In March 2003, DTBE formed an Evaluation Working Group (EWG). By
May of the same year, EWG membership included program staff from
several of the 68 sites funded by DTBE, representatives from the
National TB Controllers Association (NTCA), and DTBE headquarters
staff. Spearheaded by Dr. Mark Lobato and Maureen Wilce of DTBE,
this group of over 20 active members was charged with developing
a national 5-year strategic plan to improve program evaluation capacity
at the state and local level. The group plans to ensure implementation
of a comprehensive approach to building capacity through participatory
evaluation of program performance.
Evaluation capacity building is different from, yet contributes
greatly to, program evaluation. Program evaluation is “the systematic
collection of information about the activities, characteristics,
and outcomes of programs to make judgments about the program, improve
program effectiveness, and/or inform decisions about future program
development.”1 As defined by Compton, Baizerman, and
Stockdill, evaluation capacity building is “the intentional work
to continuously create and sustain overall organizational processes
that make quality evaluation and its uses routine.”2
Based on this definition, evaluation capacity building can be described
as both a mandatory and supportive process needed to create an environment
in which quality program evaluations can be performed. Of the frameworks
for describing ECB that have been proposed in the past decade, Milstein’s
and Cotton’s3 includes five elements that are demonstrated
through the work of the EWG:
- Motivational forces
- Organizational environment
- Workforce and professional development
- Resources and supports
- Learning from experience
Milstein and Cotton simply define motivational forces as the reasons
why evaluation happens. In 1997, CDC charged an agency-wide Evaluation
Working Group with identifying and organizing the essential elements
of program evaluation and with promoting program evaluation both
within the agency and throughout the public health system. Two major
products of this working group were released in 1999: the CDC Framework
for Program Evaluation in Public Health1 and a set of
recommendations for engaging in program evaluation at CDC and among
programs funded by CDC. In addition to this new agency-wide emphasis
on program evaluation, the Institute of Medicine recommended in
2000 that all TB programs regularly evaluate their performance.4
As a result of these recommendations, DTBE has included program
evaluation as a required recipient activity in its 2005 cooperative
agreements. In response, the DTBE EWG is working to prepare state
and local programs for this requirement by developing an evaluation
toolkit and evaluation expertise at the federal, state, and local
levels. Already, TB programs assess their progress toward meeting
national objectives with each funding cycle. They also routinely
monitor their activities and submit surveillance information to
CDC using the Report of Verified Case of Tuberculosis (RVCT) form.
The evaluation toolkit is designed to improve evaluation capacity
at the program level in order to turn those data into justified
recommendations for program improvement.
The organizational environment describes the characteristics and
setting in which capacity building is undertaken. Institutional
support for program evaluation is not only demonstrated by the agency-wide
Evaluation Working Group, but also at all levels of TB leadership.
With assistance from NTCA, the DTBE EWG issued a web-based evaluation
needs assessment to 59 state and local programs in September 2003
with a 92% response rate; 72% of respondents ranked evaluation as
being “very beneficial” and only 17% listed “staff don’t value evaluation”
as being a barrier to evaluation. These findings suggest that state
and local programs are receptive to program evaluation. In addition,
the presence of state and local TB program staff on the EWG implies
that their respective managers are also supportive of their participation.
This support was further evidenced at the 2004 NTCA meeting, during
which an evening session to share the progress of the EWG and solicit
additional participation drew a capacity crowd and several commendations
for the EWG efforts.
As encouraging as the NTCA meeting was, concerted efforts need
to be taken to ensure that a receptive organizational environment
is maintained at both local or state and federal levels. To accomplish
this, “evaluation champions” must emerge who consistently promote
evaluation and ECB. EWG members from DTBE, state, and local
TB programs have served as champions in their respective communities,
sharing the progress of the EWG, encouraging others to join, and
reassessing evaluation activities at their sites. The membership
of the EWG is held open so that as people in programs around the
country become interested and available, they are encouraged to
contribute. These actions help to ensure commitment to ECB from
senior management all the way through local program staff. A logic
model created by EWG members details the efforts that will be taken
to generate a commitment to evaluation from state and local leadership.
Workforce and Professional Development
Workforce and professional development refers to the skills of
professional evaluators and of other stakeholders who participate
in the evaluation. The EWG has several members who are professionally
trained evaluators. In addition, an outside contractor with expertise
in ECB was hired to guide each phase of the 5-year plan and to support
development of evaluation skills in the employees who will work
with the EWG.
Among the capacity-building objectives of the EWG is for evaluation
training to be offered to TB program staff through multiple venues.
The EWG plans to compile existing training materials, supplement
them as needed, and design a training course that can be operated
nationwide. The final program evaluation training will include guidance
on using the evaluation toolkit and will be offered at DTBE program
manager courses, regional TB meetings, and other appropriate venues.
Furthermore, the EWG encourages an assortment of TB staff to be
trained in evaluation so that evaluation expertise is not limited
to one person. Instead, multiple program staff should be trained
to perform program evaluations in order to maximize the program’s
In addition to evaluation training, technical assistance will
be available to all states and locales as well as stakeholders through
a peer network by summer 2006. The members of the peer network will
share TB-specific evaluation guidance so as to provide support for
programs as they plan and perform evaluations. In establishing the
peer network, the EWG expects to improve the evaluation capacity
at the state and local levels.
Resources and Supports
Resources and supports are the tools evaluators use to do their
work. In order to build evaluation capacity at the state and local
level, the EWG is developing an evaluation toolkit that will include
references, guidance, and evaluation tools and indicators that programs
can use or adapt. Also included in the toolkit will be instructions
and sources of technical assistance so that programs will be able
to plan and implement their own evaluations.
As the toolkit is being developed, support for EWG members is
available in the form of an eRoom. The eRoom is a password-secure,
interactive website that stores and posts documents, provides a
bulletin board for announcements, and offers meeting planning features.
Through the eRoom, the EWG is exploring ways to use Web-based technology
to engage people interested in evaluation, regardless of home base.
For example, EWG Tools Team members located around the country have
come together in the eRoom to assign tasks, report progress, and
post findings. Through this interactive forum, the EWG is more likely
to not only attract and retain members, but also to share experiences
and “evaluation wisdom,” thereby helping to build capacity within
Learning from Experience
Sharing lessons learned from the evaluation process is the final
element in this framework for ECB. As mentioned earlier, the EWG
wants to ensure that TB program evaluators share lessons learned
through a peer network. Since many programs’ evaluation questions
will be specific to their site (for example, How successful was
the peer education program in reducing the incidence of TB in the
US-born African-American community in Fulton County?), it is important
for evaluation skills to be shared within programs as well. It is
for this reason that the intended audience for the evaluation toolkit
and evaluation trainings include program staff at all levels.
Ensuring Participation and Sustainability
Though not explicitly mentioned in the five-element framework
presented by Milstein and Cotton,3 evaluation capacity
building is also critical for ensuring participation and sustainability
in evaluation efforts. All stakeholders in the TB program should
be involved in the evaluation and should be the beneficiaries of
ECB efforts. In order to accomplish this, evaluation skill, knowledge,
and dedication should not rest with one person. What if that
person quits? What if that person is an outside evaluator?
What if that person is unable to make changes based on the results
of the evaluation in order to improve the program? By continually
building capacity at all levels within an organization, or in the
case of the TB EWG, between organizations (CDC, state programs,
and local programs), programs become empowered to perform tailored
evaluations through a participatory process.
Conclusions and Next Steps
Despite being described separately, the five elements of Milstein’s
and Cotton’s framework and the need to ensure participation and
sustainability are interrelated; the level of development of any
one of the elements impacts the others. For example, the level of
institutional support as described by the organizational environment
determines the allocation of resources and support offered by management,
which in turn impacts the participation and professional development
generated. State and local programs are encouraged to consider each
of the elements of evaluation capacity building and how they impact
each other when planning and conducting program evaluations.
As the Evaluation Working Group enters Phase II of its 5-year
plan, during which time the EWG expects to complete development
of the toolkit and proceed to pilot testing, membership in the EWG
remains open. You are encouraged to contact Maureen Wilce if you
would like to be involved. All levels of participation are welcome!
—Submitted by Carrie Bridges, MPH
Public Health Prevention Service Fellow
Div of TB Elimination
- CDC. Framework for program evaluation
in public health. MMWR 1999; 48(RR-11): 1-40.
- Compton D, Baizerman M, and Stockdill
S, Eds. The Art, Craft, and Science of Evaluation Capacity
Building. New Directions for Evaluation. San Francisco:
Jossey-Bass; Spr 2002: 93.
- Milstein B and Cotton D. Defining
Concepts for the Presidential Strand on Building Evaluation Capacity
[online]. 2000. Evaluation 2000 conference information.
Available from www.eval.org/eval2000/public/presstrand.pdf
- Institute of Medicine. Ending
Neglect: The Elimination of Tuberculosis in the United
States. Washington, DC: National Academy Press; 2000.