TB Notes Newsletter
No. 2, 2006
HIGHLIGHTS FROM STATE AND LOCAL PROGRAMS
Lessons Learned in the Process of Evaluation
Last year, the Metropolitan Chicago TB Consortium, the State of
Illinois, and the City of Chicago
TB program agreed to conduct a program evaluation as part of the
Tuberculosis Epidemiologic Studies Consortium (TBESC) Task Order
15. During our process of implementing this evaluation, we learned
the following lessons we’d like to share with others.
Lesson 1: Allay fears about the evaluation process.
We found that it was critical for all divisions or parties that
will be affected by the evaluation to have a sense of security about
what was being done. To allay fears and clear up misunderstandings,
a process of education was put in place. Education and training
sessions were arranged, and an introduction to the evaluation process
was begun. A fair amount of time was directed toward reassuring
staff that a program evaluation such as the one we were undertaking
was not the same as a personnel review. As the staff members were
educated on the concept of evaluation, they became a little more
relaxed about it. We went through a step-by-step process of describing
what was involved, and how it would require some time to discuss
what the evaluation would accomplish. As we discussed the whole
issue of stakeholders, it became clearer that the TB staff themselves
were the major stakeholders, especially when it came to who had
the most impact and who could effect change.
Lesson 2: Develop a description that sets the stage for the
evaluation. A significant amount of time was devoted
to developing a comprehensive description of the TB program. The
description included a discussion of how the TB program operates,
number of clinics, staff, models of operation, relations within
the department, relations with private providers, who oversees DOT,
who manages the case, and how reporting works. We also included
a description of the system from the patient’s perspective. We described
the process that a patient goes through prior to being diagnosed
with an active case. This description helped us narrow our ideas
and select “completion of therapy” as the objective for evaluation.
Lesson 3: Take time to determine the program’s stakeholders.
Determining stakeholders is a vital part of the entire evaluation
and can be improperly used when determined in haste. Discussions
about who would be considered a stakeholder needed to be held at
several different levels, both for input and buy-in and to ensure
that we were not inadvertently missing a critical participant. Determining
the stakeholders required extensive discussions, since both community
members and health department employees have a stake in patients’
completion of treatment. Owing to the nature of the disease, it
seems at first glance that stakeholders are almost everyone who
may be impacted as well as an assortment of constituencies. Narrowing
down the group of people to those who really have a stake in what
is being evaluated is a major element in addressing the problem
to be evaluated. It is necessary to determine not only who will
be affected but, more importantly, who can do something about an
identified situation and who can remedy any identified problems.
It is crucial to understand the layers, if they exist, of the persons
involved; a supervisor may understand that a problem exists or a
physician may know what the specific symptom means, but it may require
additional personnel to fix the problem or to set up therapy and
manage the care of the patient.
The stakeholders were all a bit apprehensive at being part of the
process and were concerned about how it would reflect on them. However,
they were guaranteed anonymity.
Lesson 4: Program context and resources available for program
evaluation are important. The TB program is a division of the
City Department of Public Health (CDPH), with major funding coming
from both CDC and CDPH corporate funding. The TB control program
is currently short-staffed owing to an early retirement in April
2004. At the time the evaluation began, there was no one available
for or dedicated to program evaluation and performance standards.
Lesson 5: Program development can be affected by the evaluation
process. We found that the process for implementing the
evaluation—engaging stakeholders and developing a description that
encompasses multiple perspectives—provided us with some valuable
information about our program. For us, the evaluation questions
became part of the evidence. The evaluation questions led to very
specific thoughts on where and how to proceed. It told us some of
the specific items that were amenable to change and that would make
an immediate difference. This information has already led to some
changes that are creating improvements. For example,
- Not all of the TB clinics were using the same form for completion
of therapy. That inconsistency was immediately addressed, and
one citywide clinical summary sheet is now being used.
- Legal action is being processed into policy that can be used
by all clinics when there is patient failure in DOT programs.
- The state and city are working together to have consistent reports
that meet reporting needs, and the staff are beginning to understand
the necessity for focused coordination.
- Written policies and procedures for data surveillance and data
assurance are being developed.
- Quarterly cohort reviews are being put in place and will be
coordinated with other quarterly reports for CDPH.
- Annual staff evaluations will be developed reflecting duties
that lead to achievement of program goals.
Lesson 6: Program evaluations have the capacity to detect both
small and large problems. Although we are still in the
process of analyzing data, our early analysis and interpretation
of findings concluded that more education and training in the area
of evaluation needed to be in place. Additionally, the findings
are still being explored, but areas that needed immediate attention
—Reported by Phyllis J. Handelman, Evaluator, Handelman
Susan Lippold, MD, TB Medical Director, CDPH
Mike Arbise, Director, State of Illinois TB Program
Released October 2008
Centers for Disease Control and Prevention
National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention
Division of Tuberculosis Elimination - http://www.cdc.gov/tb
Please send comments/suggestions/requests
CDC/Division of Tuberculosis Elimination
Communications, Education, and Behavioral Studies Branch
1600 Clifton Rd., NE - Mailstop E-10, Atlanta, GA 30333