CDC Logo Tuberculosis Information CD-ROM   Image of people
     
jump over main navigation bar to content area
Home
TB Guidelines
Surveillance Reports
Slide Sets
TB-Related MMWRs and Reports
Education/Training Materials
Newsletters
Ordering Information
Help

 

U.S. Department of Health and Human Services

  
jump over right navigation bar
TB Notes 2, 2006
Introduction
Highlights from State and Local Programs
  An Outbreak Response in a Rural, Southwest Missouri County Jail
  No Reported TB Cases in Wyoming in 2005
  Suffolk County (New York) Targeted TB Testing and Treatment Program Among the Foreign-born, 2000–2004
  The Changing Epidemiology of TB in Connecticut, 2000-2004
  Molecular Genotyping of Mycobacterium tuberculosis in Connecticut
  Third Annual Conference on TB in the U.S. Pacific Islands: Meeting Highlights, Challenges, and Solutions for Addressing the Disparities
  "Update: Tuberculosis Nursing" Workshop in Hawaii
  Lessons Learned in the Process of Evaluation – Illinois
  TB Education and Targeted Testing of Garfield County, Colorado, WIC Clients
Laboratory Updates
TB Education and Training Network Updates
Communications, Education, and Behavioral Studies Branch Updates
Information Technology and Statistics Branch Update
SEOIB Updates
New CDC Publications
Personnel Notes
Calendar of Events
 
Return to Table of Contents

TB Notes Newsletter

No. 2, 2006

HIGHLIGHTS FROM STATE AND LOCAL PROGRAMS

Lessons Learned in the Process of Evaluation

Last year, the Metropolitan Chicago TB Consortium, the State of Illinois, and the City of Chicago TB program agreed to conduct a program evaluation as part of the Tuberculosis Epidemiologic Studies Consortium (TBESC) Task Order 15. During our process of implementing this evaluation, we learned the following lessons we’d like to share with others.

Lesson 1:  Allay fears about the evaluation process. We found that it was critical for all divisions or parties that will be affected by the evaluation to have a sense of security about what was being done. To allay fears and clear up misunderstandings, a process of education was put in place. Education and training sessions were arranged, and an introduction to the evaluation process was begun. A fair amount of time was directed toward reassuring staff that a program evaluation such as the one we were undertaking was not the same as a personnel review. As the staff members were educated on the concept of evaluation, they became a little more relaxed about it. We went through a step-by-step process of describing what was involved, and how it would require some time to discuss what the evaluation would accomplish. As we discussed the whole issue of stakeholders, it became clearer that the TB staff themselves were the major stakeholders, especially when it came to who had the most impact and who could effect change. 

Lesson 2:  Develop a description that sets the stage for the evaluation. A significant amount of time was devoted to developing a comprehensive description of the TB program. The description included a discussion of how the TB program operates, number of clinics, staff, models of operation, relations within the department, relations with private providers, who oversees DOT, who manages the case, and how reporting works. We also included a description of the system from the patient’s perspective. We described the process that a patient goes through prior to being diagnosed with an active case. This description helped us narrow our ideas and select “completion of therapy” as the objective for evaluation.

Lesson 3:  Take time to determine the program’s stakeholders. Determining stakeholders is a vital part of the entire evaluation and can be improperly used when determined in haste. Discussions about who would be considered a stakeholder needed to be held at several different levels, both for input and buy-in and to ensure that we were not inadvertently missing a critical participant. Determining the stakeholders required extensive discussions, since both community members and health department employees have a stake in patients’ completion of treatment. Owing to the nature of the disease, it seems at first glance that stakeholders are almost everyone who may be impacted as well as an assortment of constituencies. Narrowing down the group of people to those who really have a stake in what is being evaluated is a major element in addressing the problem to be evaluated. It is necessary to determine not only who will be affected but, more importantly, who can do something about an identified situation and who can remedy any identified problems. It is crucial to understand the layers, if they exist, of the persons involved; a supervisor may understand that a problem exists or a physician may know what the specific symptom means, but it may require additional personnel to fix the problem or to set up therapy and manage the care of the patient.

The stakeholders were all a bit apprehensive at being part of the process and were concerned about how it would reflect on them. However, they were guaranteed anonymity.

Lesson 4:  Program context and resources available for program evaluation are important. The TB program is a division of the City Department of Public Health (CDPH), with major funding coming from both CDC and CDPH corporate funding. The TB control program is currently short-staffed owing to an early retirement in April 2004. At the time the evaluation began, there was no one available for or dedicated to program evaluation and performance standards.

Lesson 5:  Program development can be affected by the evaluation process. We found that the process for implementing the evaluation—engaging stakeholders and developing a description that encompasses multiple perspectives—provided us with some valuable information about our program. For us, the evaluation questions became part of the evidence. The evaluation questions led to very specific thoughts on where and how to proceed. It told us some of the specific items that were amenable to change and that would make an immediate difference. This information has already led to some changes that are creating improvements. For example,

  • Not all of the TB clinics were using the same form for completion of therapy. That inconsistency was immediately addressed, and one citywide clinical summary sheet is now being used.
  • Legal action is being processed into policy that can be used by all clinics when there is patient failure in DOT programs.
  • The state and city are working together to have consistent reports that meet reporting needs, and the staff are beginning to understand the necessity for focused coordination.
  • Written policies and procedures for data surveillance and data assurance are being developed.
  • Quarterly cohort reviews are being put in place and will be coordinated with other quarterly reports for CDPH.
  • Annual staff evaluations will be developed reflecting duties that lead to achievement of program goals.

Lesson 6:  Program evaluations have the capacity to detect both small and large problems. Although we are still in the process of analyzing data, our early analysis and interpretation of findings concluded that more education and training in the area of evaluation needed to be in place. Additionally, the findings are still being explored, but areas that needed immediate attention were found.

—Reported by Phyllis J. Handelman, Evaluator, Handelman Consulting Ltd.
Susan Lippold, MD, TB Medical Director, CDPH
Mike Arbise, Director, State of Illinois TB Program

 


Released October 2008
Centers for Disease Control and Prevention
National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention
Division of Tuberculosis Elimination - http://www.cdc.gov/tb

Please send comments/suggestions/requests to: hsttbwebteam@cdc.gov, or to
CDC/Division of Tuberculosis Elimination
Communications, Education, and Behavioral Studies Branch
1600 Clifton Rd., NE - Mailstop E-10, Atlanta, GA 30333