Analytic Approaches Courses
- Applied Regression Analysis for Evaluators
- Applied Statistics for Evaluators
- Hierarchical Linear Modeling
- Introduction to Cost-Benefit and Cost-Effectiveness Analysis
- Intermediate Cost-Benefit and Cost- Effectiveness Analysis
- Needs Assessment
- Practical Meta-Analysis
- Qualitative Data Analysis
- Intermediate Qualitative Data Analysis
- Social and Organizational Network Analysis
Applied Regression Analysis for Evaluators
Instructor: Gary T. Henry, PhD
Description: Evaluators often face the situation where program outcomes vary across different participants and they want to explain those differences. To understand the contribution of the program to the outcomes, it is often necessary to control for the influence of other factors. In these situations, regression analysis is the most widely used statistical tool for evaluators to apply. The objective of this course is to describe and provide hands-on experience in conducting regression analysis, and to aid participants in interpreting regression results in an evaluation context. The course begins with a review of hypothesis testing (t-tests) and a non-mathematical explanation of how the regression line is computed for bivariate regression. A major focus is on accurately interpreting regression coefficients and tests of significance, including the slope of the line, the t-statistic, and the statistics that measure how well the regression line fits the data. Participants will also learn how to find outliers that may be unduly influencing the results. Participants will have opportunity to estimate multivariate regression models on cross-sectional data; diagnose the results to determine if they may be misleading; and test the effects of program participation with pretest-posttest and posttest-only data. Regression-based procedures for testing mediated and moderated effects will be covered. On the third day, students will be given the opportunity to conduct an independent analysis and write-up the findings. Both peer feedback and instructor feedback will be provided to build skills in interpreting findings and explaining them to interested audiences. Participants will use SPSS software to compute regression analyses and given opportunity to apply it on data from an actual evaluation. Students and instructor will work on interpreting the results and determining how to present them to evaluation audiences. The class will be in a lab where each person has a computer for application of content.
Applied Statistics for Evaluators
Instructor: Theodore H. Poister, PhD
In this class students will become familiar with a set of statistical tools that are often used in program evaluation, with a strong emphasis on appropriate application of techniques and interpretation of results. It is designed to “demystify” statistics and clarify how and when to use particular techniques. While the principal concern focuses on practical applications in program evaluations rather than the mathematical support underlying the procedures, a number of formulas and computations are covered to help students understand the logic of how the statistics work. Topics include introduction to data analysis; simple descriptive statistics; examination of statistical relationships; the basics of statistical inference from sample data; two-sample t tests, chi square and associated measures; analysis of variance; and an introduction to simple and multiple regression analysis.
Students will learn how to generate a wide variety of tables and graphs for presenting results, and a premium will be placed on clear presentation and interpretation of results. This “hands-on” class is conducted in a computer lab in which each participant has a computer for running statistical procedures on a wide range of real-world data sets, using SPSS software. However, no prior knowledge of statistics or SPSS is required. While this is an introductory course, it can also serve as a refresher for those with some training in statistics, and an “eye opener” for evaluators who are working with statistics now but are not comfortable with when and how they should be used.
Hierarchical Linear Modeling
Instructor: Gary T. Henry, PhD
Description: In many evaluations, the program participants are nested within sites, schools, or groups. In addition, the nesting is sometimes multi-leveled, such as students within classes within schools within school districts. To make matters more complicated, we more frequently have multiple observations taken over time on the program participants, such as years of student achievement scores or measures of mental health status. Hierarchical linear models (HLM) have been developed to accurately analyze these types of data. These models make two important improvements over regular (ordinary least squares) regression. First, the standard errors that are used for testing statistical significance are corrected for the “nesting” or “clustering” of participants into groups. Usually, the participants in a “cluster” are more similar to each other than they are to participants in other “clusters” and this, when uncorrected, deflates the standard errors leading to “false positives” or concluding that a coefficient is statistically significant when it is not. HLM corrects the standard errors and test of statistical significance for nested data. Second, HLM appropriately apportions the variance that occurs at each level to that level, and provides realistic estimates of the effects across levels. In this course, we lay a foundation for understanding, using, and interpreting HLM. We begin with multiple regression, including the assumptions that must be fulfilled for the coefficients and tests of statistical significance to be unbiased. Using a step by step approach, we will introduce the basic concepts of HLM and the notation that has been developed for presenting HLM models. We will focus on practical aspects of the use of HLM and correctly putting the findings into language suitable for a report. The main objective of the course is to provide the participants with a better understanding of HLM, how it can improve the analysis of data in many evaluations, and how to read and interpret reports and articles that utilize it. The course will not offer hands on experiences writing and implementing HLM statistical programs.
Intermediate Cost-Benefit and Cost-Effectiveness Analysis
Instructor: Joseph Cordes, PhD
Description: The Intermediate Cost-Benefit Analysis course provides a more advanced and detailed review of the principles of social cost and social benefit estimation than is provided in TEI’s Introduction to Cost-Benefit and Cost Effectiveness Analysis. Working with the instructor, students will undertake hands-on estimation of the costs and benefits of actual programs in the computer lab. The objective is to develop the ability both to critically evaluate and use cost-benefit analyses of programs in the public and nonprofit sectors, and to use basic cost-benefit analysis tools to actively undertake such analyses. Topics covered in the course will include:
I. Principles of Social Cost and Social Benefit Estimation
- Social Cost Estimation: (a) Components (capital, operating, administrative); (b) Budgetary and Social Opportunity Cost
- Social Benefit Estimation: (a) Social vs. private benefits; (b) revealed benefit measures (Price/cost changes in primary market, Price/cost changes in analogous markets, Benefits inferred from market-trade-offs, and cost/damages avoided as benefit measures)
- Stated preference measures: Inferring benefits from survey data
- Benefit/Cost Transfer: Borrowing estimates of benefits and costs from elsewhere.
- Timing of Benefits and Costs: (a) Discounting and net present value, (b) Dealing with inflation, (c). Choosing a discount rate
- Presenting Results: (a) Sensitivity analysis (partial sensitivity analysis, best/worst case scenarios, break-even analysis, and Monte-Carlo analysis); (b) Present value of net social benefits, (c) Benefit Cost Ratio, (d) Internal rate of Return
II. Social Cost and Social Benefit Estimation in Practice
The use of the above principles of cost and benefit estimation will be illustrated using data drawn from several actual benefit cost analysis of real programs. The cases will be chosen to illustrate the application of the benefit/cost estimation principles in the case of social programs, health programs, and environmental programs. Working with the instructor in the computer lab, students will create a benefit-cost analysis template and then use that template to estimate social benefits and social costs, and to present a benefit-cost bottom line.
Prerequisites: This is an intermediate level course. Participants are assumed to have some knowledge/or experience with cost-benefit and/or cost-effectiveness analysis equivalent to the TEI course Introduction to Cost-Benefit and Cost-Effectiveness Analysis.
Intermediate Qualitative Data Analysis
Instructor: Delwyn Goodrick, PhD
Description: Data analysis involves creativity, sensitivity and rigor. In its most basic form qualitative data analysis involves some sort of labeling, coding and clustering in order to make sense of data collected from evaluation fieldwork, interviews, and/or document analysis. This intermediate level workshop builds on basic coding and categorizing familiar to most evaluators, and extends the array of strategies available to support rigorous interpretations. This workshop presents an array of approaches to support the analysis of qualitative data with an emphasis on procedures for the analysis of interview data. Strategies such as, thematic analysis, pattern matching, template analysis, process tracing, schema analysis and qualitative comparative analysis are outlined and illustrated with reference to examples from evaluation and from a range of disciplines, including sociology, education, political science and psychology. The core emphasis in the workshop is creating awareness of heuristics that support selection and application of appropriate analytic techniques that match the purpose of the evaluation, type of data, and practical considerations such as resource constraints. While a brief overview of qualitative analysis software is provided, the structure of the workshop focuses on analysis using manual methods. A range of activities to support critical thinking and application of principles is integrated within the workshop program. Qualitative data analysis and writing go hand in hand. In the second part of the workshop strategies for transforming analysis through processes of description, interpretation and judgment will be presented. These issues are particularly important in the assessment of the credibility of qualitative evidence by evaluation audiences. Issues of quality, including validity, trustworthiness and authenticity of qualitative data are integrated throughout the workshop.
Participants will receive a text, Qualitative Data Analysis: Practical Strategies by Patricia Bazeley (Sage, 2013) to support learning within and beyond the workshop. Specific issues to be addressed:
- What are the implications of an evaluator’s worldview for selection of qualitative data analysis (QDA) strategies?
- Are there analytic options that are best suited to particular kinds of qualitative data?
- How can participant experiences be portrayed through QDA without fracturing the data through formal coding?
- What types of analysis may be appropriate for particular types of evaluation (program theory, realist, transformative)
- What strategies can be used to address interpretive dissent when working in evaluation teams?
- What are some ways that qualitative and quantitative findings can be integrated in an evaluation report?
- How can I sell the value of qualitative evidence to evaluation audiences?
Prerequisites: This is an intermediate level course. Participants are assumed to have some knowledge/or experience with qualitative data.
Introduction to Cost-Benefit and Cost-Effectiveness Analysis
Instructor: Clive Belfield, PhD
Description: The tools and techniques of benefit-cost and cost-effectiveness analysis will be presented. The goal of the course is to provide analysts with the skills to interpret benefit-cost and cost-effectiveness analyses. Content includes: identification and measurement of costs using the ingredients method; how to specify effectiveness; shadow pricing for benefits using revealed preference and contingent valuation methods; discounting; calculation of cost-effectiveness ratios, net present value, benefit-cost ratios, and internal rates of return. Sensitivity testing and uncertainty will also be addressed. Individuals will work in groups to assess various costs, effects and benefits applicable to selected case studies across various policy fields. Case studies will be selected from across policy fields (e.g. health, education, environmental sciences).
Instructor: Ryan Watkins, PhD
Description: The initial phase of a project or program is among the most critical in determining its long-term success. Needs assessments support this initial phase of project development with proven approaches to gathering information and making justifiable decisions. In a two-day course, you will learn how needs assessment tools and techniques help you identify, analyze, prioritize, and accomplish the results you really want to achieve. Filled with practical strategies, tools, and guides, the workshop covers both large-scale, formal needs assessments and less formal assessments that guide daily decisions. The workshop blends rigorous methods and realistic tools that can help you make informed and reasoned decisions. Together, these methods and tools offer a comprehensive, yet realistic, approach to identifying needs and selecting among alternative paths forward.
In this course, we will focus on the pragmatic application of many needs assessment tools, giving participants the opportunity to practice their skills while learning how needs assessment techniques can improve the achievement of desired results. With participants from a variety of sectors and organizational roles, the workshop will illustrate how needs assessments can be of value in a variety of operational, capacity development, and staff learning functions.
Instructor: Emily Tanner-Smith, PhD
Description: Meta-analysis refers to a range of statistical techniques used for summarizing and synthesizing quantitative findings from two or more primary research studies. Applied researchers and evaluators often use meta-analysis techniques to review research evidence to address questions about the effects of intervention programs, differences between demographic groups, prevalence of conditions, strengths of relationships between risk/protective factors and outcomes, reliability and validity of measurement instruments, and diagnostic test accuracy. This course will provide practical instruction on how to conduct meta-analysis using the R statistical environment. The course will cover content on (a) formulating research questions and identifying relevant studies for inclusion, (b) coding procedures used to collect data, (c) structuring and managing meta-analytic databases, (d) analyzing meta-analytic databases, and (e) interpreting findings from a meta-analysis. Participants will be given a detailed guide for conducting meta-analysis along with corresponding R code and example datasets. The first day of the course will focus on the procedures and methods for meta-analysis. The second day will focus on hands-on application of meta-analytic procedures in R and conclude with a discussion around reporting and interpretation of meta-analysis findings.
Qualitative Data Analysis
Instructor: Patricia Rogers, PhD
Description: Many evaluators find it challenging to analyze textual, visual, and aural data from interviews, diaries, observations, and open-ended questionnaire items in ways that are rigorous but practical within the time and staffing constraints of real evaluation. Analysis of qualitative data can range from simple enumeration and illustrative use to more detailed analysis requiring more expertise and time. In this class, participants will work through a structured approach to analyzing qualitative data based on an iterative process of considering the purpose of the analysis, reviewing suitable options, and working through interpretations. Techniques include grouping, summarizing, finding patterns, discovering relationships, and developing and testing relationships. The session will address practical and ethical issues in analyzing and reporting qualitative data-particularly who participates in interpretation, how confidentiality can be maintained, how analysis can be tracked and checked, and standards for good practice in qualitative data analysis. Hands-on exercises for individuals and small groups will be used throughout the class. Manual analysis of data will be used in exercises and participants will also be introduced to NVivo and other computer packages to assist analysis. As part of the course, participants will receive the textbook, Qualitative Data Analysis by Miles, Huberman & Saldaña (Sage, 2014).
Social and Organizational Network Analysis – Evaluating the Way Individuals and Organizations Interact
Instructor: Kimberly A. Fredericks, PhD
Description: This course is an introductory course for evaluators who want to explore how social or organizational network analysis can be added to their repertoire of tools and methods. Network analysis is a technique that allows us to better understand social structures by enabling visualization (through its network plots) and analyzing interactions among actors (through its associated network statistics).
Social or organizational network analysis can help us build understanding of why a particular network may or may not be successful in achieving its goals, or be sustained over time. The linkages between actors (individuals or organizations) can include various types of connections, such as exchange of information, human and financial resources, power and influence, and social support.
This course will focus briefly on the theory behind social and organizational network analysis, but focus mostly on how social or organizational network analysis can add value to evaluation, covering when evaluations can make best use of it (what kinds of evaluation questions can it help answer?), key decisions in designing an SNA/ONA, strategies for and pitfalls in data collection, approaches to analysis, and how to help clients draw meaning from the results.
The course will also cover the steps of implementing SNA/ONA, and highlight issues in collecting and analyzing data, and interpreting findings or “reading” the SNA/ONA. Through discussions, group work, hands-on analysis of case study data, participants will experience the whole process of using social and organizational network analysis.
By the end of this course, participants will be able to:
- Understand when social/organizational network analysis can be useful in evaluation
- Outline the key components of a social/organizational network analysis design
- Understand trade-offs in scope and sampling decisions
- Develop network plots and statistics using software
- Explain to a client how to make sense of network analysis data