Using Evaluation – Strategies and Capacity Courses
- Culture and Evaluation
- Effective Reporting Strategies for Evaluators
- How to Build a Successful Evaluation Consulting Practice
- Implementation Analysis for Feedback on Program Progress and Results
- Making Evaluation Data Actionable
- Presenting Data Effectively: Practical Methods for Improving Evaluation Communication
- Project Management & Oversight for Evaluators
- Strategy Mapping
- Using Program Evaluation in Nonprofit Environments
Culture and Evaluation
Instructor: Leona Ba, EdD
Description: This course will provide participants with the opportunity to learn and apply a step-by-step approach on how to conduct culturally responsive evaluations. It will use theory-driven evaluation as a framework, because it ensures that evaluation is integrated into the design of programs. More specifically, it will follow the three-step Culturally Responsive Theory-Driven Evaluation model proposed by Bledsoe and Donaldson (2015):
- Develop program impact theory
- Formulate and prioritize evaluation questions
- Answer evaluation questions
Upon registration, participants will receive a copy of the book chapter discussing this model.
During the workshop, participants will reflect on their own cultural self-awareness, a prerequisite for conducting culturally responsive evaluations. In addition, they will explore strategies for applying cultural responsiveness to evaluation practice using examples from the instructor’s first-hand experience and other program evaluations. They will receive a package of useful handouts, as well as a list of selected resources.
Prerequisites: Understanding of evaluation and research design.
Effective Reporting Strategies for Evaluators
Instructor: Kathryn Newcomer, PhD
Description: The use and usefulness of evaluation work is highly affected by the effectiveness of reporting strategies and tools. Care in crafting both the style and substance of findings and recommendations is critical to ensure that stakeholders pay attention to the message. Skill in presenting sufficient information — yet not overwhelming the audience — is essential to raise the likelihood that potential users of the information will be convinced with both the relevance and the validity of the data. This course will provide guidance and practical tips on reporting evaluation findings. Attention will be given to the selection of appropriate reporting strategies/formats for different audiences and to the preparation of: effective executive summaries; clear analytical summaries of quantitative and qualitative data; user-friendly tables and figures; discussion of limitations to measurement validity, generalizability; causal inferences, statistical conclusion validity, and data reliability; and useful recommendations. The text provided as part of course fee is Torres et al., Evaluation Strategies for Communicating and Reporting (2nd Ed., Sage, 2005).
How to Build a Successful Evaluation Consulting Practice
Instructor: Michael Quinn Patton, PhD
Description: This class offers the opportunity for participants to learn from someone who has been a successful evaluation consultant for 30 years. Issues addressed include: What does it take to establish an independent consulting practice? How do you find your consulting niche? How do you attract clients, determine how much to charge, create collaborations, and generate return business? Included will be discussion on such topics as marketing, pricing, bidding on contracts, managing projects, resolving conflicts, professional ethics, and client satisfaction. Participants will be invited to share their own experiences and seek advice on situations they’ve encountered. The course is highly interactive and participant-focused.
Participants will receive a copy of the instructor’s text: Essentials of Utilization-Focused Evaluation (Sage, 2012).
Implementation Analysis for Feedback on Program Progress and Results
Instructor: Arnold Love, PhD
Description: Many programs do not achieve intended outcomes because of how they are implemented. Thus, implementation analysis (IA) is very important for policy and funding decisions. IA fills the methodological gap between outcome evaluations that treat a program as a “black box” and process evaluations that present a flood of descriptive data. IA provides essential feedback on the “critical ingredients” of a program, and helps drive change through an understanding of factors affecting implementation and short-term results. Topics include: importance of IA; conceptual and theoretical foundations of IA; how IA drives change and complements other program evaluation approaches; major models of IA and their strengths/weaknesses; how to build an IA framework and select appropriate IA methods; concrete examples of how IA can keep programs on-track, spot problems early, enhance outcomes, and strengthen collaborative ventures; and suggestions for employing IA in your organization. Detailed course materials and in-class exercises are provided.
Making Evaluation Data Actionable
Instructor: Ann Doucette, PhD
Description: Interventions and programs are implemented within complex environments that present challenges for collecting program performance information. A general problem for performance measurement initiatives — and what often causes them to fall short of their intended objectives — is the failure to choose performance measures that are actionable, meaning that they are linked to practices that an organization or agency can actually do something about, and the changes in those practices can be linked directly to improved outcomes and sustained impact.
This class introduces complex adaptive systems (CAS) thinking and addresses the implication of CAS in evaluating the outcomes and impact of interventions and programs. Examples used in this case range from healthcare, education, transportation and safety, developing countries, and research and development environments. The class examines performance measurement strategies that support actionable data. The focus will be on data-based decision making, value-based issues, and practice-based evidence that can assist in moving performance measurement and quality monitoring activities from a process, outcome, and impact evaluation approach to continuous quality improvement. Business models such as Toyota Production System, Six-sigma, Balanced Scorecards, as well as knowledge management and benchmarking strategies will be discussed in terms of how they can inform improvement strategies.
Note: Persons with some experience in program evaluation, and those with interest in a systems perspective will likely derive the most benefit from this course.
Presenting Data Effectively: Practical Methods for Improving Evaluation Communication
Instructor: Stephanie Evergreen, PhD
Description: Crystal clear charts and graphs are valuable–they save an audience’s mental energies, keep a reader engaged, and make you look smart. In this workshop, attendees will learn the science behind presenting data effectively. We will go behind-the-scenes in Excel and discuss how each part of a visualization can be modified to best tell the story in a particular dataset. We will discuss how to choose the best chart type, given audience needs, cognitive capacity, and the story that needs to be told about the data–and this will include both quantitative and qualitative visualizations. We will walk step-by-step through how to create newer types of data visualizations and how to manipulate the default settings to customize graphs so that they have a more powerful impact. Working in a computer lab, attendees will build with a prepared spreadsheet to learn the secrets to becoming an Excel dataviz ninja. Attendees will get hands-on practice implementing direct, practical steps that can be immediately implemented after the workshop to clarify data presentation and support clearer decision-making. Full of guidelines and examples, after this workshop you’ll be better able to package your data so it represents your smart, professional quality.
Note: Attendees are strongly encouraged to maximize the workshop experience by bringing a slideshow that contains graphs under current construction.
In the second day of workshop, Dr. Stephanie Evergreen will lead attendees through how to manipulate Excel into making impactful charts and graphs, step-by-step, using provided data sets distributed to the audience. Audience members will leave the session with more in depth knowledge about to craft effective data displays. The demonstration will occur in the computer lab on PCs running Office 2010. Completing the session moves one to Excel Ninja Level 10.
Attendees will learn:
- Visual processing theory and why it is relevant for evaluators
- How to apply graphic design best practices and visual processing theory to enhance data visualizations with simple, immediately implementable steps
- Which chart type to use, when
- How to construct data visualizations and other evaluation communication to best tell the story in the data
- Alternative methods for reporting
Workshop attendees will leave with helpful handouts and a copy of Effective Data Visualization (Sage, 2016).
Registrants should regularly develop graphs, slideshows, technical reports and other written communication for evaluation work and be familiar with the navigational and layout tools available in simple software programs, like Microsoft Office.
Project Management and Oversight for Evaluators
Instructor: Tessie Catsambas, MPP
Description: The purpose of this course is to provide new and experienced evaluation professionals and funders with strategies, tools and skills to: (1) develop realistic evaluation plans; (2) negotiate needed adjustments when issues arise; (3) organize and manage evaluation teams; (4) monitor evaluation activities and budgets; (5) protect evaluation independence and rigor while responding to client needs; and (6) ensure the quality of evaluation products and briefings.
Evaluation managers have a complex job: they oversee the evaluation process and are responsible for safeguarding the methodological integrity, evaluation activities, and budgets. In many cases they must also manage people, including clients, various stakeholders, and other evaluation team members. Evaluation managers shoulder the responsibility for the success of the evaluation, frequently dealing with unexpected challenges, and making decisions that influence the quality and usefulness of the evaluation.
Against a backdrop of demanding technical requirements and a dynamic political environment, the goal of evaluation management is to develop, with available resources and time, valid and useful measurement information and findings, and ensure the quality of the process, products and services included in the contract. Management decisions influence methodological decisions and vice versa, as method choice has cost implications.
The course methodology will be experiential and didactic, drawing on participants’ experience and engaging them with diverse material. It will include paper and online tools for managing teams, work products and clients; an in-class simulation game with expert judges; case examples; reading; and a master checklist of processes and sample forms to organize and manage an evaluation effectively. At the end of this training, participants will be prepared to follow a systematic process with support tools for commissioning and managing evaluations, and will feel more confident to lead evaluation teams and negotiate with clients and evaluators for better evaluations.
Instructor: John Bryson, PhD
Description: The world is often a muddled, complicated, dynamic place in which it seems as if everything connects to everything else – and that is the problem! The connections can be problematic because – while we know things are connected – sometimes we do not know how, or else there are so many connections we cannot comprehend them all. Alternatively, we may not realize how connected things are and our actions lead to unforeseen and unhappy consequences. Either way, we would benefit from an approach to strategizing, problem solving, conflict management, and evaluation that helps us understand just how connected the world is, what the effects of those connections are, and what might be done to change some of the connections and their effects.
Action-oriented strategy mapping (AOSM) is a simple and useful technique for addressing situations where thinking – as an individual or as a group – matters. An action-oriented strategy map is a word-and-arrow diagram in which ideas and actions are causally linked with one another through the use of arrows. The arrows indicate how one idea or action leads to another – and specifically how coherent sets of actions, strategies, and goals can be created, communicated, managed, and implemented. AOSM makes it possible to articulate a large number of ideas and their interconnections in such a way that people can know what to do in an area of concern, how to do it, and why, because the arrows indicate the causes and consequences of an idea or action. AOSM therefore is a technique for linking strategic thinking, acting, and learning; helping make sense of complex problems; communicating to oneself and others what might be done about them; and also managing the inevitable conflicts that arise. The technique is useful for formulating and implementing mission, goals, and strategies and for being clear about how to evaluate strategies. The bottom line is: AOSM is one of the most powerful strategic management tools in existence. AOSM is what to do when thinking matters!
When can mapping help? There are a number of situations that are tailor-made for mapping. Mapping is particularly useful when:
- effective strategies need to be developed
- persuasive arguments are needed
- effective and logical communication is essential
- effective understanding and management of conflict are needed
- it is vital that a situation be understood better as a prelude to any action
- organizational or strategic logic needs to be clarified in order to design useful evaluations
These situations are not meant to be mutually exclusive. Often they will overlap in practice. In addition, mapping is very helpful for creating business models (that link competencies and distinctive competencies to mission, goals, and critical success factors) and Balanced Scorecards (which are more useful for strategy implementation than for strategy formulation). AOSMs are related to logic models, as both are word-and-arrow diagrams, but are more tied to goals, strategies, and actions and are more careful about articulating causal connections.
Objectives: (Strategy Mapping)
At the end of the course, participants will:
- Understand the theory of mapping
- Know the difference between action-oriented strategy maps, business model maps, and balanced scorecard maps
- Be able to create action-oriented strategy maps for individuals – that is, either for oneself or by interviewing another person
- Be able to create action-oriented maps for groups
- Be able to create a business model map linking competencies and distinctive competencies to goals and critical success factors
- Know how to design and manage change processes in which mapping is prominent
- Have an action plan for an individual project
Using Program Evaluation in Nonprofit Environments
Instructor: Kathryn Newcomer, PhD
Description: Funders and oversight boards typically need data on the results obtained by the programs they fund. Within foundations program officers want information about grantees, and about lines of effort they fund to guide planning and future allocation of resources. Executive officers and members of the boards that oversee nonprofit service providers also want to know what works and what does not. This class provides background that program officers and overseers need to understand how evaluation can serve their information needs, and how to assess the quality of the evidence they receive.
Drawing upon cases from foundations and nonprofits, the session will help attendees:
- Clarify where to start in using evaluation to improve nonprofit social service programss
- Learn what/who drives program evaluation and performance measurement in public and nonprofit service providers
- Explore uses of evaluation and outcomes assessment in the non-profit sector
- Understand how to frame useful scopes of work (SOWs) and requests for proposals (RFPs) for evaluations and performance measurement systems
- Identify and apply relevant criteria in choosing contractors and consultants to provide evaluation assistance
- Discuss challenges to measurement of social service outcomes
- Understand what questions to ask of internal evaluation staff and outside consultants about the quality of their work