TEI 340: AI-Powered Program Evaluation: From Administrative Data to Automated Insights

Instructor: Peter York

Description: This three-day hands-on course teaches evaluators to leverage artificial intelligence, large language models (LLMs), and machine learning to transform program administrative data into automated evaluation systems. Participants will work with two complementary classes of tools: KNIME, a free visual-based analytics platform that requires no coding experience, for building causal models and automated workflows; and today’s leading LLMs—including Anthropic’s Claude Code, and OpenAI’s Codex for accelerating qualitative analysis, data preparation, and the design of automated evaluation pipelines. Together, these tools significantly reduce evaluation timeframes while increasing quasi-experimental rigor, depth of insight, and actionability.

Through guided exercises with sample datasets, participants will be provided the foundational tools and techniques for the complete workflow of modern AI-powered evaluation, including:

• Cleaning and transforming both structured (numeric, ordinal, and categorical) and unstructured (text) program data, using LLMs to speed and simplify the work
• Using large language models for qualitative and text analysis
• Applying LLM-based assistants (such as Claude Code, Codex, and Gemini) to draft, troubleshoot, and automate analytic workflows—no prior coding required
• Training machine learning algorithms to conduct causal modeling that identifies and evaluates natural experiments within historical program data

Key learning outcomes include: converting program administrative data into evaluation-ready formats; transforming qualitative/text data into structured insights using LLMs; using AI assistants to build and automate analytic pipelines more quickly and easily; implementing causal modeling to evaluate program outcomes across different types of beneficiaries; and automating these processes end-to-end.

The workshop includes in-depth, hands-on practice with both KNIME and leading LLMs, using sample datasets to build real evaluation models with automation. Participants will leave with foundational skills to begin applying these techniques in their own evaluation work.

Recommended Audience: This advanced course is designed for experienced evaluators who want to modernize their practice with AI, large language models, and machine learning. Participants should have experience conducting mixed-methods evaluations and working with program administrative data. While no coding experience is required, comfort with data analysis software (e.g., SPSS, SAS, Stata, Excel) is ideal. We recommend participants bring a laptop capable of running KNIME (downloadable and installable free of charge) and accessing web-based AI tools, though KNIME is not mandatory.

Prerequisites: We recommend that participants download and install KNIME before the first session, but it is not required. Participants who are unable to install KNIME—or who wish to attend as observers—can still fully follow and benefit from the AI/LLM portions of the course. Sample datasets and workflows will be provided, and guidance on accessing the AI tools used in the course will be shared in advance.

TEI Certificate: This course fulfills the following requirements:

• Analysis Elective for the Certificate in Evaluation Practice (CEP)
• Evaluation Practice course for the Certificate in Advanced Evaluation Practices (CAEP)
• Design and Method course for the Certificate in Analytic Evaluation Methods (CAEM)