program evaluation resource guide

PROGRAM EVALUATION RESOURCE GUIDE

Out-of-School Time Peer Learning Group

1 | P a g e

Tool/Framework/Resource

Link to Relevant Information

Brief Summary/Description

Partnership for After

School Education: Youth

Outcomes Initiative

http://pasesetter.org/initiatives/youth-outcomes

As afterschool providers strive to support young peoples’ development into responsible, healthy

citizens, they require a comprehensive and nuanced set of criteria and measurements to guide

both their practices and their evaluation of those practices. Furthermore, providers need an

accurate vocabulary to articulate to a variety of stakeholders – including parents, schools, funders,

and youth themselves – the impact that high-quality programs can have on young people.

National Summer

Learning Association:

Comprehensive

Assessment of Summer

Programs (CASP)

Quality Assessment:

http://summermatters2you.net/program-

providers/

Quick Reference Guide:

http://www.summerlearning.org/?page=caspgui

Comprehensive Planning Guide:

https://c.ymcdn.com/sites/summerlearning.site-

ym.com/resource/resmgr/ssins_/ss_in_s_samp

le_pages.pdf

Companion Guide:

http://c.ymcdn.com/sites/summerlearning.site-

ym.com/resource/resmgr/ssins_/TRAIN_ASSE

SS_REFLECT_sampler.pdf

These tools are organized to reflect the domains of the National Summer Learning Association’s

quality standards, and are intended to support your efforts to build a high quality summer learning

program. The Quick CASP is a quality self-assessment tool developed by the National Summer

Learning Association in partnership with the Summer Matters Campaign to help summer learning

programs continuously improve their program quality.

Higher Achievement: 2-

and 4-Year Follow-Up

Randomized Control

Trials Studies

2-Year Study:

http://www.issuelab.org/resource/testing_the_i

mpact_of_higher_achievements_year_round_o

ut_of_school_time_program_on_academic_out

comes

4-Year Study:

http://www.mdrc.org/sites/default/files/staying_

on_track_testing_higher_achievement.pdf

In the spring of 2006, 2007 and 2008, youth who were entering fifth or sixth grade and met Higher

Achievement's admission criteria completed a standardized test and were surveyed (as were their

parents). Then a lottery was used to determine which students would be offered the opportunity to

participate in Higher Achievement and which would not.

2 | P a g e

Classroom, Inc.

Resources

Scientific Research Base:

http://www.classroominc.org/?attachment_id=2

007

Evaluation Work:

http://www.classroominc.org/?attachment_id=1

934

http://www.classroominc.org/?attachment_id=1

937

Survey Tool:

http://www.leadtoreadkc.org/wp-

content/uploads/2012/12/Professor-Garfield-

reading-survey-used-by-Lead-to-Read-KC.pdf

Results:

http://www.classroominc.org/impact/results/

Scientific Research Base: Serves as the basis of the evaluation work that Classroom, Inc. does in

their programs.

Evaluation Work: CI's research team in partnership with Metis Associates determines program

evaluation and impact for students and teachers.

Classroom, Inc. (CI) takes a multi-method approach to program evaluation, depending on the

needs and interests of partner districts or schools and the product used by that set of schools.

Across all of our work, CI is interested in documenting how well our learning games, curriculum

and professional development tools:

1. Improve students’ reading behaviors, attitudes, and achievement.

2. Bolster students’ non-cognitive skills like problem solving abilities, decision-making skills, and

persistence.

3. Help teachers to use more effective and innovative instructional strategies in their classrooms.

Our multi-method evaluation strategy includes collecting qualitative data from schools about the

successes and challenges of implementation through focus groups with participating teachers, and

capturing quantitative data through a set of research-based assessment tools. These tools include

using normed pre/post tests to measure overall literacy achievement, using embedded

assessments to measure reading behaviors and achievement in-game, using on-line student

surveys to measure reading attitudes, 21st Century skills and overall program experience and

educator surveys to focus on instructional goals and strategies.

Harvard Family Research

Project - Measurement

Tools for Evaluating Out-

of-School Time Programs:

An Evaluation Resource

http://www.hfrp.org/out-of-school-

time/publications-resources/measurement-

tools-for-evaluating-out-of-school-time-

programs-an-evaluation-resource2#table1-1

This is an incredible resource, listing dozens of evaluation tools and describing the different things

that those tools can be used to measure. Definitely worth a look.

"The information in this guide can help practitioners and evaluators find evaluation instruments

that match their program and evaluation goals and characteristics. The instruments can be used

alone, or in conjunction with publications in our Snapshots series focused on performance

measures and data collection methods, to help you develop an overview of previous evaluations

and design future evaluations intended for a range of purposes and stakeholders."

City Year- Devereux

Student Strengths

Assessment (DESSA)

http://www.centerforresilientchildren.org/school

-age/assessments-resources/the-devereux-

student-strengths-assessment-dessa-kit/

The Devereux Student Strengths Assessment (DESSA) is a 72-item, standardized, norm-

referenced behavior rating scale that assesses the social-emotional competencies that serve as

protective factors for children in kindergarten through the eighth grade. This year, City Year corps

members have assessed approximately 6,900 students in 107 schools in 13 cities using this tool at

the start of the year. These students will also be assessed at the mid-year and end of year.

3 | P a g e

Horizons National

http://indexgroups.org/msa/

Several of our affiliates are using the Mission Skills Assessment (MSA) to measure readiness and

resiliancy during the 8th to 9th grade transition. We aslo use the Quick CASP and other proprietary

tools aligned with our program guidelines to measure quality and student growth. Our most useful

and simple indicators are attendance, year-to-year return rate, academic/school performance, and

high school graduation rate.

NYSAN's Quality Self-

Assessment Tool

http://www.nysan.org/quality-self-assessment-

tool/

The QSA Tool is meant to be used in concert with other formal and informal evaluation methods,

such as youth, parent and staff meetings, youth and parent focus groups, and external motioning

and evaluation. Each of these methods can help identify program strengths and areas in need of

improvement.

Measuring Social and

Emotional Learning with

the Survey of Academic

and Youth Outcomes

(SAYO) - NIOST

http://www.niost.org/pdf/MeasuringSELwithSA

YO.pdf

This paper analyzes the way that the SAYO can be used to measure the different types of

noncognitive/SEL skills, as those skills are defined by four different frameworks; CASEL's SEL

Core Competencies, Every Hour Counts Measurement Framework, Forum for Youth Investment

Skill Areas, and CCSR Noncognitive Factors

4-H Common Measures

http://www.afterschoolalliance.org/documents/4

-H_Common%20Measures.pdf

Using logic modeling, this team of experts identified desired outcomes and impacts of positive

youth development programming overall as well as for each of 4-H’s program areas: Science,

Healthy Living, Citizenship, College/Career Readiness. The ability to measure these outcomes with

the 4-H Common Measures provides a base to strengthen programs and adult practitioners.

4-H - The Positive

Development of Youth:

Comprehensive Findings

from the 4-H Study

http://www.afterschoolalliance.org/documents/4

H_Positive%20Youth%20Development%20Re

port.pdf

The 4-H Study of Positive Youth Development is a longitudinal study that began in 2002 and was

repeated annually for eight years, surveying more than 7,000 adolescents from diverse

backgrounds across 42 U.S. states. The study discovered that the structured out-of-school time

learning, leadership experiences, and adult mentoring that young people receive through their

participation in 4-H plays a vital role in helping them achieve success.

4 | P a g e

Looking at the Data:

Afterschool Programs

Using Data to Better

Serve Students

http://www.afterschoolalliance.org/issue_66_us

ing_data.cfm

The Afterschool Alliance, in partnership with MetLife Foundation, is proud to present the final issue

brief in our latest series of four issue briefs examining critical issues facing middle school youth

and the vital role afterschool programs play in addressing these issues. This series explores

afterschool and: the Common Core State Standards, students with disabilities and other special

needs, data utilization to improve programming, and keeping children safe and supported. The

briefs examine just a few of the ways afterschool programs support middle school youth, families

and communities.

Y Scholars: Pre-Survey

Form

http://www.afterschoolalliance.org/documents

/documents/Y%20Scholars%20Pre-

%20Survey%20Form%202014-15.pdf

This is the full pre-program survey used by the Y Scholars program run by the YMCA of Greater

New York.

Weikart Center: Youth

Program Quality

Assessment Tool (PQA)

http://www.cypq.org/downloadpqa

The Youth Program Quality Assessment (PQA), School-Age Program Quality Assessment and

several extensions (STEM, Arts, Academic Climate, Health and Wellness) were designed to

assess the quality of learning environments and to identify staff training needs. These are the field-

leading quality assessments for out-of-school time organizations, in part because you and many

others have found them useful. In the past, these instruments have only been available – at cost –

from the Weikart Center at the Forum for Youth Investment. We are pleased to share these tools

as pdf downloads, subject to the following recommendations and terms.

New York State

Afterschool Network:

Quality Self-Assessment

Tool (QSA)

http://www.nysan.org/quality-self-assessment-

tool/

Improving afterschool program quality must begin with a commitment on the part of practitioners to

examine their programs with a critical eye. A quality program is one that is reflective, willing to

improve, change and grow, and believes in successful outcomes for its participants. Moreover,

practitioners must understand that improving program quality is an ongoing process. Self-

assessment provides an important opportunity for programs to identify strengths and weaknesses

free from the pressures of external monitoring and evaluation. It is upon these concepts that the

NYSAN Quality Self-Assessment (QSA) Tool is based. NYSAN is working in a number of ways to

encourage the adoption of common quality standards statewide.

Social and Emotional

Learning: A Resource

Guide and New Approach

to Measurement in

ExpandED Schools

http://www.tascorp.org/tools/social-and-

emotional-learning-resource-guide-and-new-

approach-measurement-expanded-

schools#sthash.peh82uW4.PGFxFQF8.dpbs

This guide is a list of tools that can be used in continued implementation of strong programming

across the landscape of policy, research and practice, with a description of each tool gathered

directly from its website.