JRSA Concludes Evaluation of Utah Mentoring Program

Stan Orchowsky, Ph.D., JRSA Research Director

JRSA Forum. December 2012. Volume 30, Number 4.

JRSA recently completed the final report on the evaluation of Utah's Youth and Families with Promise (YFP) program. With funding from the National Institute of Justice, JRSA conducted a process and outcome evaluation of the YFP program, a statewide 4-H mentoring program in which at-risk youths who are approximately 10-14 years old are paired with mentors, participate in 4-H activities, and also attend Family Night Out activities to strengthen family bonds. The results of the process evaluation were reported in the June 2008 issue of the JRSA Forum; this article focuses on the results of the outcome evaluation.

JRSA used a nonequivalent groups design for the outcome evaluation (pre- and post-testing for an experimental group as well as a comparison group) that included up to a three-year follow-up for youths attending the program. We collected pre- and post-test data from youths entering YFP from 2005 to 2008 as well as their parents (YFP group), and from two cohorts of students in grades 4-8 from various schools throughout the state (comparison group). The major component of the pre- and post-tests was the Behavioral and Emotional Rating Scale Version 2 (BERS-2), a 52-item scale designed to assess the behavioral and emotional strengths of children in five domains (subscales): Interpersonal Strength, Intrapersonal Strength, Affective Strength, Family Involvement, and School Functioning. For youths in the YFP group, the post-test also included questions on mentor relationship quality and perceptions of the YFP program.

To assess the long-term outcomes of program participation and sustainability of program effects, we scheduled follow-up telephone interviews with YFP and comparison group youths to occur annually for up to three years after program discharge/date of the last post-test. Given the program objective associated with improving school performance, we obtained official school records of attendance and grades prior to, during, and up to three years following involvement in the program (or last post-test for the comparison group). Since the program targets at-risk youths, we collected data on YFP and comparison youths' involvement with the juvenile court system prior to, during, and up to a maximum of three years following their involvement in the program/study.

A total of 392 YFP youth and 327 comparison youth completed pre-tests, but we ultimately received valid pre- and post-tests for 257 YFP youth and 280 comparison youth.

Analyses using simple gain scores (from pre-test to post-test, roughly eight months later) showed that mean scores improved significantly for YFP youths for all BERS-2 scales. However, when change relative to the comparison group was considered, YFP youth improved significantly only on one subscale, affective strength (ability to accept affection and express feelings). The positive changes seen for the YFP group were affected by age: Younger YFP youth were more likely to show changes than older youth. School and delinquency data showed either no differences between the groups or the YFP youth performing worse than the comparison group. Data collected as part of the long-term follow-up showed that although YFP youth reported positive feelings about the helpfulness of the program and their mentors, they also reported more school, family, and social issues in the first year after they left the program than comparison group youths.

The report concludes that despite a number of positive findings, the YFP program in general did not produce the desired outcomes. One possible explanation for this was methodological. Examination of the YFP and comparison groups showed significant differences between the two groups on a number of sociodemographic and risk factors at the pre-test, with YFP youth more likely to be non-white, younger, living with only one parent, and having one or more self-reported school suspensions in the six months prior. For the BERS-2 subscales, YFP youths had lower average scores and showed greater variability in their scores than youths in the comparison group. The report urges caution in interpreting the findings given this lack of initial comparability between the two groups.

Another possible explanation for the lack of observed program effects relates to the findings of the process evaluation: specifically, that not enough youth received enough program services. Although collecting accurate data on mentoring and other services was a challenge, the data collected suggested that youth did not receive as many mentoring sessions as they were supposed to according to program documentation. This may have been a factor in the failure to find significant program effects.

A copy of the entire report, A Process and Outcome Evaluation of the 4-H Mentoring/Youth and Families with Promise (YFP) Program, along with a summary of the findings, is available on JRSA's Web site.