1994 Consortium Connections: Winter 1994 [s]
The Children Youth and Family Consortium University of
Minnesota University Gateway, 270A
200 Oak Street SE
Minneapolis MN 55455 Phone 612-626-1212
Email:cyfstaff@maroon.tc.umn.edu
MN Children Youth and Families Consortium Electronic Clearinghouse. Permission is granted to create and distribute copies of this document for non-commercial purposes provided that the author and MN CYFCEC receive acknowledgement and this notice is included. Phone 612-626-1212 EMAIL: cyfcec@staff.tc.umn.edu
Consortium Connections is published three times a year (Winter, Spring, and Fall) by the Children, Youth, and Family Consortium: A University and Community Collaboration.
In This Issue--Spotlight on Program Evaluation
For those of us working to improve the lives of children and families, one of the most pressing issues today is how to evaluate our efforts--how to demonstrate that we are doing what we say we are doing and that it really makes a difference. Because promoting good evaluation is a primary objective for the Consortium, we are devoting this issue of Consortium Connections to the topic. In our lead article, Laura Bloomberg presents a broad and comprehensive overview of what evaluation is and should be. John Romano brings
the concepts to life with a summary of findings from his evaluation of an innovative school personnel training program. And Aimee Reedy's article provides important lessons in linking evaluation to policy development.
This newsletter is only one of the Consortium's tools for promoting new ways of thinking about evaluation. In this issue you will find a registration form for a conference on evaluation which the Consortium is sponsoring with the MN Chapter of the American Evaluation Association this spring. The conference will be an exciting opportunity to hear about model evaluation designs and to explore with others how to conduct evaluations that will help us be more effective in addressing the needs of young people and families. Also, the Consortium Electronic Clearinghouse is a dynamic resource for sharing information about evaluation. Let us know when you discover new approaches or good resources on evaluation and we will see that the ideas reach others who can use them.
Program Evaluation: An Overview
Laura Bloomberg, Assistant Director, Institute on Community
Integration
"Well-planned and implemented evaluation is coordinated with the vision of the program, the context in which services are provided, the goals and activities of the program and the program outcomes. Program evaluation is the glue that binds the program together, the helm that steers that program on the course defined by its vision, and the anchor that keeps the program in touch with consumers, funding agencies, and the community. When approached from a broad perspective, program evaluation can provide a strategy to unify and focus all of the activities within a program." (Szymanski, E. [1992] Making Evaluation Work for You, p i.)
A comprehensive evaluation system is a vital component of any human service-related program that seeks to meet community needs. While many human service professionals understand the importance of evaluation, both for improving service delivery and for explaining what programs do, there is still a lot of confusion about the "how" and the "what" of program evaluation. The following overview sheds some light on the primary functions of program evaluation and some of the major components included in most evaluation models.
Evaluation Purposes and Functions
Good evaluation systems can fulfill many functions, including:
Key Components of Evaluation Models
Good evaluation systems typically use multiple information collection strategies over a period of time. The strategies must be flexible and multi-dimensional, just as the evaluation system itself must remain flexible and evolve as the program it seeks to measure evolves. Thus, there is no single prescription for evaluation that applies equally well to all human-service-related programs. There are, however, some components that remain common to most evaluation models:
1) Evaluation models should include both formative and summative components.
Evaluative information should be available on an ongoing (sometimes informal) basis to inform program decision-making and to address questions about how the program can be improved (formative evaluation). It is often important, however, to formally summarize program activities, outcomes, and impact as well (summative evaluation). Formative evaluation is often most helpful to program implementers and planners, while summative evaluation is often most relevant to program funders, external supporters, and interested constituents.
2) Evaluation models should include both quantitative and qualitative information.
The identification of quantitative outcomes and outcome indicators that are both valued by the community and related to the work and goals of the program is a key component. While many desired outcomes could take years to measure, interim indicators of projected outcomes can be identified and measured. Although some information about human service-related outcomes and indicators already exists, determination of specific program outcomes is a process that should include a wide-range of constituents. This assures an array of measures that are culturally relevant and valued by the community, while still being scientifically sound measures of success-related outcomes.
Some of the most important evaluative information is gleaned from qualitative processes, including interviews, focus groups, informal discussion, and surveys. The perspectives of key stakeholders, including service providers, primary service recipients, and families, provide important insights into the successes and barriers of a program.
3) The evaluation process must be participatory.
Although program evaluations are typically coordinated by a lead evaluator (or a small team), the process must actively include a wide range of key participants. Since the process of evaluation is really one of close examination, it stands to reason that several sets of eyes and ears are better than one. Thus, the evaluation must be action-oriented and participatory.
Enhancing Student Well-Being:
School Personnel Training for Alcohol and Drug Prevention John L.
Romano, Assoc. Professor, Educational Psychology, College of
Education, U of MN
During the last three summers, 118 Minnesota elementary and secondary school educators have received in-service-training through a three week Institute entitled "Enhancing Student Well-Being." While a major focus of the Institute is on the prevention of alcohol and other drug use, the Institute takes a comprehensive approach to the issues of youth, recognizing that problem behaviors of youth are often inter-related. Institute curriculum topics include model prevention programs, relationships between alcohol/drug use, sexual acting out and violent behavior, adolescent development, parent and family involvement, educator stress and burnout, leadership, and program evaluation. The Institute offers an opportunity for educators to not only receive information and discuss issues of importance, but also to work together in project development teams. Educators, who come to the Institute in school teams, are required to develop a prevention project during the Institute which they can implement in their building during the next school year. Therefore, the Institute takes an applied and action oriented approach, and the educators leave the Institute with a project which is important to their school community. The Institute, under the direction of John Romano, faculty member in the Department of Educational Psychology, has been made possible through training grants received from the U.S. Department of Education's Drug Free Schools and Communities Program.
The Institute evaluations have focused on four aspects: new knowledge gained by participants; changes in educator self-efficacy related to implementing student well-being projects; educator satisfaction with goals, content, and activities of the Institute; and impact of projects developed during the Institute on the school community the following year. Measures of knowledge-gained and self-efficacy are administered pre and post Institute and at four month (knowledge) and eight month (self-efficacy) follow-up. Measures of satisfaction are given at the end of the Institute and at 10 month follow-up. At the end of the school year, impact of the prevention project on the school community is assessed. All of the evaluation questionnaires, which allow for the collection of both qualitative and quantitative data, were developed specifically for the Institute. Results for the previous two years ('91 and '92) showed that compared to a control group of similar educators, knowledge increased as a result of the Institute and was maintained at four month follow-up. The self-efficacy questionnaire showed significant changes in educator confidence in their ability to perform skills related to prevention programs, and this confidence was maintained at eight month follow-up. Follow-up data collected from each school team at the end of the school year revealed that the majority of the school prevention projects developed during the summer were successfully implemented (86% in '92 and 81% in '93), and that the projects facilitated change in their schools (93% in '92 and 68% in '93). Some of the specific changes that were noted included more parent/family involvement in school functions, greater teacher and parent involvement in drug education/prevention programs, and fewer student incidents of cigarette, alcohol, and drug use.
Participant satisfaction with the Institute have been uniformly high, even at 10 month follow-up. Participants valued the range of topics, quality of presenters, the applied focus of the Institute, and the opportunity to engage in project development away from the hectic schedule of the school year. One participant described the Institute as "essential for educators of the 90's."
The Institute will be offered again this summer from June 14-30 from 8 am to 12 noon on the University of MN campus. School teams are invited to submit applications (space is limited due to funding restrictions) prior to April 22. Institute participants receive academic credit plus a stipend for their participation. Call Elisabeth Nealy, Counseling and Student Personnel Psychology Program, for further information and an application (612/624-6827).
Questions that Shape Program Evaluation Strategies Laura Bloomberg
The following evaluation questions are derived from The Evaluator's Handbook, (Sage Publications, 1987), and Making Evaluation Work for You, (Univ. of Texas at Austin, 1992). They are typically used in developing program evaluation plans and in providing a monitoring mechanism for ongoing evaluation of long-term programs.
Needs Assessment
* What are the goals of the program?
* How can goal priorities be determined? * Is there agreement on
the goals from all groups? * To what extent are these goals being
met? * What are the areas in which the program is most seriously
failing to achieve goals?
Program Implementation
* How many participants and staff are taking part? * Is the
program running on schedule?
* Is the program being implemented according to plan? If not,
why? Is this important?
* How are time, money, and personnel allocated in the program? *
What activities do participants in the program become involved
in?
Goal Achievement
* What are the goals of the program?
* How can they be measured?
Formative Evaluation
* What are the goals and objectives?
* What are the program's most important characteristics? * Why
should these particular activities meet the stated goals? * Do
the activities lead to goal achievement? * What programs are
available as alternatives to this program? * How effective is
this program in comparison to other programs? * How costly is the
program?
Systemic change
* What changes in permanent structures have occurred or are
occurring as a result of this program?
* What policies have changed or are changing as a result of this
program?
* How has the behavior of key stakeholders at an administrative
and implementation level changed as a result of this program? *
What strategies have been most effective for actively involving
ALL stakeholders in implementation and decision-making? * What
resources (financial, in-kind, or other) have been leveraged as a
result of this program?
Family Impact Project: Linking Evaluation to Policy Development by Aimee Reedy, Consultant to the Family Impact Project
The Family Impact Project is a policy development project that grew out of the McKnight Foundation's Aid To Families In Poverty (FIP) initiative. This article provides a brief background on the McKnight FIP initiative and then describes the Family Impact Project, illustrating the link between evaluation and policy.
Between 1988 and 1993, McKnight funded 34 projects in the Twin Cities area with the goal of improving the lives of families who live in poverty. Four clusters of projects were funded: comprehensive programs; effective parenting/family stability programs; programs to broaden support for child care; and programs to enhance employment opportunities and income. All programs were required to participate in an evaluation for which specific funding was allocated. A final evaluation report was prepared. The full report, The Aid To Families In Poverty Program, A Synthesis of Themes, Patterns and Lessons Learned, will be on the Consortium Electronic Clearinghouse.
Four major lessons learned from initiative:
An additional key lesson learned from the FIP evaluation is that the "dissemination of model programs, public policy education and advocacy require special skills and resources that are not an inherent part of most projects." To a large extent, the FIP initiative did not address institutional change. This void was discussed at the final McKnight conference which brought together all of the FIP projects. The FIP funded projects identified public policy as critical to their ability to effectively address issues of poverty, and thus established a committee to identify the policy implications derived from The McKnight Foundation's FIP evaluation.
With additional funding from McKnight, this group is now a formal coalition known as the Family Impact Project. The coalition of 30 or so public and non-profit organizations exists to disseminate the lessons learned from McKnight's FIP initiative, extract the policy implications from those lessons, and develop and implement policy strategies. A unique feature of the Family Impact Project is the creation of a policy development model which depends upon the direct and ongoing input of program staff and families who are experiencing poverty. Family Impact Project is seeking opportunities to further disseminate the lessons learned from the FIP initiative. For more information about the project or to request a presentation of lessons learned, contact Aimee Reedy, 507/645-5098 or Laura Kadwell, Children's Defense Fund-MN 612/227-6121.
Consortium Update
Fraser Gives Speech/Receives Award
On September 29, 1993, Mayor Donald Fraser presented a speech
"Building Families, Strengthening Communities," for the
Consortium. Prior to his speech Mayor Fraser was given the
Consortium's 2nd Annual Award for Outstanding Service on Behalf
of Families. U of MN Regent, Jean Keffeler, presented the award.
In his speech, Fraser noted how the health of a community is intricately tied to the capacity of the community to nurture its children, youth, and families. Fraser gave examples of what policy changes need to take place at a national, state, school, and community level in order to improve the lives of children and families. His suggestions included: making sliding fee child care an entitlement for all families, initiating voluntary day-long kindergarten beginning at age 3, having good quality programs that strengthen families in the community, and encouraging support networks in neighborhoods.
Editors note: Copies of Mayor Fraser's speech are available upon request from the Consortium office, and the speech is on the Consortium Electronic Clearinghouse .
1,100 Books Collected for Children
Chi Psy (the U of MN Child Psychology student organization) and
the Consortium held the 3rd Annual Food for Thought Holiday Book
Drive. Over 1,100 books were collected in the Twin Cities. The
books were distributed to children through family shelters and
social service agencies that work with families in need. Thanks
to everyone who donated books and who helped to make this effort
successful!
MN Children's Summit
On October 28 the Consortium co-sponsored a Children's Summit
which brought together leaders from through-out the state for a
day long research-based, action oriented discussion about what
really makes a difference for children. The highlights of this
Summit are being produced into a 40 minute videotape that will be
used statewide to promote similar discussions. The next issue of
Consortium Connections will include a summary of what evolved
from the Summit and information on obtaining a copy of the video.
Working Families--Bigger Tax Refunds in 1994!
The Consortium, as a member of the MN Alliance for Children, is working to get the word out on tax credits that can benefit working families. The federal Earned Income Credit and the state Working Family Credit can provide up to $2,710 more dollars in a family's budget, even if the family owes no taxes! Refund amounts depend on family size and income; last year's average refund was $960.
Who is eligible?
Families earning between $1 - $23,050 in 1993 with at least one
child (including a foster child, step child or grandchild) living
at home are eligible. Even families that earn too little to owe
federal income tax or had no tax taken out of their paychecks may
be eligible.
To receive the credits families must:
- File for the Earned Income Credit on the federal tax form and -
The Working Family Credit on the state tax form.
Free help!
Eligible families can get free help in applying for their credits
from trained volunteers at many sites in Minnesota. For locations
call 297-3724 or 297-3725 in the Twin Cities or 1-800-652-9094
from other parts of Minnesota. For 24-hours-a-day information on
how the tax credits work, call 296-6417 in the Twin Cities area
or 1-800-937-KIDS in other parts of Minnesota. People with
hearing or communication impairments can call 1-800-829-4059 for
TDD.
Connection Corner
The report, "Behind From the Start", details county-by-county, the number of children enrolled in Head Start, receiving WIC assistance, and fully immunized by age two. The report lists specific actions Minnesotans can take to make these proven programs accessible to more children. For a full copy of the KIDS COUNT report "Behind from the Start" call 612/870-3660. The report is a cooperative effort of the Children's Defense Fund-MN and Congregations Concerned for Children. Data are from the MN Department of Jobs and Training and MN Department of Health.
You will also have an opportunity to leave a message about a successful children's project, action, or event in your community. During the legislative session the messages will be updated weekly.
Consortium Calendar
February 24 Youth Policy Forum on Southeast Asian Issues, 2-4:30 pm at the West River Place Conference Center in Minneapolis. Sponsored by Search Institute. For more info call Beki Saito or Laura Krump at 612/376-8955.
February 26 Parenting Works, a special series on parenting, will air on Channel 2 at 3 pm. This episode will focus on what to do when kids get out of hand. (Also airs on Channel 17 at 9 pm on February 28 and at 5 pm on March 2.)
March 9 12-1 pm. Managing the Transitions of Life. Noontime series sponsored by CEE and the Consortium. Speakers include Dale Blyth, Search Institute, and Joyce Weinsheimer, University Counseling. President's Room (3rd floor), Coffman Union, U of MN. For more info call the Consortium at 612/626-1212.
March 13 Marti Erickson and Free Spirit will present a concert at 7:30 pm, at the Great American History Theatre in St. Paul (in the old wing of the Science Museum). A reception will follow to celebrate the release of the group's new recording, "Light of My Life." For ticket info, call 612/292-4323.
April 13-14 Evaluation Conference (see pages 5 &6).
April 15 Children in the Shadows: The Fate of Children in Neglecting Families. All day conference sponsored by the Center for Advanced Studies in Child Welfare, School of Social Work, U of MN. Speakers include Jeanne Giovannoni, UCLA; Diana English, Washington State; and Byron Egeland, U of MN. For more information contact Chris Carlstrom at 612/626-1520.
April 18 The 3rd Annual MN Organization on Adolescent Pregnancy, Prevention and Parenting (MOAPP) Conference will be held at the Earle Brown Center, U of MN, St. Paul Campus. For details contact Theresa Winger at 612/296-6550.
April (date to be announced) Youth Policy Forum on Experiential Learning sponsored by Search Institute. For more information call Beki Saito or Laura Krump at 612/376-8955.
May 20-21 MN Round Table - Prevent Violence_For Kids' Sake (details page 7).
Submit entries for calendar to the Consortium office. Deadline for May newsletter is April 15.
Who is the Consortium?
Bob Blum, M.D., Professor and Director of the Adolescent Health Program at the U of MN, is a founding members of the Consortium's Steering Committee. Bob is an eloquent spokesperson for children and youth issues and he brings and important public health perspective to the Consortium's efforts.
Katie O'Brien, Project Facilitator of The Village Project in Stillwater, serves as Co-chair for the Advocacy Work Group. Katie brings extraordinary energy, commitment and innovation to her roles in advocating for children.
Laura Bloomberg, the Assistant Director of the Institute on Community Integration, U of MN, is a member of the Research Work Group and one of the driving forces behind the Consortium and MN AEA's spring evaluation conference.
John Couchman, Senior Program Specialist for The St. Paul Foundation recently joined the Consortium's Steering Committee. John has spearheaded the MN Alliance for Children's efforts to disseminate information about the Earned Income and Working Family Tax Credits.
And the Consortium is you!
CONFERENCE NOTICE AND REGISTRATION INFORMATION Building Effective Child and Family Programs: Research and Evaluation Perspectives
Wednesday, April 13 and Thursday, April 14, 1991 Thunderbird Hotel and Convention Center Bloomington, MN
A Conference Sponsored by
Children, Youth, and Family Consortium, University of Minnesota
Minnesota Chapter of American Evaluation Association
Co-Sponsors
Search Institute, PATH, Wilder Research Center
ABOUT THE CONFERENCE
Evaluations are critical to the success of programs serving
children and families. They can point out strengths and
weaknesses in programs, give direction for changes in
programming, and are essential for future funding of
programs. The following are the goals of this conference:
CONFERENCE GOALS
WHO SHOULD COME?
Staff from public and non-profit organizations serving children and families; evaluators, researchers, policy makers and others interested in children and family issues and evaluation.
KEYNOTE SPEAKERS
Michael Q. Patton, Ph.D. April 13
The Future of Evaluation in Programs Serving Children and
Families
Michael Patton is Director, Utilization-Focused Information and Training; on the Faculty of Union Institute Graduate School; Past President, American Evaluation Association; and author of Utilization Focused Evaluation.
Karen Pittman, April 14
Promoting the Positive: A Framework for Defining and Marketing
Positive Youth Outcomes
Karen Pittman directs the Center for Youth Development and Policy
Research at the Academy of educational Development in Washington,
DC. She previously served as Director, Adolescent Pregnancy
Prevention Policy Division, Children's Defense Fund.
CONFERENCE AGENDA
Wednesday, April 13
9:00 - 10:00 a.m. Registration
10:00 - Noon Keynote Address--Michael Q. Patton, Ph.D. The Future of Evaluation in Programs Serving Children and Families--Reaction Panel to follow Presentation
Noon - 1:15 p.m. Lunch/Roundtable Discussions
1:30 - 3:15 p.m. Session I - Skill Workshops/Presentations
3:15 - 3:30 p.m. Break
3:30 - 5:00 p.m. Session II - Skill Workshops/Presentations 5:00
- 7:00 Resource and Information Exchange and Reception
Thursday, April 14
8:00 - 9:00 a.m. Registration
8:30 - 10:15 a.m. Keynote Address--Karen Pittman Promoting the Positive: A Framework for Defining and Marketing Positive Youth Outcomes--Reaction Panel to follow Presentation
10:30 - Noon Session III - Skill Workshops/Presentations
Noon - 2:30 p.m. Luncheon and Closing Panel: Forging New Links: Research and Community Roles in Building Effective Programs for Children and Families
Session I - Skill Workshops and Paper Presentation a.
Everything You Need to Know When Hiring an Evaluator (skill) b.
Making Sense of Focus Group Data (skill) c. Lessons Learned in
Evaluating Early Childhood Education Programs (papers)
d. Perspectives on Defining and Using Outcome Measures (papers)
e. Ethnocultural Issues in Evaluation (panel)
Session II - Skill Workshops and Paper Presentation f. Getting
to Outcome Indicators (skill) g. Evaluating Child Abuse
Prevention Efforts (papers) h. Understanding the Process,
Outcomes, and Political Context of Community Collaborations
(papers)
i. Funders' Perspectives on Evaluation in a Changing Environment
(panel)
Session III - Skill Workshops and Paper Presentation j. Using
Qualitative Evaluation to Guide Program Development (panel)
k. Foster and Adoptive Parenting: What Makes Them Work? (papers)
l. Evaluating a Multi-Program Poverty Initiative: A Focus on
Utilization (papers)
m. The Dual Challenge: Getting the Right Data and Using It for
Effective Action(panel)
Hotel Information
Thunderbird Hotel
2201 East 78th Street, Bloomington, MN
Located just north of the Met Center, just south of 494 and south
of the airport.
1-800-328-1931
Daily Rates are $67 single, $73 double, $6 each additional adult
(tax not included)
Reservations must be made by March 29 for these prices.
Facilities are accessible to handicapped persons.
Resource Exchange Opportunity
On Wednesday, April 13, 1994, an opportunity to exchange research
and evaluation interests, needs, materials and information will
be offered from 5:00 to 7:00 p.m. An invitation is extended to
nonprofit agencies, interested individuals and all University
departments to attend. To reserve a table for an exhibit, please
indicate below. Reception and cash bar. Call Kendra Brooks at
612-642-5367 to reserve a table for an exhibit at the Resource
Exchange.
REGISTRATION FORM registration deadline - March 15 (no refunds after that date)
Registration Fees:
$60 2-Day Conference Registration
$40 2-Day Student Registration
$35 Wednesday only
$35 Thursday only
Make checks payable to Wilder Research Center and mail check and registration to: Wilder Research Center, 1295 Bandana Blvd. N., Suite 21, St. Paul, MN 55108
For further information call Marilyn Conrad at 612-647-4603