The URL can be used to link to this page

Home My WebLink AboutMINUTES - 07151997 - C43 ;r F&HS - 01 TO: BOARD OF SUPERVISORS 5E Contra FROM: FAMILY AND HUMAN SERVICES COMMITTEE '. C osta July 7, 1997 y" °° County DATE: S 'CA V u*r SUBJECT: STATUS REPORT ON THE WORK OF THE POLICY FORUM SPECIFIC REQUEST(S)OR RECOMMENDATION(S)&BACKGROUND AND JUSTIFICATION RECOMMENDATIONS: 1. ACCEPT the following report from the Family and Human Services Committee on the status of the work of the Policy Forum. 2. DIRECT the County Administrator's staff to report to the Family and Human Services Committee during the month of December on the activities of the Policy Forum since July. BACKGROUND: On May 6, 1997, the Board of Supervisors approved a report from our Committee which included the following recommendations: 1. ACCEPT the attached report from the County Administrator's Office on the work of the Policy Forum. 2. DIRECT the County Administrator to report to the Family and Human Services Committee again during the month of June 1997 on the results of the work currently being undertaken by the Policy Forum and its four Task Forces and INVITE the Chairs of the four Task Forces to meet with our Committee to present those reports. 3. REQUEST the staff to the Policy Forum,when they report back to the Family and Human Services Committee in June 1997,to identify specific action steps the Policy Forum and its Task Forces plan to pursue in the coming months so our Committee can see what specific directions the Policy Forum will be taking. CONTINUED ON ATTACHMENT: YES SIGINATURE: RECOMMENDATION OF COUNTY ADMINISTRATOR -RECOMMENDATION OF BO D OMMITTEE APPROVE O ER SIGNATURE s : MARK DeSAULNIER DONNA GEJR ACTION OF BOARD ON July APPROVED AS RECOMMENDED X OTHER VOTE OF SUPERVISORS _ I HEREBY CERTIFY THAT THIS IS A TRUE X UNANIMOUS(ABSENT ) AND CORRECT COPY OF AN ACTION TAKEN AYES: NOES: AND ENTERED ON THE MINUTES OF THE BOARD ABSENT: - -ABSTAIN: OF SUPERVISORS ON THE DATE SHOWN. ATTESTED July 15, 1997 Contat: PHIL BATCHELOR,CLERK OF THE BOARD OF cc: Bounty Administrator SUPERVISORS AND COUNTY ADMINISTRATOR Members of the Policy f=orum (Via CAO) Sara Hoffman, Senior Deputy County Administrator BY— DEPUTY "•a F&HS - 01 4. REQUEST the Policy Forum and its staff to consider ways in which the Family and Human Services Committee can stay in better touch with the work of the Policy Forum and its Executive Committee so that the Board of Supervisors is a part of the decision-making process. On July 7, 1997 our Committee met at the Bay Point Service Integration site with the County Probation Officer, Terry Starr, and staff from the County Administrator's Office. Sara Hoffman and Caroline Kelley from the County Administrator's Office reviewed the attached report with our Committee. We would commend the Task Force reports to the Board members to review in an effort to get a real picture of the work of the Policy Forum and its Task Forces. Additional written status reports were provided by Caroline Kelley on the work of the Parenting Task Force and the Safety Task Force. Sara Hoffman noted that the Policy Forum was initially given a very large and somewhat vague role and is trying to focus and define that role more precisely. The work of each of the four Task Forces was reviewed by Ms. Hoffman and Ms. Kelley. We are pleased to see the progress that is being made. We continue to look for action as opposed to too much process. The Outcomes Task Force has identified five community outcomes which the Policy Forum has approved. The Policy Forum has also approved the four criteria for selecting outcomes for Contra Costa County, which include that the measure is clear, compelling, measurable, and relevant. The Family Support Task Force has done a review of child care programs in the County and has concluded that there is no coordinated system for the provision of child care services. The Task Force has made recommendations for how to proceed with child care programs and will next address the transportation system in the County as an important support service. The Safety Task Force has prepared the Children's Report Card which our Committee will have an opportunity to review on July 28, 1997 and will report to the Board of Supervisors on August 5, 1997. The Task Force has also identified the need to coordinate more closely with JSPAC (the Juvenile Systems Planning Advisory Committee) on prevention and early intervention of delinquency and family violence. The Parenting Task Force is working to market the Policy Forum and the Task Force's goals and activities. We look forward to continuing to work closely with the Policy Forum and its Task Forces. -2- CONTRA COSTA COUNTY OFFICE OF THE CO UNTY ADMINIS TRA TOR -- 651 Pine Street, 10th Floor Martinez, California 94553 Telephone: 335-1052 Fax: 646-1353 DATE: June 9, 1997 TO: Supervisor Mark DeSaulnier Supervisor Donna Gerber Family & Human Services Committee FROM: Sara Hoffman 4- Senior Deputy Administrator SUBJECT: Status Report on Children & Families Policy Forum The Children &-Families Policy Forum met on May 14, 1997 to hear status reports from its task forces: • Outcomes Task Force • Family Support Task Force • Parenting Task Force • Safety Task Force A copy of each task force report is attached, together with a meeting summary that delineates the actions taken by the Children's Policy Forum. The Executive Committee of the Children's Policy Forum focused on piloting outcomes at their May 19, 1997 meeting. The study Accountability for Results in County Government: Implementation Strategy for Adopting Outcome Based Measurements in Contra Costa County (attached) was the basis for their discussion of appropriate programs for piloting outcomes. The study provides some excellent insights on use of outcomes in our efforts to improve the well being of children, youth and families as well as pointing out potential pitfalls in implementation of outcome based accountability. cc: Children's Policy Forum (without attachments) t � Date: May 19, 1997 To: Policy Forum and Task Force members From: Caroline Kelley, staff Subject: Summary of May 14, 1997 meeting Attachments: *attendance record* Task Force summaries *meeting evaluation responses *June 11, 1997, Bay Area Partnership meeting supporting comprehensive, community-based, school-linked supports and services for children,youth and families. Following self introductions by Policy Forum members/staff and Task Force members, reports by Task Force chairs were presented with findings and recommendations for action. Discussion and consensus on recommendations followed each report. Discussion points: • The Executive Committee will review and respond to the Task Force reports. • Task Force chairs will meet with the Family and Human Services Committee of the Board of Supervisors in June. • Task Forces will identify and include additional representatives for missing expertise • Task Forces will align their goals with Community Outcomes and Measures • The distinction between the Family Support and Safety Task Forces will be clarified • The District Attorney's office should be included in the Policy Forum • The Policy Forum should include linguistic, ethnic, culture and gender representation • Task Forces will continue their work and report back with next steps at the July._ meeting of the Policy Forum. • Interns may be available for Task Force supervision in gathering information Action taken: • The name of the Policy Forum is changed to the Children and Families Policy Forum Next meeting: Wednesday,July 9, 1997 Ito3Pin Probation, Sequoia Room 50 Douglas Drive, Martinez Mark your calendars for a meeting October e. from 1 to 3 pm. If after the July meeting,it is necessary for the Forum to meet before that, we will meet on Wednesday, September 10th Action taken for each task force, discussion items generated for committee consideration and next steps follow: Outcomes 'I i.sk Force Jim Bouquin reporting , The Policy Forum approved: • the five Community Outcomes • criteria for selecting outcomes for Contra Costa County Next steps: • The Task Force will produce the first annual Contra Costa County Children's Report Card by June 1997 • The Task Force will seek the endorsement of the Community Outcomes from the Board of Supervisors, community-based organizations, other local government and professional associations and community members • The Task Force will refine and finalize the list of community measures emphasizing a proactive, strengths-based focus • Outcome measurements will be aligned with Task Force goals • The task force will study methods to evaluate programs and resources Family, Support Joanne Durkee and Sara Hoffman reporting The Policy Forum approved: • the elements of family suppert systems • the findings and recommendations of the task force on child care including: ■ the Board of Supervisors calling together the stakeholders involved in Child to identify critical service delivery, organizational infrastructure issues and options with a child and family focused perspective ■ that the plan developed involve consensus and collaboration ■ development of an implementation action plan for issues under local control utilizing existing resources and leveraging new resources ■ development of an advocacy action plan which coordinates existing and new resources and advocates for necessary support from state and federal government Additional action steps agreed upon: • the Family Support Task Force will identify the major stakeholders • the Family Support Task Force will continue to monitor the child care issue Next steps: • .The Task Force will identify and invite the major transportation stakeholders for study and evaluation Other items for Task Force consideration: • food be added to the graphic of necessary support systems for families • elements aligned with Community Outcomes for"preparation and ready to succeed in school" and"family self sufficiency" Safety Task Force Gemma Pasto reporting The Policy Forum agreed: • the Board and other stakeholders endorse the need for a prevention and early intervention strategy for delinquency and family violence prevention • to the usefulness of an assessment of prevention and early • intervention needs and resources • an easily accessed "Crisis and Resource Response System" is needed Next steps: • The Task Force will continue to meet to further define and set direction • The Task Force will supervise an intern for resource assessment if available Other items for Task Force consideration: • studying the feasibility of program and resource evaluation • a Task Force name change include"Delinquency and Family Abuse Prevention" • a name change reflect a proactive approach • integration of a proactive focus to the"Community Prevention for Success"graphic Parentinz Task Force Charlotte Weinstein and Susan Petersen reporting The Policy Forum agreed: • a study to inventory current children and family resources and existing linkages be supported by the Board of Supervisors • a centralized,user friendly Resource and Information System to link individuals,families and communities to needed resources is important • the Parenting Task Force continue to plan, promote and advocate for parents by facilitating the implementation of a community based, continuum of support and services which is strengths-based, recognizing individual's and family uniqueness Next steps: • The Task Force will develop organizational structure and expand membership, prioritize goals and tasks and finalize a one year work plan Other items for Task Force consideration: • Task Force Charge including"mentoring"with parenting Summary May 19, 1997 Policy Forum Executive Committee Meeting Attending: Phil Batchelor Bill Walker, Art Miner, Joe Ovick, John Cullen, Terry Starr, Mike Tye, Lois Haight, Scott Tandy, Rob Henderson, Sara Hoffman, Audrey Seymour, Caroline Kelley. Audrey Seymour, consultant to the East Bay Community Foundation and the County, summarized the concept for the proposed Future Corps: a collaborative of private industry, foundations and the public sector that would enhance resources for prevention and early intervention programs for children and families in Contra Costa County. Audrey is developing a preliminary business plan for the Future Corps. Review of Policy Forum Meeting Decision: The Executive Committee will review the organization, process and work plan of each Task Force. Next Steps: The Outcomes Task Force chair and staff will be invited to the next Executive Committee Meeting. Piloting Programs for Outcomes Decision: tach agency will select one (or more) program(s) for piloting outcomes (see attached list of suggested selection criteria). Possible candidates include Summit Center (Health Services), PO's in schools (Probation),Youth Employment (PIC), Diversion Program (Hercules), Head Start (Community Services), Employment (Social Service). Next Steps: Candidate programs will be discussed and agreed upon at the next Executive Committee meeting. Notes: 1) On-going training and technical assistance will be available for pilot programs. 2) For questions regarding pilot selection, please call Sara Hoffman, 335-1090. Next Meeting: Monday, June 16, 1997 11:00 a.m. - 1:00 p.m. Superior Court, Department 10 Subsequent meetings - the third Monday o each month QUESTIONS TO GUIDE SELECTION OF PROGRAMS FOR PILOTING OUTCOMES EASE OF IMPLEMENTATION: , • How many staff does the program have? How many people does the program serve? (Smaller programs may make the shift to outcomes more easily than larger programs.) • Is the program currently subject to federal or state restrictions that might inhibit a. shift toward outcomes accountability? .For example, does an outside funding source (public or private) currently subject the program to overly burdensome process accountability measures or does the program have a very low administrative cap? • How does the program currently evaluate its programs? (A program that is currently not evaluating or is using process-based measures may require a great deal more assistance than a program that is currently using a mixed outcome/process approach or an outcomes approach.) • Does the program currently collect any data that would facilitate the shift toward outcomes? INTEREST IN MOVING TOWARD OUTCOMES: • Are program administrators and line staff interested in moving toward outcomes? • Does the program have the ability to leverage more funding if it moves toward outcomes? • Does the program see the shift toward outcomes as a direction that its funding sources are likely to move toward anyway? (In such a case, piloting outcomes in the County may provide them with a head start.) • Is there already a national evaluation model or some other expectation of how the program will be evaluated that will provide the program with the evaluation information that it needs? BENEFIT TO PROGRAMS: • Does the program have enough operational flexibility (ability to change) to make necessary programmatic changes as a result of outcomes evaluation? • Does the work of the program lead to impacts on its client population that can be measured over a relatively short time frame, thereby making outcomes a useful feedback mechanism? BENEFIT TO SYSTEM: 0 Is there a potential that shifting this program toward outcomes will reduce the 1 - need for crisis or safety net services? INTERDEPENDENCIES: • Are the outcomes that this program strives to achieve so closely linked to those of another program that the programs should have joint outcomes? ' RESOURCES: • Generally, what types of additional resources might the program need to make the shift toward outcomes? (Computer hardware, computer software, evaluation/technical experience of staff, sufficient staff, etc.) is �. i i € 3 � x r�h 4 -i pit. .. h f h - . NEW ril_2 ': �,Yil ri�u.,w��i�ts�{`C)'+��K,�d✓ �KA}.a; � iu i?.��y{ yPRIPTr� 4mL� Alm ddEft �4 i f r.. h S i k. .i.. I MEMBERS OF THE OUTCOMES TASK FORCE OF THE POLICY FORUM Jim Bouquin CCC Cont. Alliance - New Connections David Carrillo United Way Rich Clarke Mt. Diablo Unified School District Mary Foran Health Services Department Ruth Goodin East CCC Sara Hoffman CCC Administrator's Office Larry Larson CCC Office of Education Art Miner Private Industry Council Steve Peavler Social Service, Family Preservation Grace Schmidt Housing Authority Chuck Stevenson WCC Youth Services Bureau Mike Tye CCC Police Chiefs' Assn. Doug White Health Services - Mental Health Staff: Nina Goldman,.CCC Administrator's Office mot p C`� rte '� p �, �',► '�+' a y,. 9 ®u e k Force Charge . . . • Develop a set of community outcomes for children&families in CCC • Identify measures to track the status of each of the community outcomes • Explore linkages between community outcomes&measures and performance measures for County programs. Definitions • Community Outcome: A desired condition of well being for children,families and communities. • Community Measure: An indicator,for which data is available,that allows the desired outcome to be quantified ,' 1 Sources for Outcomes &Measures 0 Policy Forum Executive Committee 0 United Way M CCC Service Integration Team M Placer County's SMART Program 0 LA.County's Children's Score Card M Children Now's County Data Book Q Children's Scorecard for Santa Barbara County and others... Proposed Community Outcomes ♦Children ready for and succeeding in school +Children A youth healthy and engaged in preparing for productive adulthood ♦Families that are economically self sufficient *Families that are safe,stable and nurturing ♦Communities that are safe and provide high quality of life for children A families ............ Criteria for Selecting Community Measures Ef Clear Ef Compelling 0 Measurable Ef Relevant 2 Next Steps . . . • Obtain endorsement of proposed Community Outcomes for CCC by the _ —Board of Supervisors —Community based organizations —Local governments —Professional associations —Community members Next Steps . . . • Refine and finalize list of community measures(Outcomes Task Force) • Producc first annual Contra Costa County Children's Report Card Next Outcomes Task Force Meeting Monday,June 23, 1997 2:30- 4:30 p.m. 651 Pine St.,Room 105 Martinez 3 PROGRESS REPORT: OUTCOMES TASK FORCE OF THE POLICY FORUM (May 14, 1997) The Outcomes Task Force of the Children's Policy Forum met three times during the m�.nths of April and May 1997. The Task Force members undertook a process to (1) develop a set of community outcomes that articulate in cross-agency, cross-program terms what we want for the children, families and communities of Contra Costa County, and (2) begin identifying community measures to help quantify the achievement of the desired community outcomes. Community Outcomes The attached list of community outcomes builds upon the Policy Forum's vision statement and outcomes identified by the Executive Committee of the Policy Forum. 'In addition, this list draws upon work done by the United Way of Contra Costa County, the Contra County Service Integration Teams and Placer County's SMART Program. The Outcomes Task Force reviewed outcomes identified from the above-listed groups and initiatives and undertook a prioritization process aimed at consolidating like outcomes and selecting outcomes which have the greatest significance for Contra Costa County. Community Measures The attached list of community measures represents the Outcomes Task Force's first attempt to translate the community outcomes into concrete, quantifiable indicators. In order to facilitate the process of identifying the measures, the Task Force members applied the criteria that community measures should be: clear, compelling, measurable and relevant (see attachment). Fortunately, a significant amount of work has been done by other initiatives to sort through community measures and identify those that meet these criteria. The Outcomes Task Force reviewed community measures ' . included in Los Angeles County's Children's Score Card, Children Now's County Data Book, the Children's Scorecard for Santa Barbara County as well as measures identified by the Executive Committee of the Children's Policy Forum and Contra Costa County's Service Integration Team program. Next Steps (1) Obtain the endorsement of the proposed Community Outcomes for Contra Costa County by the Policy Forum. (2) Obtain the endorsement of the proposed Community Outcomes for Contra Costa County by the Board of Supervisors, community-based organizations, other local government and professional associations and community members. (3) Refine and finalize the list of community measures (Outcomes Task Force). (4) Produce the first annual Contra Costa County Children's Report Card. CRITERIA FOR SELECTING COMMUNITY MEASURES FOR CONTRA COSTA COUNTY CLEAR: Are the measures common sense enough to have meaning for a wide range of stakeholders? Given that a number of measures are inter-related, is there one measure that can serve as a good proxy or representative for a number of others? COMPELLING: Are the measures chosen powerful and convincing enough to maintain the interest of diverse audiences? MEASURABLE: Is high-quality data available on this measure on a regular and frequent basis? RELEVANT: Is there an established relationship between the measure and the community outcome it represents? DRAFT COMMUNITY MEASURES FOR CONTRA COSTA COUNTY Outcome #1: CHILDREN READY FOR AND SUCCEEDING IN SCHOOL. • Percentage of children fully immunized at age 2 • Percentage of eligible children enrolled in Headstart or equivalent • Number of homeless children and youth • Rates of(public) school attendance • . Percentage of school children and youth enrolled in school lunch program • Average standardized test scores • High school graduation rate Outcome #2: CHILDREN AND YOUTH HEALTHY AND ENGAGED IN PREPARING FOR PRODUCTIVE ADULTHOOD. • Percentage of births to mothers receiving late or no pre-natal care • Percentage of adequate birthweight births (over 2,500 grams) • Percentage of children with health insurance • Teen birth rate • Substance use, self-reported in last 30 days (7th, 9th, 11th grade) (Alcohol, Tobacco, Illicit Drugs) • Juvenile probation:petitions filed • Number of children referred to mental health services Outcome #3: FAMILIES THAT ARE ECONOMICALLY SELF-SUFFICIENT. • Percentage of persons below the poverty line • Percentage of children below the poverty line • Unemployment rate • Number of persons receiving Food Stamps (no cash grant) • Percentage of eligible families in subsidized housing (or wait list info) • Number of homeless families • Median household income • Percentage of owner occupied homes Outcome #4: FAMILIES THAT ARE SAFE, STABLE AND NURTURING. • Number of family violence reports • Number of children and youth in out-of-home placement • Number of child abuse cases reported and substantiated Outcome #5: COMMUNITIES THAT ARE SAFE AND PROVIDE HIGH QUALITY OF LIFE FOR CHILDREN AND FAMILIES. • Violent deaths age 0-19 -Homicides - Suicides -Accidents • Violent deaths age 20+ - Homicides - Suicides -Accidents • Adults on probation • Neighborhood stability. same house 5 years + • Neighborhood playground and park availability, • Percentage of communities with local governing boards • Number of reported hate crimes i �s �.•'. ,. ®� I '* Family Support Task Force Joanne Durkee, Co-Chair Mt. Diablo Unified School District Sara Hoffman, Co-Chair County Administrator's Office Gloria Omania Assemblyman Torlakson's Office Rick Aubry Rubicon Programs Linda Chandler Private Industry Council Cathy Chiverton Supervisor Gerber's Office Sandy Lacy Supervisor Gerber's Office Suzanne Strisower Supervisor Uilkema's Office Scott Tandy Community Services Wilda Davisson Community Services, Child Development Tony Colon Social Services, Head Start MaryKay Miller Social Services, FACSAC Catherine Giacalone Contra Costa County Office of Education Felicia Estrada Contra Costa County Youth Commission Lynn Basket Hospital Association of Northern California Kate Ertz-Berger Child Care Council Amalia Gonzales-Del Valle Health Services, Substance Abuse Prevention Carol Hatch Congressman Miller's Office Deya Brashears Diablo Valley College, Child Development Caroline Kelley County Administrator's Office Paul Ward, Staff Social Services Department, GAIN i Family Support Task Force Tasks . . . • assess current and needed supports for parents newly entering the workforce and those at risk of losing employment •focus on issues such as child care, transportation,transitional MediCal,drug and alcohol use • provide linkages to workforce development,economic development and Social Service employment redesign Task Force Vision Every parent and family should have access to support systems which allow them to achieve economic self sufficiency. i The is Task Force Process • Fact Finding • Analysistssues identification • Recommendations First Topic: Child Care • Matrix for Analysis -available -accessible -affordable -quality I 2 i Fact Finding • Ir formation from individuals,organizations and departments -The Child Care Council -Community Services - Social Service -Child Care Provider-Martinez Child Care -CCC Child Care Task Force -Regional Child Care Campaign Policy Statement -Outline of California's Child Care Dcvelops,ant Services Plan Available . . . • Offers diversity of models for parent choice • Meets the needs of parents,children and youth • Includes birth to school age • Includes wdended dare care and activities for elementary through age fourteen • Provides year roundtfull day/shift and weekend care • Includes sick care,respite and special needs • Provides adequate numbers and types of slots Accessible . . . • Offers a"one stop"system with resources, referrals and subsidy eligibility • Provides geographic accessibility • Addresses transportation issues for low income consumers 3 i Affordable . . . • Provides subsidies based on income • Maximizes Federal,State and local funding • Encourages employer support programs • Recognizes financial support requirements of low income areas Quality . . . • Nurtures and prepares children for success in school • Supports social skills and workforce readiness for older youth • Protects consumers through licensing&quality standards • Requires quality improvement activities for exempt providers • Provides safe,healthy,physical environment Quality . . . /.APtGo('al�Thuf • Provides well-trained,compcnsated,culturally competent caregivers Insures appropriate child/stafratios • Includes education for parents and caregivers Includes family support services 4 Task Force Findings • Child Care-critical support for working parents • Welfare Reform-increases demands Task Force Findings Child Care does not operate as a"system" to: • meet the needs of working parents • maximize use of resources Major D*fferences: • governance • funding • service delive-Ty models Task Force Findings System development limited by: • insufficient child care funding • categorical restrictions Major stakeholders not working together to: • examine child care issues • create a child care action plan 5 Recommendations for the Board of Supervisors Bring togerher major child care stakeholders to: • agree upon elements of quality,affordable, accessible child care system for working parents • identify critical service delivery and organizational infrastructure issues and options • develop consensus and an action plan -implunentation -advocacy I 6 i Family Support Task Force Task Force Charge: The Family Support Task Force will assess current and needed supports for parents newly entering the workforce and those at risk of losing employment. The Task Force will focus on issues such as child care, transportation, transitional Medi-Cal, drug and alcohol use, and other issues. Linkages will be made to groups such as work force development , economic development and Social Service employment redesign. Task Force Vision: Every parent and family should have access to support systems which allows them to achieve economic self-sufficiency. Task Force process: • fact finding • analysis/issues identification • recommendations First Topic: Child Care Matrix for analysis - • available • accessible • affordable • quality Fact Finding: Information from individuals, organizations and departments - • The Child Care Council • Community Services • Social Services • Child Care Provider, Martinez Child Care • Contra Costa County Child Care Task Force • Regional Child Care Campaign Policy Statement • Outline of California's Child Care Development Services Plan i Child Care Support Elements Available • offers diversity of service models for parents • meets the needs of both working parents, children and youth • includes birth to school age for social, emotional, physical and psychological development • includes extended day care and activities for elementary through age fourteen • covers full day, year round and accommodates shift and weekend workers • includes sick care, respite and special needs • provides adequate numbers and types of slots for all ages Accessible: • offers a seamless, interconnected, "one-stop" system to the consumer, including resources, referrals and subsidy eligibility • provides geographic accessibility by neighborhood,or work site • tal-Is into account transportation issues for low income consumers Affordable: • provides subsidies based on income • maximizes use of available Federal, State and local funding • encourages employer support programs ie. utilization of tax dollars for child care • recognizes additional financial support requirements in low income areas Quality. • nurtures and prepares children for success in school through language, pre- mathematics and social skills development • sustains development of social skills and workforce readiness for older youth + protects consumers through licensing& quality standards ie. facilities, trained and Trustlined staff • requires on-going quality improvement activities for exempt providers • provides safe, healthy, age appropriate physical environments • provides well-trained, well-compensated, culturally competent caregivers • insures appropriate child/staff ratios • includes education for parents and caregivers to enhance child development, parenting and family solidarity • includes needed family support services ie. counseling, stress management, and substance abuse Task Force Findings • Child Care is a critical support for working parents and welfare reform will put increased demands on the system. • Child Care in Contra Costa County does not currently operate as a "system" to best meet the needs of working parents or to maximize use of resources. • Major differences of opinion regarding governance, funding, and service delivery models are currently dividing the Child Care community. • Limited child care funding and extensive categorical restrictions has discouraged collaboration and greatly limited the designing and development of a broader, effective child care system. • All major stakeholders are not working together in a process of examining child care issues and creating a child care action plan for Contra Costa County. Task Force Recommendation That the Board of Supervisors bring together the major stakeholders in child care including parents to: • agree upon the necessary elements of a quality, affordable, accessible child care system for the working parents of children and youth of Contra Costa County. (Consider the Family Support Task Force findings) • identify critical service delivery and organizational infrastructure issues and options from the perspective of all stakeholders • develop consensus, where possible, on critical issues with an implementation action plan for issues under local control and an advocacy action plan for issues under state or federal control • create a child care implementation action plan that will: 1. coordinate existing resources 2. leverage new resources 3. advocate for necessary support from state and federal government i ti _ t Parenting Task Force Charlotte Weinstein,Chr Interfaith Coalition/St. Vincent de Paul Susan Petersen, Chr Mt. Diablo School District Danna Fabella Social Services Debra Mason Community Rep East County Ellen Peterson Drug & Alcohol Task Force Sharon Tucker St. Vincent de Paul Eileen DeMaria Child Abuse Council Barbara Bysick Family Stress Center Carrie Frazier Family Wellness Center Angelica Partida East County REACH Albione Becnel Ujima Family Recovery Catherine Giacalone Contra Costa County Office of Education Mary Foran Health Services Kathy Renfrow Homeless Task Force Stephan Betz Community Services - Head Start Shelley Murdock UC Cooperative Ext. June Harrison CC Child Care Council W Anne Millang Bovo Parent Mary Kay Miller Social Services Deya Brashears Diablo Valley College Caroline Kelley, Staff Co. Administrator's Office Parenting Task Force Task Force Charge • Increase community awareness and involvement in Parenting Issues • Develop media and marketing approaches to support Parenting • Promote"Best Practices"and other activities Parenting Vision • All Contra Costa County parents will have access to support and information which will empower them to be the primary mode4 advocate and nurturer of their children,enabling their children to become responsible,contributing members of the community I 16 "When a society values its children, it will cherish its parents." As"4 author unknown Every Parent Deserves and Needs: r to be cherished V to be prepared and educated for his/her role V personal support and connectivity r a continuum of localized,sustainable community-based support Parenting Task Force Goals • Develop a continuum of care with a strengths-based focus including mentoring,education,and support • Market resources and models • Support parenting efforts • Promote an"employer friendly" approach to parenting 17 The Parenting Continuum R., Task Force Recommendations Request the Board of Supervisors to. • Support a study to inventory convent children& family resources and linkages • Authorize the development and maintenanoe of a centralized Resource and hiformation System to link individuals,families and communities to needed resources Task Force Recommendations Request the Policy Forum to: • Authorize the Task Force to facilitate the implementation of a community based continuum of support and services for parents which is strengths-based and recognizes the uniqueness of each individual and family. A# 18 The Parenting Continuum Next Steps . . . • Develop Task Force structure and membership • Determine goals • Prioritize tasks • Finalize one year work plan Next Parenting Task Force Meeting Wednesday,May 28, 1997 2-4 p.m. District Attorney's Meeting Room 50 Douglas Drive,First Floor Martinez I 19 : .:. Thankyou Task Force Groups! j 20 e� M � � .t a. err Rs � e u� o a e 4 Parenting Task Force Parenting Task Force charge: The Parenting Task Force will develup a plan to support the vision of the Policy Forum by - • increasing community awareness and involvement in parenting issues • developing media and marketing approaches to support parenting • promoting "best practices" and other activities Parenting Vision: Every Contra Costa County parent will have access to support, and information which will empower them to be the primary model, advocate and nurturer of their children, enabling their children to become responsible, contributing members of the community. Core Beliefs: Every parent deserves and needs - • to be cherished • to be prepared and educated for their role as parents • personal support and connectivity • a continuum of localized, sustainable community based support Task Force Goals • Develop a continuum of cane including mentoring, education and support with a strengths-based parenting focus for diverse populations at neighborhood levels • Develop and deliver innovative and multi-faceted marketing approaches featuring resources and models • Support Countywide, regional and community parenting efforts • Encourage and facilitate an employer friendly approach to parenting aM-M*e::r z1r: ;sr ir. sE: n 4' , wane 7.3� r� t 9 tt i r r e VT It r is ea { f 1 1 I P _Y.' T r rs r G f ---j y .+tr gc r s 1,n{�.E w ' a 4 s. � r- F t 1 R C..2.:.. .. . .: s a 4 �En Task Force Recommendations 1. That the Board of Supervisor's support a study to inventory cturent children and family resources and existing linkages. 2. That the Board of Supervisor's authorize the development and maintenance of a centralized Resource and Information Center to link individuals, families and communities to needed resources. 3. That the Contra Costa County Policy Forum Parenting Task Force continue to plan, promote and advocate for parents by facilitating the implementation of a community based, continuum of support and services for parents which is strengths-based and recognizes the uniqueness of each individual and family. 9� p�vEN�j�N 0 v g1°� � y� �� y�� Safety Task Force Gemma Pasto, Chair Probation Bianca Bloom County Office of Education Terry Starr Probation Shirley Marchetti Consumer Representative REACH Capt. Mark Moczulski Antioch Police Department Mike Tye Police Chiefs Association Greg Rothaus Hercules Police Department Ken Duckert Mt. Diablo Unified School District Amalia Gonzales Del-Valle Health Services/Substance Abuse Stephanie Mann Klout for Kids Carol Hatch Congressman Miller's Office Bruce Sterling Juvenile Court Referee Ray Merritt Social Services/CPS Gloria Sandoval Battered Women's Alternatives Cassandra Youngblood Health Services/Violence Prevention Project Caroline Kelley, Staff County Administrator's Office t Safety Task Force 10� ° ACCOUNTABILITY Task Force Charge • To reduce juvenile arrests and violence in families by —Rzviewing altemative strategies —Assessing existing resources and linkages Steps to Goals • Identify programs and linkages • Determine critical needs • Focus on prevention • Integrate Task Force priorities with national research models i 1 r - How Prevention Systems Work I�v.��mv�aon r}+sort�k�c Il.c.nrsrlca CREN e � DIS MMZ ,TSRYMZM DEUNQVWCY . RESULTING IN HEALTHY YOUTH Task Force Findings For prevention and intervention to be effective,a comprehensive community based continuum of timely,integrated education,intervention and support services with graduated consequences are necessary Task Force Findings Resources and services for victims and perpetrators in Contra Costa County are currently fragmented and not readily available. 2 Task Force Findings Services are currently centered on offenders rather than high risk populations. Task Force Findings Major stakeholders are not currently collectively engaged in a process to identify and evaluate resources or to develop an integrated service delivery system. Continuum Services & Supervision 10 �Less Restrictive More Restrictive—O. ;ac:.:x':Bi,T<,.t:, he= MIONAM i rAV Gn 1,c. � Hit PM bt-R N.6f A M dM A et 914 A �• 7'- 3 Recommendations for the Board of Supervisors What an interdisciplinary prevention and early intervention strategy focusing on delinquency and family violence be developed to include: —collaborative,community cantered approaches —mentoring and diversion —extended day programs Recommendations for the Board of Supervisors ®That an RFP to study prevention and early intervention needs be authorized to. —inventory and define resources —construct a continuum of available programs&services —clearly define roles and linkages —determine overlaps and effectiveness —identify gaps —develop a collaborative plan and strategy for a prevention continuum Recommendations for the Board of Supervisors OAcknowledge the need for an easily accessed "Crisis and Resource Response System"and conduct a feasibility study to develop an action plan to: —provide timely response,intervention and resources —collect data and update resource information —link wstomers to resources —provide opportunities for collaboration —access sources of funding 1 4 Recommendations for the Policy Forum • Rename the Task Force the Delinquency &Family Volence Prevention Task Force • Include the District Attorney's office in the Policy Forum • Insure that representation is inclusive of Contra Costa demographics including linguistic,ethnic,culture and gender and that all constituencies are heard 1 5 o r 0�0 v y � r t� z A n Policy Forum Safety Task Force Safety Task Force Charge: The Safety Task Force will develop a plan to reduce juvenile arrests and violence in families by - • reviewing alternative strategies • assessing existing resources and linkages Task Force process: • began to identify programs and linkages • determined critical needs • focused on prevention • reviewed county, state and national research Research models: • Contra Costa County's "Continuum of Care" by the Juvenile System's Planning Advisory Committ • California's Task Force to Review Juvenile Crime - the Juvenile Justice Response • Combating Violence and Delinquency: The Juvenile Justice Plan by the Coordinating Council for OJJDP Task Force Findings • For prevention and intervention to be effective, a comprehensive community based continuum of timely, integrated education, intervention and support services with graduated consequences are necessary. • Resources and services for victims and perpetrators in Contra Costa County are currently fragmented and not readily available. • Services are centered on offenders rather than high risk populations • Major stakeholders are not currently engaged in a process to identify and evaluate resourdes or to develop an integrated service delivery system ,gyp O co R, o � va VA c Oona `� CIO o pot � � n a Task Force Recommendations That the Executive Committee of the Policy Forum review, approve and forward to the Board of Supervisors and other stakeholders the following recommendations: • That a prevention and early intervention strategy focusing on delinquency and family violence, incorporating a comprehensive, interdisciplinary strategy, be developed to include - * collaborative, community centered approaches * mentoring and diversion * extended day programs Second, that the Board authorize the Policy Forum to prepare an RFP to study local prevention and early intervention needs involving delinquency and family violence. The study would - • inventory and define resources • construct a continuum of available programs and services • clearly define roles and linkages • determine overlaps, effectiveness and gaps • develop a collaborative plan and strategy for a prevention continuum Third, that the Board concur that an easily accessed "Crisis and Resource Response System"is needed and to authorize the Executive Committee to conduct a feasibility study to develop an action plan which would - • provide timely response, intervention and resources • collect data and update resource information • link customers to resources • provide opportunities for collaboration • access sources of funding Additional Recommendations • Rename the Task Force the Delinquency and Family Violence Prevention Task Force • Include the District Attorney's office in the Policy Forum • Insure that Policy Forum representation is inclusive of demographics - linguistic, ethnic, culture, gender and that all constituencies are heard Parenting Task Force Task Force Charge The Parenting Task Force will develop a plan to increase community awareness and involvement in parenting issues, develop media and marketing approaches to support parenting and to promote "best practices" and other activities. Conclusion: • that the charge to the Parenting Task Force can best be met by first developing a proposal for a strengths-based parenting continuum which includes education, mentoring and support resources Accomplishments: 1. Expanded task force participation 2. Scheduled monthly meetings to report progress and to provide resources 3. Explored first steps to accomplish task force charge 4. Identified initial sub-committees and charges: • Parenting Resources * identify and inventory parenting providers and services * assess availability, affordability and accessibility of services • Strengths-based Parenting * develop a proposal for a parenting continuum based upon developmental models Next steps: • Establish sub-committees with clear focus and leadership Recommendations: . • that the Children and Families Policy Forum develop and utilize a Marketing Task Force to promote Forum and task force goals and activities • that county commissions and advisory boards be linked to the Children and Families Policy Forum for mutual benefit Safety Task Force Charge: To reduce juvenile arrests and violence in families by • reviewing alternative strategies • assessing existing resources and linkages Goals: • Focus on prevention • Identify programs and linkages • Determine critical needs • Integrate Task Force priorities with national research models The Safety Task Force is developing an interdisciplinary strategy focusing on prevention and early intervention of delinquency and family violence. Accomplishments: • Expanded task force representation of regions and organizations providing prevention and intervention resources and referrals • Confirmed alignment of task force charge and goals with state and federal guidelines for community violence prevention mask forces and the Policy Forum's Community Outcomes • Identified critical prevention and early intervention strategies in addition to those in the 1997 Health Services Survey • Determined supervision and direction for the task force intern to develop a prevention continuum to be completed by August • Investigating collaboration with JSPAC(Juvenile Systems Planning Advisory Committee) Next steps: • Formalize collaborative relationship with JSPAC • Complete the family and youth focused prevention and early intervention continuum • Determine task force focus areas Recommendations to the Policy Forum: • Determine goals and linkages to county commissions and advisory boards • Develop a community education strategy of what must be accomplished and why • Encourage communities to develop new collaborative attitudes and approaches to issues ie. developing alternative schools • Expedite links between multiple departments and providers through a comprehensive resource and referral system • Work for timely service delivery at local levels for kids in the system through collaboration between schools, law enforcement, probation and the courts ie, regional courts and juvenile facilities A^ � 5 � ' z� AC FOR RESULTS IN O MENT: x Impleme € trator ." ,ting Outcome-Based M emeCounty x 7 v� May 1997 Susan Fletcher Prepared for: County Administrator's Office Contra Costa Countv, California The author conducted this study as part of the program of professional education at the Graduate School of Public �i Policy.University of California at Berkeley. This paper is submitted in partial fulfillment of the course requirements for the Master of Public Policy degree. The judgements and conclusions are solely those of the author.and are not necessarily endorsed by the Graduate School of Public Policy,the University of California or by any other agency. TABLE OF CONTENTS SECTION PAGE Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Section I: Outcome-Based Accountability in the Public Sector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Section II Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Section III.- Criteria II:Criteria for Implementing Outcome-Based Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Section IV Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Appendix I: Interview List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Appendix 11: 14 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 CHARTS Chart I.- Components :Components of an Outcome-Based Measurement System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Chart 2: Application of the Outcome Model at the Program Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Chart 3: Contra Costa Programs Included in the Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Chart 4: Feasibility of 14 County Programs to use Outcome-Based Measurements . . . . . . . . . . . . . . . . . 27 1 EXECUTIVE SUMMARY A discussion has surfaced across the public sector about the need to incorporate accountability for results in government. Federal, state, and local initiatives are advancing these discussions through the implementation of outcome-based evaluation systems. These initiatives have taken various forms, but their basic tenet, that government be accountable for the dollars it receives in terms of concrete impacts, is fundamentally reshaping government today. R Contra Costa County is currently examining the possibility of moving toward outcome-based measurements in the area of child and family programs. Contra Costa's approximately 89 child and family programs have goals that range from meeting b,-sic needs to creating opportunities for economic and social change. While all programs are held accountable for performing certain processes when delivering services, in the past most of the child and family programs have not been asked to specify how their services produce measurable impacts in the lives of the children and families served. County programs currently conduct program evaluations when required by particular funding sources. Since funds arrive at the county from a variety of funding streams,through the federal government,the state, the county, and private funders, each program has different mechanisms in place to determine the effectiveness and efficiency of program operations. While some programs have extensive accountability systems that are linked to outcomes, the majority of programs utilize informal accountability mechanisms that measure process but do not measure the impact of their work. As a result, it is difficult for program managers, department heads, the County Administrator's Office, and the Board of Supervisors to assess the effectiveness of each program, program cost-effectiveness, or to determine if the services provided are meeting the needs of clients and the community. Moving toward a system of outcome-based accountability requires a fundamental rethinking of the role of the public sector. In order to make such a transition throughout the county system, most programs would require additional resources in terms of staff time, computer hardware and software, training, and/or technical guidance. Implementing an outcome-based accountability system in Contra Costa County would provide a framework for county-wide goal setting and give the administration and county programs a better sense of the individual and collective impact of their work. Based on literature reviews, interviews with a number of county staff, and research on other states, counties, and cities that have implemented outcome-based systems,this paper recommends that Contra Costa County pursue a system of outcome-based accountability. This recommendation is made on the condition that Contra Costa proceed by piloting outcomes to several programs and assessing the success of the model within those programs before it is imposed county-wide. Two phases of implementation are recommended: • Phase One involves the initiation of a process to establish county-wide outcomes. For outcome measures to be widely felt,the work of programs should support larger community goals. Key stakeholders in the community and at the program-level should be involved in this process. Since these groups will be most impacted by the outcomes identified by the county, their participation should help ensure that the system designed will be useful at the program/client level. • Phase Two involves the actual implementation of outcome meas>>rements at the program-level. Since resources are scarce and not every program will benefit to the same degree from outcome measures, it may be best to select a few programs to pilot the new evaluation system. These programs will highlight universal implementation challenges and, as such, become guides and models for other programs making the transition in the future. 2 INTRODUCTION The goal of this paper is to provide Contra Costa County decision-makers with(1)a set of criteria for deciding whether adopting a system of outcome-based measures is beneficial for the county, and(2) an implementation strategy for moving to outcome-based measurements that specifically addresses the current status of county programs in relation to evaluation systems. In order to accomplish this goal the paper is divided into four sections. Section I begins with an analysis of current and past evaluation systems in the public sector. It then defines a new trend in government thinking, outcome-based measurements, that asks programs to be accountable for the results of the work they perform. Scarce resources, public opinion, and the private sector have all contributed to this movement. The application of the outcomes model at the program-level is presented and the distinction between community- wide outcomes and program-level outcomes is made. This section concludes by presenting some current models of outcome-based measurements at the federal, state and local level. Section II identifies the county programs included in this study and the methods used to collect information on each program. The results of the hour-long interviews with 14 programs in Contra Costa County that deliver child and family services are included as case studies in an appendix to this document. Each case study summarizes the methods of evaluation currently undertaken by each program, data collection resources, and the program's use of, or confidence in, outcome measures. Section III develops a set of criteria to help determine whether or not to implement an outcome-based model. The criteria were developed by examining other researchers' works and by listening to the experiences and opinions of Contra Costa County programs. By linking the criteria with the actual experiences of Contra Costa County programs, county decision-makers can see a fuller picture of the benefits and challenges raised by adopting outcome-based measurements. The criteria to be considered are: (1)Are sufficient resources available to ensure that service delivery will not be negatively impacted by the "costs"associated with shifting to outcome-based accountability?, (2) Can all program goals be incorporated in an outcomes model?, (3) Will federal and state regulations restrict movement towards outcome accountability?, (4) If multiple interventions from multiple programs are needed to impact a client, can one program be held accountable for the end result?, (5) Will outcome measures be useful for direct staff providers, administrators, and outside funders?, (6) What is the value of this model to the Contra Costa Board of Supervisors and what political benefits/risks do they face as a result of implementing outcomes? " Section IV concludes with a recommended strategy that the Board of Supervisors and the County Administrator's Office could pursue in order to implement outcome-based measurements at the program-level. This strategy is directly influenced by the information gathered from the 14 county programs. The interview data is condensed and categorized in order to identify the types of programs best suited for piloting outcome-based measurements. A summary chart is provided that ranks each program by the level of resources available to begin collecting outcome-based measures, the difficulty each program would have, and the usefulness of this model to improving the program's effectiveness and efficiency. A two phase recommendation strat��-- :s p-oposed. 3 SECTION L• OUTCOME-BASED ACCOUNTABILITY IN THE PUBLIC SECTOR Role of Accountability in the Public Sector Prior to the early 1990's, evaluation within government was primarily expenditure-based. Government accounting mechanisms would track the amount of money spent on a particular service, the manner in which a department allocated those dollars, and whether or not proper accounting systems were in place and followed.' Accounting did not track the impact of the dollars spent, rather it tracked the process of disbursement and reimbursement of funds. David Osborne and Ted Gaebler have theorized that the reason why dollars were the focus instead of impact is that funding for government programs is a highly politicized process. Examining impact may show that popular programs need to be eliminated, or that the services provided are not meeting the needs of the community. If the dollars spent were not creating a positive impact, then politicians may fail to get reelected. In testimony to this theory, Osborne and Gaebler quoted one state legislator who said that "pleasing the voters is our performance evaluation."' Historically, programs serving children and families have also focused on measuring process rather than results. In general, this is not the fault of the individual program but rather a result of the requirements imposed by funding sources. Who was served and how well that service was delivered has largely been the concern of funders of child and family programs. So, for example, a program serving juvenile offenders might be asked to provide information on the number of juveniles served in a year, rather than on the recidivism rates of those served. Researchers at the Casey Family Program have found that, "Practice has seldom reached beyond the activities of the social worker to determine what difference those methods have made in people's lives,what role they have played in reaching goals that society deemed desirable, and thus the degree to which services are worthy of financial and moral support."' The 1990's have seen a major shift in thinking in regards to government accountability. Three factors have contributed to this shift: scarcity of resources, public opinion, and the private sector. Resources for government programs have become incredibly scarce. Government was once considered a safety net for the entire population; now the safety net has time limits and Campbell, Michael D., "Outcome and Performance Measurement Systems: An Overview," Alliance for Redesigning Government, National Academy of Public Administration, http://www.clearlake.ibm.com/Alliance/clusters/op/overview.6.html., pg. 1. Z Osborne, David, and Gaebler, Ted (1992), Reinventing Government: How the Entrepreneurial Spirit is Transforming_the Public Sector,Addison-Wesley Publishing Co., Inc., pg. 140. 3 Traglia, James, "Implementing an Outcome-Oriented Approach to Case Planning and Services, Improvement," pg. 4.5. 4 restrictions. The advent of welfare reform is testimony to this new philosophy. As federal dollars dry up for local services, counties are finding that tough choices must be made about how ` and where dollars should be spent. Without federal and state dollars for essential services like drug treatment,public housing and long-term welfare payments, counties like Contra Costa, will have to decide whether resources are diverted to fill gaps in funding. A system that identifies which government programs are effective and cost-efficient at what they do is necessary as resources become scarce, creating the need for difficult tradeoffs. The second factor contributing to the movement towards government accountability is public opinion. Over the last decade, many taxpayers have come to view government as large, ineffective, and inefficient. Many argue that this is because taxpayers have seen government expenditures increase without a corresponding improvement in social conditions like public safety, poverty, and the high school dropout rate.' Mark Friedman notes that, "Voters have spoken clearly. They want more for their money. They have called for more and better services, but they also have demanded balanced budgets and cuts in income and property taxes. After more than a decade of chronic deficits, they want government at all levels to operate more effectively and efficiently."' Since most government programs have systems in place that track process instead of results it is difficult to alleviate the fear taxpayers have that funds are not being well spent. Until taxpayers know how well current dollars are being spent,they may continue watch-dogging government and failing to approve future appropriations. The private sector has also paved the way to public sector results-based accountability. Programs that were once considered solely the domain of the public sector like health care, education, and social services today are all potential candidates for private-sector contracted services. The private sector has always been held to demonstrable results and is willing to step in and take over government functions if there is room to make a profit. Examples now exist, nation-wide of instances where decision-makers and taxpayers, frustrated by the public sectors inability to show impact or improvement, chose to contract"government" services to the private sector. Public health services have been impacted the most by these developments through the emergence of medical and mental health managed care. However, it has not always been a smooth transition to privately managed public health services. The recent backlash against HMOs illustrates the widely held view by the populous that the private sector is often willing to sacrifice quality for cost-savings. The result of external competition from the private sector has been a concerted effort by public health agencies to ensure that quality services are provided that are tangible and measurable. In many states and counties, including Contra Costa, the heath care ' Friedman, Mark, "A Strategy Map for Results-Based Budgeting," The Finance Project, September 1996, pg. 3. 5 Friedman, pg. 3. 5 field has been the first to adopt aggressive outcome measures in order to compete with the private sector. Outcomes Defined A general working definition of outcomes is "the results of government programs as measured by the differences they make, for example, in the economy or in program participant's lives."' Currently most government programs use process measures to evaluate, for example, by reporting on the number of clients seen, the speed at which paperwork is processed, or the number of errors reported in a given time period. Adopting a system of outcome measurements requires programs to shift focus away from what goes into a program to what comes out. Programs that transition to a system of outcome-based measurements generally start by identifying program goals, inputs, outputs, outcomes, and performance indicators. Most programs can already identify program inputs, outputs, and goals. Programs transitioning to outcome-based measures generally need to link services already performed with tangible measurements that can be used to show progress towards program goals. In order to understand the components that make up an outcome-based system,the concepts are defined below. Similar versions of this chart have appeared in other researchers' works(Osborne & Gaebler, United Way, KPMG Peat Marwick, and the National Performance Review). Chart l: Components Of An Outcome-Based Measurement System Goal The reason for all activities performed. The objective of the program. Input The human and financial resources that go into delivering the service. Output Units of service, work, or units produced. The results achieved through the program's actions Outcome that help meet the goal of the program. A quantifiable measurement that captures the Performance Indicator desired outcome of the program. The closest proxy for the identified outcome. Comparison of inputs expended to outcome or Efficiency output achieved. Factors that could help explain the results achieved. Factors that are outside of the control of the y Explanatory Information' department that need to be considered in relation to the efficiency measurement. 6 General Accounting Office, 1996. ' Campbell, pg. 2. 1 6 Chart 2 takes two programs -- a summer job training program and mental health services for youth-- and shows a possible application of the outcomes model at the program-level. Applying outcomes at the program level is distinctively different than developing community- wide outcomes. At the program-level outcomes should enable management and line staff to identify specific impacts resulting from the money and time expended. Programs develop quantifiable measurements (performance indicators)that show progress towards their goals. In the summer youth employment example in Chart 2, the number of youth returning to school is a proxy for the outcome of youth becoming better skilled and more employable as a result of the program. Chart 2: Application Of The Outcome Model At The Program Level Goal Input Output Outcome Performance Efficiency Explanatory Indicator Information Provide Wages for Number of Youth are Number of kids Wages spent on Increase in work youth,classes youth who more who return to youth as compared minimum experience and training, worked and employable school or get a to the number that wage means for low- personnel attended and aware of job after the end the program that fewer income costs. classes over skills needed summer. with a job or back clients can youth. the summer. to succeed. in school. be served. Identify and Case Number of Mental health Number of kids Dollars spent on Welfare address the managers, contacts with needs of kids with diagnosed case managers, Reform mental dollars spent families in addressed. problems that education and may lead to health needs on education need,hours They are are at home, in outreach as greater of kids and and outreach. spent with a living a school,and not compared to the numbers of families. family. normalized in trouble. results of kids homeless,more life. living with economic normalized lives. disparity,and a greater need for county mental health services. Program Outcomes vs. Community-Wide Outcomes This paper presents a strategy for developing program-level outcome measurements. The development of community-wide outcome measures may greatly facilitate the movement to program-level outcomes. Ideally, program-level outcomes feed into a larger set of clearly defined community-wide priorities. Community-wide outcome measures have been developed in a number states, most notably Oregon. In general,the process begins by holding public meetings to identify key measures that reflect the "health"of the community. Once the measures are identified then baseline data on each condition is gathered. Some measures include: literacy rates, employment rates, number of teenage pregnancies, and environmental pollutants. "The public communicates its priorities to government on the results to be achieved for the state through the collective action of all of the relevant stakeholders, including government. Government then reports back to the public on its success in achieving those results."' By deciding which measures are important as a community, governing bodies can then allocate resources in order to impact those areas. Benefits That Could Be Gained From Outcome Measurements Conceptually, outcome measures move away from traditional government systems that place control in the accounting and fiscal divisions. Instead outcome measures place control directly at the program-level, asking managers to be accountable for certain results. The National Governor's Association finds that, "Focusing on results rather than process and measuring results achieved over time promotes strategic thinking and collaboration within government and between government and communities to achieve the desired results. It also shifts the emphasis from one of ensuring compliance with rules and regulations to learning more about what aspects of programs work, implementing best practices at scale, and investing in strategies that have a substantial return on investment."' Several authors have written on the merits of outcome-based thinking. The most notable is Lisbeth B. Schorr who with the collaboration of Frank Farrow, David Hornbeck and Sara Watson identified the hopes that public administrators have invested in results-based accountability. Administrators"hope that results-based accountability could be the key to: • freeing schools, health care, social agencies, and other human services from the rules that prevent them from operating flexibly in response to the needs of those they serve; • restoring the public's faith that both public and private human service institutions can accomplish their intended purpose; and • encouraging communities to be more planful, more intentional, in how they support children and.families."10 Outcome-based measurements can offer the public sector a great opportunity for change. Successful outcome-based systems should give line workers the flexibility and independence to serve clients in the most efficient and effective manner. However, the greatest benefit of outcomes may not necessarily be the in the system that is designed, but rather in the discussions that occur regarding outcomes at the program and administrative level. If public sector s"Benchmarking Results for Children and Families,"The National Governor's Association, pg. 2. 9 "Benchmarking Results for Children and Families," pg. 3. 10 Schorr, Lisbeth, "The Case for Shifting to Results-Based Accountability," pg. 13. 8 employees buy in to the concept that all work needs to produce results,then a major shift in government thinking will have occurred, regardless of the system put in place. Status Report: Where Outcomes Are Used Today Federal Legislation Paving the Way to Outcome Measures At the forefront of the movement to results-based government accountability are President Clinton, Vice President Al Gore, and the U.S. Congress. Building from an election platform of"reinventing government",this presidency has worked to streamline government operations and reduce duplication of services. These efforts are described in the "National Performance Review" released by Vice President Gore in 1993. This document states that "federal, state and local government attention should focus on mutually agreed-upon measurable outcomes for public service delivery."" Four pieces of legislation were passed to facilitate this process: the Government Performance and Results Act of 1993 (GPRA), the Chief Financial Officers Act of 1990, the Government Management Reform Act of 1994, and the Information Technology Management Reform Act of 1996 (ITMRA). Paramount to the development of outcome measures nation-wide are the GPRA and the ITMRA. The GPRA "requires that federal agencies prepare strategic plans that define an agency's mission and long-term general goals, annual performance plans containing specific targets, and annual reports comparing actual performance to the targets set in the annual performance plans."" The ITMRA has similar legislative requirements but links the development of these management systems to specific information technology. Included within the GPRA is the provision of waivers "which if granted would effectively dispense with certain administrative procedures, requirements, and controls in return for anticipated future performance improvements."" Specifically the NPR stated that "federal financial support should be provided to achieve broad goals, but also be tailored to real local needs. Rather than defining accountability by inputs, transactions, error rates, and failure to progress, the federal government should hold state and local governments accountable for performance."14 The practice of waivers, along with block granting, has come under some attack. " National Performance Review, Vice President Al Gore, 1993. 'Z Gore, Al, "Reaching Public Goals: Managing Government for Results", National Performance Review, October 1996. 13 Kravchuk,Robert S, Schack,Ronald W., "Designing Effective Performance-Measurement Systems Under the Government and Performance and Results Act of 1993," Public Administration Review, Vol. 56,No., pg. 348. 14 National Performance Review, 1993. 9 Some fear that the provision of waivers will allow essential services to be cut to some of the neediest populations. State-Wide Initiatives in Outcome-Based Measurements Over ten states have adopted some form of comprehensive outcome-based measurement system. Some of the most lauded programs are happening in Oregon, Minnesota, and Texas. Each state has customized the application of outcomes to their specific management systems. For example, some programs are linked to the budget structure and others are separate from the budgetary process. California has not yet adopted a state-wide system of outcome-based measurements. However, California has begun to take some preliminary steps in this direction. California's efforts will be discussed later in this section. Oregon Benchmarks The State of Oregon which pioneered the system of benchmarking in 1991 saw an opportunity to utilize the GPRA, and proposed the "Oregon Option", a pilot project that tests out the concepts in the NPR on a state-level. Oregon seized the opportunity to put these concepts included in the GPRA in place, and building upon their existing system of benchmarks, they began a demonstration project with the federal government that allowed them to combine funding streams, provide multi-year funding streams, eliminate program restrictions, contract for measurable results, renegotiate funding amounts and rates, and most importantly empower those closest to front-line service to choose the delivery mechanisms, initiative, and investment criteria they deem most suitable.15 Oregon believes that by eliminating some federal restrictions and giving programs more management control, program managers can be held accountable to outcomes. "Under the Oregon Option, federal, state and local partners work together to define outcomes - in the form of benchmarks -that they want to achieve with federal dollars. State and Local service providers then have the latitude to determine how best to achieve those outcomes.16 Oregon is considered to have the most comprehensive system of outcome-based measurements. Their program is structured around the establishment of benchmarks. Benchmarks measure the progress Oregon has made in achieving its strategic vision of"a state of well-educated, competent people living in thriving communities, working in a well-paying, competitive economy, and enjoying a pristine environment."" Oregon identifies historic and present data on 90 core benchmarks (272 total)that impact the vision described above. The state-level Oregon Progress Board, a legislatively-mandated bipartisan board headed by the 15 Oregon Option, pg. 4. 16 Oregon Benchmarks: Standards for Measuring Statewide Progress and Institutional Performance, Report to the 1995 Legislature, Oregon Progress Board, December 1994., pg. 2. 17 Oregon Benchmarks, pg. 2. 10 Governor, sets baseline measures for each benchmark until the year 2010. This model was _ developed through community-wide forums and it is used for establishing budget priorities, focusing management attention on high priority matters, and in community and local government planning. The state has also asked counties to develop local benchmarks. This will be discussed under the section"Multnomah County". Minnesota Milestones Minnesota has followed the model developed by Oregon. The program was established in 1991 because Minnesotans"recognized that the future they wanted for themselves and their children would require time to develop; but without goals and measures of results, arriving at that future would be unlikely."" Minnesota's model, like Oregon's, is citizen-based and identifies broad goals and 79 "milestones" (same definition as a benchmark)to measure progress. Progress is monitored through report cards issued to the public and those documents are used to identify budget priorities. In a survey of Minnesota managers after the completion of the 1994 Performance Reports, mangers found that the initial time and expense of producing the reports outweighed the benefits gained. However,the survey respondents believed that in future years the benefits gained from data analysis and ability to improve program performance would outweigh the costs of collecting the outcome data. In addition, agencies estimated that it cost over 1 million dollars to develop the reports.19 Texas Tomorrow: Performance-Based Budgeting Texas adopted a strategic planning system in the budget year 1994-1995. The state did not engage in any incremental or start-up programs. As soon as the program was underway, it was fully incorporated into the budget process. Government agencies identify the outputs and outcomes to be generated, set targets for achievement, and then request appropriate dollars to meet this goal. The budget in essence becomes a contract between the department and the state to achieve certain results. Elected officials are able to use this information to determine if, based on past contract performance, this department is able to meet the desired results. California California has taken some preliminary steps to bring outcome measures to the state. Assembly Bill 1741 was a first step towards improving the functioning between states and counties by giving counties more flexibility over program operations. Since Contra Costa County is an AB 1741 county, it can request the right to blend various categorical child and ' family service funds in order to achieve specific outcomes. The county is required to develop a '8 Minnesota Milestones, Summary, 1996 Progress Report., pg. 1 19 Minnesota Legislative Budget Committee Report, July 20, 1995., pg. 3. 11 strategic plan that outlines community priorities, programmatic changes, and the need to have blended funds. AB 1741 allows Contra Costa County to identify children and family service programs and transfer those funds into a child and family services fund. "The county Child and Family Service Fund must be used to fund comprehensive, integrated services for high risk, low- income, multi-problem children and families in alternative and innovative ways, as detailed in the county strategic plan. The fund may be used to provide services in one or more designated geographical areas within the county or to targeted populations."10 A second project occurring in California that may eventually provide guidance for Contra Costa County is performance-based budgeting. In 1993, California enacted the Performance and Results Act (SB 500, Hill)which established four pilot projects in performance-based budgeting. Included in the goals of this project were: managers are provided with sufficient operational flexibility to achieve stated outcomes, outcome measures are the primary focus of management accountability, and productivity benchmarks measure progress toward strategic goals. Four state departments are currently running pilots in performance-based budgeting and their experiences may eventually be a source of support for counties in California interested in moving towards outcome-based management structures. Multnomah County, Oregon:Are Program-Level Outcomes Working Here? The National Governor's Association reported that, "Some of the best work with the benchmarks framework is being done by county governments in Oregon. At least seven of Oregon's 36 counties are voluntarily developing comprehensive, locally-oriented benchmark systems."21 Counties in Oregon that have developed local benchmark systems link local measures to larger state-wide goals. County councils are required to evaluate success based on the county's ability to impact short-term objectives and long-term benchmarks." In 1994, the Multnomah County Commission developed the system of"Key Results". The "Year-end Performance Report for 94-95" declared that, "Key Results measure the performance of county programs or services. Each program (except management or administrative functions) is responsible for developing and reporting Key Results that measure program outcomes or efficiency."" Key Results have been collected for the last two years. The Multnomah County Evaluation Specialist finds that county programs are not using this information in any sort of 20 "Moving Toward Accountability for Results:.A Look at Ten States' Efforts," Summer 1995, http://www.ccsso.org/ibsum952.html, pg. 6. 2' Friedman, pg. 9. 22 "Benchmarking Results for Children and Families,"pg. 8. 21 Year-End Performance Report, Fiscal Year 1994-95, Multnomah County, Oregon, pg. 15. 12 beneficial way. He finds that line staff, program managers, county administrators and elected officials do not know what to do with the information now that it has been collected. A major problems is the failure of program information to translate into budgetary decisions. As the county struggles to cut $40 million out of a$860 million budget, the system of Key Results is not being used in the decision-making process. The Evaluation Specialist believes that"you can make programs collect outcome data but you cannot make them use the information or make the County Administration act on it.1114 The examination of the Multnomah County case and some of its shortcomings revealed several factors critical to successful implementation of an outcomes-based accountability system. The Evaluation Specialist emphasized that in Multnomah County the process of developing and implementing the Key Results program was essentially top down. The Board of Supervisors initiated the process with very little input from the departments. Program managers and service delivery staff had varying, but typically limited, roles in the process. Top- and mid-level departmental staff went through general training and received proscribed consultation from a consultant. Departments then came up with measures which they negotiated with the Board of Supervisors; the Board went on to change some of the measures. The Evaluation Specialist stated that more intensive one-on-one consultation and follow up that went well beyond the general training provided would have helped the departments to come up with better measures. In addition, he stated that service delivery staff had a very limited role in the process of developing measures; he recommended that line staff should be more involved in the process, which would result in outcomes that are meaningful to the people who must collect them. In addition, he recommended that a system be put in place to review the outcome measures on a regular basis and to revise them in cases where they are not working. The Evaluation Specialist also noted that Multnomah County made the mistake of initiating the Key Results program in all 10 county departments simultaneously and on an extremely fast time schedule. The process began in the fall of 1993 and measurements were collected in the spring of 1994. As a result of the rapid six month time frame, many of the measures are not reflective of the programs' work,they are difficult or impossible to track and they have little utility to line staff, program administrators and the county Board of Supervisors. He recommended a far more incremental process of implementing outcomes, with the initial focus on a select group of programs that are ready and willing to make the transition. Another lesson from Multnomah County is that it appears that too great an emphasis was placed solely on outcome measures. In fact, the work of some programs may not be appropriately assessed using outcome measures alone. A combination of multiple evaluation methods may be more effective. The Evaluation Specialist recommended looking at the Baldridge Quality standards which include seven organizational competencies. Outcomes focus z4 Interview with Jim Carlson, Multnomah County Evaluation Specialist, May 5, 1997. 13 all the attention on one of these competencies: Information Analysis. He recommended the need for seeing a shift to outcomes as one part of a broader movement toward improving program performance and suggested that attention simultaneously be focused on other key areas of quality improvement. The goal of the Multnomah Key Results program was to increase program accountability and to use the data collected to make budget decisions. It appears that due to the hurried time line and lack of buy-in from county staff, these goals have not been met. To date, the Board of Supervisors has not been able to use the data collected on Key Results to make budgetary decisions, in part because there are no comparable measures across programs. Finally, while Multnomah County has invested a substantial amount of money in improving computer resources over the past several years, the Evaluation Specialist noted that relatively little investment was made in the resources needed to move county programs toward an effective and efficient outcomes-based system. Making this transition requires more than technology, it requires having training and human resources as well. Sunnyvale, California: Performance Pioneer Sunnyvale, California has one of the most comprehensive performance management systems in place at the local level. It was developed in the 1970s and has been the incubator for future programs like the GPRA. OMB testified in front of the State Committee on Governmental Affairs that "Sunnyvale's system stands out as the single best example of a comprehensive approach to performance measurement that we have found in the United States...One underlying reason for the success achieved in Sunnyvale is the fact that every program manager uses the system to plan, manage, and assess progress on a day-to-day basis."" Sunnyvale's model begins with the general plan. The plan is comprised of seven elements and for each element a problem statement, set of goals, policies and actions are identified. The budget uses the same categories identified in the general plan. Budget items are identified by task instead of overall service and departments are on a pay-for-performance system. At the end of the fiscal year, planned results are compared with actual results and the efficiency, productivity and effectiveness of each program is analyzed in the report. Sunnyvale finds that as a result of these management systems, it uses 35%-45% fewer employees to deliver most services than do other cities of comparable size and type, and its per unit cost for service has dropped by 38%from 1986-1993.26 zs Mercer,John, "The performance management and budget system of the city of Sunnyvale, California," pg. 1. 16 Mercer, pg. 12. 14 The United Way Induces Its Member Organizations to Use Outcome Measures Other models outside the public sector exist that are helping influence the movement towards outcome-based measurements in the public sector. For example, the United Way which funds many community-based organizations, has been working over the last few years to help their member organizations define the impact of their services and develop the tools to measure the results of their programs. By providing training for their member organizations on outcome measures, the United Way is initiating a project that could eventually impact the public sector since many government agencies contract services to community-based organizations. The United Way funds programs that can demonstrate an ability to impact nine broad community goals. Conclusion At the federal, state and local-level successful models of outcome-accountability now exist. As a relatively recent movement little data is available on the impacts of incorporating outcomes at each level of government. The implementation challenges faced by each program can guide Contra Costa in its move toward outcome-accountability. In particular, the case of Multnomah County offers a warning against implementing an extensive system of outcome data collection without knowing how the information will be used, who will use it, and at what cost. 15 SECTION II: METHODOLOGY Background on Contra Costa County Contra Costa County has begun to take the initial steps towards adding accountability in government through outcome-based measurements. The 1996-7 Child and Family Services Budget identifies seven primary goals for the county: • Improve the health and safety status of children and families. • Increase the number of children who succeed in school. • Increase the number of children and families who have adequate basic housing. • Enhance developmental opportunities for children. • Increase the economic stability and self-sufficiency of families. • Increase physical safety and promote justice. While these goals have not been formally adopted, they can be seen as a first step towards a more comprehensive plan that will set goals for the county. The Children's Policy Forum has formed an Outcomes Task Force dedicated to examining county measures of well-being and specific program outcomes. In addition, Contra Costa County plans to release a children's report card this year that will be the first attempt at benchmarking certain conditions in the community and setting targets for progress in the future. Part of the challenge for Contra Costa County is that the State of California has not begun to use outcome-based measures in any sort of systematic way. Local governments in other states have had the experience and expertise of state-wide systems to draw from when establishing local models. Contra Costa County is initiating this process,largely on their own. Program-Level Interviews Prior to proposing a strategy for a county-wide outcome-based measurement system, the County Administrator's Office wanted to assess current accountability systems in place at the program-level and to assess the capacity of programs to shift to outcome accountability. Specifically, they wanted to know the current mechanisms county programs use to measure program effectiveness, whether programs use or are familiar with the concept of outcome-based measurements, and what resources would be needed to transition county programs to a system of outcome-based measurements. In order to gather this information, interviews were conducted with 14 programs in Contra Costa County. The interviews lasted approximately one hour each and some of the questions included: • What are the goals of this program? • How do you currently evaluate the performance of this program? • How is the information you gather used? 16 • What percent of your time is spent on data analysis? _ • If you could collect any data to show the effectiveness of your program, what would it be? The programs selected for this study ranged in size, scope, current methods of evaluation, and the anticipated ease or difficulty for this program to move to a system of outcome-based measurements. The interviewees were informed that this study was designed to gather information and that the findings would be reported to the County Administrator's Office and the Board of Supervisors. The group of programs selected for this study are not random and are not considered to be a perfect reflection of the experiences of all county programs. This study should not be viewed as a complete picture of the evaluation systems used in all county programs or as representing the ability of all programs within the county to move to a system of outcome-based measurements. The table below identifies the programs included in the study, each program's budget and the services provided. Readers should consult the case study appendix for specific information on each program. Chart 3: Contra Costa County Programs Included In The Study Program Department Services Born Free Health Services Counseling, Support Groups, Referrals. Child and Adolescent Health Services Assess and treat mental health conditions in children Mental Health and youths. Child Development Community Provide child care and training. Centers Services CHDP Health Services Maintain a CHDP health provider network, assess, refer, and monitor health problems. Court-Ordered Residential Probation Place youth outside of their homes and plan for Placements possible family reunification. Electronic Monitoring/ Probation Monitor clients sentenced to this program. Home Supervision Emergency Response Social Services Receive and assess child abuse reports. Foste:Youth Services/ Mt. Diablo USD Provide integrated services to community residents, Healthy Start support foster youth in school. Independent Living Skills Social Services Provide support and practical skill building for juveniles leaving foster care system. Orin Allen Youth Probation House and provide rehabilitation services to 74 boys .Rehabilitation Facility sent by the courts. Prevention Programs Health Services Offer workshops, assemblies and services to prevent certain activities/events from occurring. Section 8 Housing Authority Subsidized housing and support services. SYETP PIC • Provide summer employment and training to low- income youth. Youth Development Cooperative Train the trainers, Community Needs Assessments, Extension Youth Classes. 17 SECTION III: CRITERIA FOR IMPLEMENTING OUTCOME-BASED MEASURES Prior to deciding how to implement an outcome-based measurement system,the Contra Costa County Board of Supervisors needs to be able to evaluate the benefits and costs of outcome accountability systems. Implementing an outcome-based system at the county-level will require a rethinking of many current practices and systems of operation. If moving in this direction will not cause a net gain in information access to county managers and improved effectiveness within programs, administrators might want to reconsider this approach. The six criteria for deciding to implement an outcome-based measurement system were developed through the experiences of other researchers, the insights of government agencies that have already adopted outcome-measures, and the testimony of Contra Costa County programs interviewed for this study. The six criteria are as follows: • Are sufficient resources available to ensure that service delivery will not be negatively impacted by the "costs"associated with shifting to outcome-based accountability? • Can all program goals be incorporated in an outcomes model? • Will federal and state regulations restrict movement towards outcome accountability? • If multiple interventions from multiple programs are needed to impact a client, can one program be held accountable for the end result? • Will outcome measures be useful for direct staff providers, administrators, and outside funders? • What is the value of this model to the Contra Costa Board of Supervisors and what political benefits/risks do they face as a result of implementing outcomes? (1) Are sufficient resources available to ensure that service delivery will not be negatively impacted by the "costs" associated with shifting to outcome-based accountability? Collecting outcome-based data will likely require a significant financial investment in terms of staff time, computer hardware and software, and technical training. Based on the sample of programs interviewed for this study very few programs currently have all the resources in place to begin collecting outcome data. The majority of programs felt that based on current staff levels, outcome data could not be collected without sacrificing quality service delivery. In addition, computer capabilities are not sufficient to store and manage outcome data. The majority of programs would require a capital investment of computer hardware and software. Training would also be needed to teach staff about the benefits of outcomes and how to design and use the information effectively. One program director stated, "If the result of this is that resources are going to be committed to help us develop dat-, to develop useful outcome measures, and help us figure out what is effective and what is not, then that is one thing. If what 18 comes out of this is that existing resources will be reallocated, then what will suffer is the service delivery in an already stretched system." Human Resources Government programs have faced severe cuts in the 1990's. The result of these cuts has been fewer resources for administrative services. Therefore, many programs may currently be operating with a reduced staff assigned to management, data collection, analysis, and program evaluation. Since adopting an outcome-based system would potentially require additional resources dedicated to these functions, it is possible that outcome-based measurements would be challenging to compile because the necessary staff are not on board. If programs are required to collect this data, they may do so by diverting resources away from direct service delivery. Computer Resources The cost of data collection systems will also be a pressing concern for programs moving to outcome-based measurements. The ideal system would interconnect departments by providing compatible computer and database access. However, many government programs currently operate either with incompatible or insufficient computer technology. Investing resources in hardware, software and training will be an important consideration for any department moving in this direction. Sid Gardner finds that, "Agencies will need to make hard decisions about what data they can afford and how to deal with outside agencies who may possess the key data needed to prove the agencies' impact."27 The interviews with the sample of county programs showed that currently no programs are able to access data-bases cross-programs, instead programs maintain individual databases for the particular needs of their program, or some programs even track clients on paper. Training Resources Shifting from a system of process evaluation to a system of outcome-based evaluation is as much about revisioning the work environment as about assembling the physical resources. Programs that have made the transition to outcome-based measurements have found it advantageous to provide training both on evaluation design and the philosophy behind outcome- based measurements. One Contra Costa County program found that"not all staff are at the same level in terms of their knowledge or ability to do good evaluation." Technical training and staff development may help ensure that the outcome system established is used by all staff members. 27 Gardner, Sid, "Moving Toward Outcomes: An Overview of the State of the Art and Key Lessons for agencies," October 11, 1996, pg. 5. 19 (2) Can all prop-ram goals be incorporated in an outcomes model? Two related challenges that government officials will face when implementing outcome measurements are (1) identifying the goals that they want the system to reinforce and (2) developing the measurement tools that will show progress toward those goals. Even "proponents of performance measurement acknowledge that it can be extraordinarily difficult to identify program goals and objectives, let alone develop meaningful measures of goal attainment".28 While some programs have very clear outcomes, such as a job training program, other programs will be more difficult. For example, programs that are oriented towards prevention and long- term systemic change may be forced to identify an annual measurement of their progress that does not fully reflect all that they do. As a result,the measurement will not convey the effectiveness of the program and/or it may not produce data that are useful to the program managers. Again, Gardner believes that, "If service providers don't believe that the indicators are fair, they will resist the whole process of results-based accountability.i29 A number of programs interviewed were hesitant to endorse a move towards outcome measures because they believed that the goals of their program did not fit within an outcomes framework. This was the case for some prevention programs and for programs whose clients achieve a range of acceptable outcomes. For example, Cooperative Extension receives funding from the University of California, USDA, and the county. While this program has some direct service programs that work with youth,these services are unusual. Its primary role is to identify problems and discover solutions to those problems through research and training. Since the goal of many of their programs is not impacting clients, an outcomes measure may be ineffective for determining the success of this program. Child and Adolescent Mental Health has found a way to link challenging programmatic goals with outcome measures in their program. By measuring external indicators of heath, home, school, and arrests,the department avoids the difficulties of trying to directly measure change in a youth's mental health. The program manager has found that, "The field of Mental health has gotten to trouble in the past by trying to measure mental health outcomes since it is almost impossible to measure some mental health improvements like, for example, self-esteem." (3) Will federal and state regulations restrict movement towards outcome accountability? Since the activities of government programs are mandated and heavily regulated, it may be difficult in some cases for programs to have the necessary flexibility to truly impact their clients. Many government programs, particularly at the local level, receive funding from • multiple sources. Each of these funders may have a separate reporting and data collection 28 Kravchuk, cited Dilulio, Garvey, & Kettl, 1993, Gore, 1993, pg. 349. 29 Gardner, pg. 6. 20 mechanism that is imposed on the program. In addition, these funders may impose strict _ regulations on program operations. For example, funders can restrict the amount of funds that go into administration and set limits on the types and quantities of activities a program can provide. Local government is also restricted by how it spends its funds. "In most governments the size of the discretionary budget is quite small; 90% of the budget or more is allocated before the budget- making process even begins because of legal obligations (e.g., debt service), legislated entitlements, and tradition."" Essential to the success of outcome-based measurement systems is sufficient flexibility so that program managers can have the ability to shift resources as they see fit to improve the results of their work. Every program interviewed collects some type of data on their clients. The extent of the data collection and the type of evaluation system is largely determined by the requirements of a particular funding source. Some programs are starting to feel pressure from funding sources to develop outcome-based measures. However, other programs receive funding for a limited interventions and impact on clients is not observed. If the county initiates an outcome-based system it may run into conflicts with other funding sources like the federal and state government. County programs may be unable to comply with the county's request for outcome data because a larger funding source does not require or permit that activity. Some funders are facilitating the movement towards outcome-based measures. For example, Child and Adolescent Mental Health is under a state-wide mandate to begin collecting outcome-based measurements by January 1998. On the other hand, the program director of the Child Development Centers finds that"our funding does not require nor provide resources for data collection along the lines you have described. We have an 8% CAP on indirect administrative expenditures on each contract we receive from the state. We simply could not justify the use of contract funds to collect data that is not mandated by the contract and would further erode our ability to provide for the eleven hundred children under our care." (4) If multiple interventions from multiple programs are needed to impact a client, can one program be held accountable for the end result? The goal of outcome measurements is to make programs accountable for results, but individual measurements for individual programs may not be possible when so many programs are reliant on each other to accomplish the same outcome. Sid Gardner asks program managers to think about the question, "who do we need to succeed?"31 He asks this question because as programs identify the outcomes that they are able to impact, they may realize that it is impossible for them to meet that goal without the cooperation of several other players. For example, a 3o Allen, John R., "The Uses of Performance Measurement in Government," Government Finance Review, Vol. 12,No. 4, pg. 11. 31 Gardner, pg. 6. 21 program concerned with increasing a child's academic achievement might need the assistance of the schools, the child's family, and perhaps social services, to really make a substantial improvement. Robert Kravchuk finds that, "The multiplicity of federal programs itself adds a further layer of difficulty. For the design and development of performance measures is challenging enough where the evaluation of individual programs is the task. It becomes positively daunting when the measurement of multiprogram/multigoal systems is attempted."32 A number of programs felt that a client's success is dependent on the interaction of several agencies. In addition, since some programs are working towards long-term change, a individual program's affect may not be felt for some time. Born Free is one program that works to impact women over the long-term. The director finds that, "Sometimes a woman will not feel the impact of the program for some time, and if she recovers much later, many people will have intersected with that person, influencing her experience." (5) Will outcome measures be useful for direct staff providers, administrators, and outside funders? Many of the programs interviewed have never been asked by a funding source to show measurable outcomes. As a result, program dollars have gone to direct service delivery and have not been invested in the creation of data collection systems. Some of the programs that are not currently collecting outcomes saw potential benefits from moving in this direction. Other programs saw outcomes as costly, trendy, and irrelevant for the uses of this program. Potential usefulness of outcomes is examined below in relation to service providers, administration, and outside funders. Service Providers Since this study did not include the perspective of direct service providers, it is difficult to comment on how outcomes would directly impact line workers in Contra Costa Programs. However, based on the experiences of other programs, it is essential for outcomes to be designed with the input and involvement of staff. A successful model will be used by line workers each day to clarify the role of the agency and to focus intervention efforts. Child Health and Disability Prevention Program integrates its evaluation model within annual staff workplans for professional staff members with program component responsibility. By linking staff responsibilities with the outcome goals of the program, one director is able to both monitor staff performance and program performance: "This is my roadmap [the annual workplan] because I hold people accountable for what they say that they will do." 12 Kravchuk, pg. 350. 22 Program Administrators Program administrators varied in opinion on how outcome-based measurements would benefit their program operations. Several administrators acknowledged that shifting to outcomes would give them more data on their clients which would be beneficial in designing service components and advocating for funding and program changes. One director stated that, "As a manager you have to know what you are spending your money on and what you did with that money." Other programs felt that the systems in place were sufficient and that outcomes would simply be another piece of paperwork to complete. Programs that have already moved toward outcomes see this trend as inevitable for most programs. A program manager reflected the view of a number of programs by stating, "It is those programs that can't show any outcomes that will be cut. It would be nice if I had more statistical evidence that shows our impact. It would also help us show other groups in the community what we are doing. I wouldn't mind doing it, if I could. It would just take staff. I could probably do it with one full time clerk." Outside Funding Funders outside of the county play a pivotal role in the movement of county programs to outcomes. Many funders, such as the United Way, already require outcome data to be collected. Other funders may create barriers to collecting this information. Many programs saw funders as the impetus for moving to outcome measures. As funds for public services decline, programs are looking for outside funding. A number of programs felt that the need to show impact from services is greater when looking for private dollars; many private grant applications require programs to include measurable outcomes. Some programs felt that a county-wide outcome- based measurement system would help prepare them for competition with other agencies for private dollars. Once funders are able to collect data on their contracted services, it is then up to the funder to use that information responsibly. Gardner finds that, "For funders, closing the gap between collecting outcomes and using them to make critical budget choices is perhaps the hardest work of all.1133 (6) What is the value of this model to the Contra Costa Board of Supervisors and what political benefits/risks do they face as a result of implementing outcomes? When program managers were asked whether a good use of the Board of Supervisor's time was to push programs towards outcome measures, a number of program managers said "yes". Those programs that responded in the affirmative believe that the Board can use its position of leadership to provide a vision for the county which includes results-based accountability. The Board of Supervisors will now have to weigh the costs of establishing this system to the benefits gained. It is also vital that the Board decide how this information be used. 33 Gardner, Sid, pg. 8. 23 Until programs understand the motivation behind moving towards outcomes, they may tend to remain suspicious and cautiously guarded. Some programs expressed the fear that the Board might be looking at this new tool as a "quick fix"to all of the county's problems. These programs believe that the Board is looking for a way to clearly discern which programs in the county are"good" and which programs need to be cut. Many programs do not believe that the information contained within outcome measures will give the Board sufficient information to come to those conclusions. Some programs asked, "If the Board of Supervisors does not know anything else about a program, will the outcome measures give them enough information to make the right decision?" For example, Born Free worries that the quantifiable outcome of its program may not be seen within the larger context of existing social conditions. The program manager stated that, "We maybe have one birth a month, that would probably not justify our budget to someone who is looking only at that outcome...if I could say that January through December we had 12 drug free babies and someone thought that was great, fine, but if they are looking for numbers like 300 drug free babies...[then someone would be disappointed]." Other programs were more optimistic about outcomes and the role of the Board in designing them. The head of Child and Adolescent Mental Heath believes that the county should move all programs in the direction of outcomes. She recommends adopting the same measures that Child and Adolescent Mental Health are using: keeping kids at home, in school learning, and out of trouble. By simplifying the measures in this way, she believes that all county programs can develop program-level outcome measures that relate to these common goals. "If you look at the county's vision in terms of what it wants to do for kids, it is keeping families together, in school, out of trouble with the courts and learning. When we have had policy forum retreats, those goals have been part of conversations, those are the measures we care about. It would be great if the Board of Supervisors and the Department heads really bought off on it." 24 SECTION IV: RECOMMENDATIONS Findings The County Administrator's Office initiated this study in order to find out whether transitioning programs to a system of outcome-based measurements will improve program effectiveness and efficiency. The criteria presented in Section II provided the basis for this study's finding that a system of outcome-based measures will be beneficial to county programs, county administrators, the Board of Supervisors, and people residing in the county. However, while I recommend that Contra Costa County move ahead with incorporating outcome- accountability in child and family service programs, I suggest that implementation be incremental, and that benefits and costs of collecting outcome data are assessed regularly. The results of the interviews conducted with county programs provide a good starting point for determining the process by which outcome measures could be adopted at the program- level. Based on the 14 interviews conducted,I will advocate in this section for an incremental approach that pilots outcome-based systems in specific programs that have the most to gain from outcome measures at the least cost. Based on current uses of evaluation measures and the types of services provided, not all programs would benefit to the same degree from the implementation of an outcome-based system. Therefore, I do not believe that it would be useful for the county to impose one system for collecting outcomes on all programs in child and family services. One universal system would create data but it would not necessarily create measurement tools that improve program operations. The analysis that follows isolates a category of programs that would benefit most from an outcome-based system with the least start-up time and cost. The 14 programs included in this study provide a wide array of services. Programs pr^vide one time interventions, prevention, intensive one-on-one counseling, and long-term and short-term direct services. Because the goals, types of services, and funding sources for each program vary drastically, so too do the evaluation methods. By examining how each program currently evaluates program effectiveness, some assumptions can be made about the resources each program would need to move to an outcomes-oriented system. This information can then be used to identify which type of programs could begin collecting outcome measures at the least cost with the greatest benefit. Programs included in this study roughly represent three distinct accountability strategies: (1) outcome-based accountability systems, (2) multiple approaches to accountability, and (3) process-based accountability systems. (1) Outcome-Based Accountability Systems Some of the programs interviewed are already using outcome-based systems. These programs include: Child and Adolescent Mental Health, Summer Youth Employment Training 25 Program, Orin Allen Rehabilitation Facility, and Section 8 Housing. In each case a funder or regulator precipitated the movement towards outcomes. In addition, the funding sources also provided the necessary start-up resources or technical guidance to help that program begin collecting outcome-based measurements. Orin Allen and Child and Adolescent Mental Health are currently in the process of defining their data collection and evaluation methods. The Summer Youth Employment Training Program and Section 8 Housing have incorporated outcome measures in their service components for some time. (2) Multiple Approaches to Accountability The majority of the programs studied use a variety of accountability systems. Some of these programs have used outcome-based systems. Regardless of the measure used, these program believe that in general they have systems in place that monitor program effectiveness. Cooperative Extension is one example of a program in this category. Cooperative Extension offers a wide range of programs, many of which are not designed to produce measurable changes. However, when funding is linked to evaluation or when multi-year funding is received for a program, evaluation components are always included. Some of these components include pre- and post-testing, observing behavior changes, or national evaluations of one of their locally- run programs. CHDP is also in this category. Extensive data collection systems are in place that monitor the treatment and diagnosis of health problems for youth in their programs. Programs like the Prevention Programs, Child Development Centers, Foster Youth Services, Born Free, Independent Living Skills, and Electronic Monitoring each gather some data on their participants but staff would agree that, with more resources and technical assistance, more data could be gathered. Some of these programs already collect outcome data, but many also employ informal evaluation mechanisms like: success stories, qualitative data, and studies done by other researchers. (3) Process-Based Evaluation The programs with the least observable outcome-based evaluation systems are also the programs most heavily regulated. Both Court-Ordered Residential Placements and Emergency Response find that working within very specific legal guidelines gives them little flexibility over the services provided. In addition, since both programs provide services for a specified period of time, it is difficult for them to gather long-term data on the results of their work. Benefits to Sample Programs to Collect and Use Outcomes Another way to categorize the programs studied is to examine the resources needed and the benefits each program would gain as a result of moving to a system of outcome-based measurements. Chart 4 gives a rough approximation of the status of 14 Contra Costa programs in terms of their ability to collect and use outcome measures. 26 The first column, "resources", rates on a scale of 1-5, the level of resources in place to begin collecting outcome data. The resources considered are sufficient staff, data collection tools (e.g., client surveys, intake forms) and computer capabilities, both in terms of hardware and software. A"5" would indicate that all resources are in place. The second column estimates the implementation challenges this program would face beyond physical resources. For example, some programs would face significant implementation challenges because of internal resistance, inexperience in designing data measurement tools, and/or hard to measure outcomes (e.g., prevention). Again, a"5" indicates that these hurdles have been overcome and the programs are already collecting outcomes or could tomorrow, and a"1" presumes that outcome measurements would be extremely difficult to implement. The final column examines the benefits a program would receive by transitioning to a system of outcome-based measurements. This column assumes that not all programs will receive the same quality of information. If a program would be more effective as a result of outcome measurements it is ranked a"5" and if no benefits would be gained, it is labeled a"1". This number is based on the program director's account of how outcomes would improve program effectiveness and my interpretation of the benefits the program could receive. 27 Chart 4: Feasibility of 14 County Programs to use Outcome-based Measurements Y Program Resources Ease to Benefit Implement Gained Born Free 2 2 4 Child& Adolescent Mental Health 5 5 5 Child Development Centers 2 3 3 CHDP 4 4 4 Court-Ordered Residential Placements 3 1 4 Electronic Monitoring/Home Supervision 2 4 4 Emergency Response 2 1 4 Foster Youth Services/Healthy Start 3 3 4 Independent Living Skills 3 3 4 Orin AllenYouth Rehabilitation Facility 4 4 4 Prevention Programs 3 3 3 Section 8 4 4 4 SYETP 5 5 5 Youth Development 5 3 2 Resources: 5=Already in place,4=Almost all in place,3=Some but not all,2=Very little, 1=No resources. Ease: 5=Already doing,4=Could do with little start-up,3=Would take some time,2=Some parts would be very hard, 1=All aspects difficult to implement. Benefits: 5=Outcomes improve effectiveness,4=Outcomes Improve Program,3=Could have positive benefits, 2=13enefits might not outweigh costs, 1=Not beneficial. Category of Programs to be Targeted for Possible Pilot Outcomes Study By categorizing the programs studied in these two ways, by current evaluation systems and by ability to use and benefits to be gained by moving to outcome-based measurements, a group of programs emerge that would benefit most from this transition. The programs that already collect outcome data and score high marks on Chart 4 (total of 11 or more) do not warrant the county's immediate attention. These programs have established systems that largely meet the needs of their funding source and internal staff members. When the county develops an TM outcome-based model, it would be ideal if the measures proposed are compatible with the tools and data already used within these programs. The category of programs that collect only process data or even little to no data would require a huge investment to begin collecting outcome data. Since systems are not currently in place to track client progress in the near or long-term, data collection mechanisms would need to 2s be completely redesigned at great time and expense. While these programs could eventually r benefit from an outcome-based approach, it would be best to work with these programs once the implementation challenges have been resolved. The group of programs that would be best to test an outcomes-based model are the programs that now employ a variety of evaluation techniques. These programs are generally also interested in collecting more data on their clients and see a direct benefit from doing so. Each program would require an investment of computer technology, staff time, and training. However, with sufficient resources and training,this category of programs would be able to move to a system of outcome-based measurements with relative ease and in so doing these programs could gain feedback on program performance, could better understand the impact of the services provided on clients in the program, and would have quantifiable data to substantiate the benefits of this program and the need for it to receive continued funding. Recommendations The framework used above for identifying programs that would benefit most from outcome-based measurements could be conducted on a county-wide scale within child and family service programs in Contra Costa County. Once the category of child and family programs that would benefit most from outcome measures is identified, I recommend that the Board of Supervisors and the County Administrator's Office initiate several pilot projects, within this group of programs,that establishes a data collection system for outcome measurements. Selecting the appropriate programs for piloting outcome measures is one part of the set of program-level recommendations that are detailed below. As the county proceeds with the implementation of outcome measures, alternative approaches to implementation may be raised. While all approaches should be considered, I would argue against an approach that imposes an over-arching system for collecting outcome measures on all programs that is not compatible with what is already in place within programs and without considering the usefulness of collecting this data to program staff and the County Administration. Since this paper is primarily concerned with how outcomes are implemented at the program-level, the process for developing community-wide outcomes will not be described in great detail. This should not, however, diminish its importance. In fact, the county has already begun that process with the development of a child and family report card. Creation of community-wide outcome measures and program-level measures can occur simultaneously. Based on states that have already developed community-wide outcome measures the county could generally follow the following process: • Identify county-wide-goals that all child and `amily service programs impact, • Hold community-wide forums to present these goals and find out if these are the goals that the community cares about, 29 • Hold meetings with department heads - get every program to agree that what they provide feeds into these goals, • Pick county-wide indicators of progress and set benchmarks for where you want the county should be in relation to those indicators in the future. Implementation Strategy for Program-Level Outcome Measures I recommend that the County Administration consider the following strategy for implementing outcomes within child and family programs: • Identify pilot programs to implement outcome measures. There are a number of methods that the county could use to identify pilot programs. One method would be to survey all county programs in child and family services, using the questions in the Methodology section of this paper to guide the survey creation. The survey would identify programs that, if given human and financial support, could gather outcome information and would use that information to improve program performance. The goal of piloting this project to several programs is to receive feedback on both the utility of outcomes and the resources required for successful implementation. For this reason, it would probably be best to start with programs that can be begin collecting outcomes in a relatively short period of time. Another method would be to conduct a recruitment process. The County Administrator's Office could hold a series of informational recruiting sessions where any programs interested in piloting outcomes accountability could learn more ahout outcomes and about the resources that would be made available to them should they choose to participate in the pilot project. The next step would then be to ask programs to volunteer for the pilot. The advantages of this method would be (1)that it would not involve a resource intensive survey and (2) that all programs participating in the pilot would be involved due to their own choice. Of those programs that volunteer, those that can derive the greatest benefit from outcomes at the least cost should be selected for the initial pilot. • Help programs identify indicators of performance within their programs. A team of trained professionals, perhaps from outside the county, should be assembled that can provide technical expertise in designing good, working measurements. Until outcome measures are relevant to the staff that collect the data, they will be seen as simply paperwork instead of as a tool for improving service delivery. Once this team has assisted the program in data design, they should determine resource needs. A number of programs said that they could begin collecting outcomes with more staff resources and minimal computer hardware. Perhaps one roving data collection person could be assigned to the several programs engaged in the pilot process. 30 • Assemble the data in a centralized location. As data becomes available it will need to be complied in a useable form. A computer linkage would be preferable so that time is not spent requesting data and waiting for the transfer of necessary documents. • Determine end use by the Board of Supervisors. When the data is delivered to the Board of Supervisors, Board members will need to learn how to interpret and use the information. Again, by piloting this process, Board members will also have time to learn about the complexities of analyzing outcome data. If Board members want to use this information to allocate and reallocate funding, a process will need to be established and a determination made as to whether this data gives enough information to do so. • Use the data from indicators to see if the needs in the community are being addressed to improve program effectiveness and efficiency. If county-wide indicators are developed, the information gathered from programs can be used to see if the priorities of the community are being addressed through existing county services. (Please note, however, that county services may never meet the service needs of the community due to either insufficient county resources or to the existence of more appropriate vehicles for meeting community needs than county agencies.) Many states have released outcome data to the public so that taxpayers can see the impact of their dollars. Depending on what the Board decides is the end use of outcome data,the information can be used to inform the public, to help programs become more effective,to determine unmet needs,to eliminate ineffective programs, and to make difficult funding determinations. 31 BIBLIOGRAPHY A Focus on New Outcomes: Families.Neighborhoods and Community Assets. Ad Hoc Committee on Systems Redesign, Discussion Paper, June 28, 1995, prepared by the Northern California Community Services Council, Inc., San Francisco. Allen, John R., "The Uses of Performance Measurement in Government," Government Finance Review, August 1996, Vol. 12,No. 4,pg. 11. Beigel, Astrid, "Competition and Cooperation: Public Sector Coalitions for Outcomes Measurement," Behavioral Healthcare Tomorrow, February 1997, pg. 81. "Benchmarking Results for Children and Families,"The National Governors' Association, May 31, 1995, Issue Brief. Bruner, Charles, Crawford, Mike, Elias, Victor, Perlowski, Karon, Way to Grow: Implementing an Outcome-Based Approach to Evaluating a Family Support Program,November 1995, Child and Family Policy Center Occasional Paper Series, #17. Campbell, Michael D, "Outcome and Performance Measurement Systems: An Overview," Alliance for Redesigning Government,National Academy of Public Administration, http://clearlake.ibm.com/Alliance/clusters/op/overview.6.html. "Community Services and Supports to Improve Outcomes for Children,"The Harvard Project on Effective Services, The Center for the Study of Social Policy, The National Education Center for Education and the Economy, June 1993. "Federal, State and Local Case Studies,"http://www.aspanet.org/research/taskfrce/lcicase.html. Friedman, Mark, "A Strategy Mai, for Results-Based Budgt`ing: Moving From Theory to Practice,"The Finance Project, September 1996, Washington ')C. Gardner, Sid, "Moving toward Outcomes: An Overview of the State of the Art and Key Lessons for Agencies,"prepared for the Hawaii Community Services Council, October 11, 1996. Garrett, Michael R., MacDonald, Todd, "Program/Activity-Based Management at the Regional Municipality of Peel: An Organization in Transition," Government Finance Officers Association of the US and Canada, August 1996, Vol. 12,No. 4, pg. 7. Gore, Al "Reaching Public Goals: Managing Government for Results,"National Performance Review, October, 1996, Washington D.C. Hatry, Harry P., Kirlin, John J., "An Assessment of the Oregon Benchmarks: A Report to the 32 Oregon Progress Board," The Urban Institute and The Sacramento Public Affairs Center, June 1994. Hatry, Harry P., Report to the Oregon Progress Board: Findings and Recommendations Oregon Benchmarks and Associated Performance Measurement Process, The Urban Institute, May 27, 1994., Washington D.C. Kravchuk, Robert S., Schack, Ronald W., "Designing Effective Performance-Measurement Systems under the Government and Performance and Results Act of 1993," Public Administration Review, Vol. 56,No. 4, pg. 348-358. "Managing Results: Initiatives in Select American Cities," August 1995, http://nprt.gov/cgi- bin/print_h..ary/fedstat/24e6.html?measurements. Melaville, Atelia I., Blank, Martin J., What it Takes: Structuring Interagency Partnerships to Connect Children and Families with Comprehensive Services, January 1991, A joint publication of the Education and Human Services Consortium. Mercer, John, "The Performance Management and Budget System of Sunnyvale, California," Counsel to the U.S. Senate Committee on Government Affairs, 1994, prepared for the Organization for Economic Cooperation& Development, Paris. McCroskey, Jacqueline, "Outcome Measurement for Family and Children's Services in Los Angeles County," drafted for the Los Angeles County Children's Planning Council, January 26, 1993. "Milestones Update Shows State Making Progress,"Minnesota Planning News Release, http://www.mnplan/state.nm.us/press/mile 1996.html. Minnesota Legislative Budget Committee Report, July 20, 1995. "Moving Toward Accountability for Results: A Look at Ten State's Efforts," Summer 1995, http://www.ccsso.org/ibsum952.html, pg. 6. 1996-97 Children and Family Services Budget, County of Contra Costa Oregon Benchmarks: Standards for Measuring Statewide Progress and Institutional Performance, ` Report to the 1995 Legislature, Oregon Progress Board, December 1994. Oregon Option, Executive Summary, Agenda Item#5. Osborne, David, and Gaebler, Ted, (1992), Reinventing Government: How the entrepreneurial spirit is transforming the public sector, Addison-Wesley Publishing Co., Inc. 33 "Performance Management: Perspectives for Today's Public-Sector Manager", Special Supplement to the January 1997 Public Administration Times, http://www.aspanet.org/research/taskfrce/perf html. Performance Measures Reference Guide, State of Oregon, Department of Administrative Services, 1994. "Reinventing Catawba County,"http://www.co.catawba.nc.us/misc/reinvent.html. Schorr, Lisbeth, with Frank Farrow, David Hornbeck, and Sara Watson, "The Case for Shifting to Results-Based Accountability,"In Making A Difference, (pp. 13-28),National Center for Service Integration. Summary Minnesota Milestones, 1996 Progress Report, Minnesota Planning Department, July 1996. Traglia, James J., Massinga, Ruth, Pecora, Peter, Paddock, Glen B., "Implementing an Outcome- Oriented Approach to Case Planning and Service Improvement," Chapter 4 of Quality Improvement and Evaluation in Child and Family Services: Leadership for Tomorrow(tentative title), Child Welfare League of America for the Committee on Quality Improvement and Evaluation of the CWLA National Council on Research in Child Welfare, authors are with the Casey Family Program. Understanding Outcomes: A Handbook for Understanding, Developing, and Evaluating Outcomes, prepared for the United Way of the Bay Area, December 1995,Northern California Community Services Council, Inc. "Utilizing Performance Measures to Achieve Organizational Goals," Workshop Materials by KPMG Peat Marwick, Presentation was developed and prepared by Marc Diamond, Carlo Grifone, and Peter Lange. Weidner, Marv, "The Council on Human Investment: Performance Governance in Iowa: ASPA's Task Force on Government Accomplishment and Accountability, Case Studies," Des Moines, Iowa. Young, Nancy, Gardner, Sid, Coley, Soraya, Schorr, Lisbeth, Bruner, Charles, Making a Difference: Moving to Outcome-Based Accountability for Comprehensive Service Reforms, National Center for Service Integration, Resource Brief#7, 1994. 34 APPENDIX I: INTERVIEWS COUNTY PROGRAMS DATE Carl Ackerly March 18, 1997, 9:00 a.m. Jennifer Balogh April 10, 1997, 10:00 a.m. Rich Clarke April 11, 1997, 9:00 a.m. Joe Davis March 14, 1997, 10:00 a.m. Wilda Davisson April 22, 1997, 2:30 p.m. Dave Grossi March 13, 1997, 9:00 a.m. Timothy Hamp April 3, 1997, 11:30 a.m. Bob Isom April 10, 1997, 11:30 a.m. Susan Laughlin March 21, 1997, 1:30 p.m. Shelley Murdock March 21, 1997, 1:30 p.m. Ruth Ormsby March 21, 1997, 4:00 p.m. Meda Reed March 20, 1997, 11:00 a.m. Grace Schmidt March 20, 1997, 9:00 a.m. Kate Teuton March 13, 1997, 1:30 p.m. Bob Whatford March 18, 1997, 11:00 a.m. MULTNOMAH COUNTY Jim Carlson May 5, 1997, 1:00 p.m. 35 APPENDIX II Contra Costa County Program Case Studies Each of the following 14 cases studies are written in the voice of the Program Manager. At times the Program Manager may have conveyed information that other staff members, who were not present at our meeting, expressed in the past. The reader should understand that these case studies may represent a range of opinions and not necessary the sole opinion of the person interviewed. The interviews were taped, transcribed, and summarized below. Some direct quotes have been added to give the reader a greater sense of the perspective this interviewee brings to the study. The Contra Costa County Administrator's Office thanks each of these programs for participating. INDEX Department Program Pase Community Services Department Child Development Division 36 Cooperative Extension Youth Development Programs 39 Department of Health Services Born Free 42 Child and Adolescent Mental Health 44 Child Health and Disability Prevention Program 46 Prevention Programs 48 Department of Probation Court-Ordered Residential Placements 50 Electronic Monitoring and Home Supervision 52 Orin Allen Rehabilitation Facility 55 Department of Social Services Emergency Response 57 Independent Living Skills Program 59 Housing Authority Section Assisted Housing and Public Housing 62 Mt. Diablo Unified School District Foster Youth Services and Healthy Start 64 Private Industry Council Summer Youth Employment Training Program 67 36 1. Community Services Department, Child Development Services: The mandate from the funding source is to provide quality child care and development services. To perform this mandate, 11 centers are open in Contra Costa County with general child care operating 11 hours a day, 250 days a year. 37 classrooms are financed by a budget of $5.5 million. Another model is state preschool, this operates 9 months a year for three hour sessions. Families must meet State income and need standards to qualify for general child care services and income guidelines for State preschool. Children coming from Child Protective Services are considered the most in need, and other need-based reasons for child care include, working parents and parents with disabilities. 95%of parents using the centers are either working poor or currently in school. 76% of the parents in the general child care program are working poor. In addition to the general child care program is a state preschool program in operation at some sites. This program provides preschool for low-income children so that these children can start kindergarten and be successful. A variety of collaboratives are in place to assist the delivery of service at the child development sites. The Community Colleges are one of the major collaborators. Another collaborative is with the Department of Social Services, Housing Authority, County Office of Education, and the West County Unified School District. Together these organizations have developed a training program for GAIN/Head Start eligible parents that provides six units of child development training for these parents can gain entry-level jobs as teacher aides. Other parents gain skills by volunteering in the classrooms. The Child Development Centers also collaborate with Head Start, the English Action Center, Service Integration Teams, Contra Costa College, and San Francisco State. Programs designed through these collaborations range from violence prevention programs to a community garden project. Staff within the community services department are active politically on the issue of child care and development at the County and State level. A State bill is being carried this session that, if passed, would make it easier for child development centers to purchase their own buildings. This effort is part of a larger commitment to building the necessary infrastructure so that the State and County will be able to provide child care to low-income families even after the increase in demand as a result of welfare reform. Goal: To provide quality child care and development services to low-income families. Current Evaluation Mechanisms: An internal quality audit is conducted each year. In addition data is collected on how many days of service are provided in a year, how many preschool hours were given, and the number of children served. "But as to whether or not it is doing the children • or the parents any good, we have not quantified that, and we should. We have told the Department of Education that this is a real weakness in the program-that we do not have any statistical data to back up what it is that we are doing." Test scores in the West County were published a few years ago and the four schools served by the centers scored higher than similar 37 populations in other areas of the county. While this type of information is not systematically collected it does give the centers an informal evaluation of the quality and impact of the services provided. Data/Resources: "If you really wanted to collect good data on the programs a study would have to be done that compared children who receive the services with children who do not. A study of this nature would need to be done over a few years and it would be time and resource intense. "We do not have the money to do that." To collect this kind of data the program would need a full-time researcher. "If the County wants the data, they should be willing to pay for it. We would need a computer, computer software, and a clerk to input the data. That would cost about $10,000-15,000." The data would be easier to collect on the impact parents but it would still require a tool to gather and manage the data. To collect data on the benefits to children a staff person would be needed with a high level of sophistication, who understands the complicated issues around educational research. It would also require legal permission from parents to be able to gather information on children. The program is not currently linked to outside databases that could readily provide some of this information. In fact, Department of Social Services cannot give the Community Development Department access to information about which clients receive aide linked to child care payments. Families are recertified for child care at least once a year and information is collected at that time. Parents in the training program provide data each semester on their progress towards school completion. Employment and wage data are collected. More information could be collected but a system would need to be established. "We told the Department of Education that they should be setting this up. We are busy enough doing the work that we are required to do, instead of doing work that we are not paid for." Right now the program manager is satisfied with the amount of data collected as it meets contract requirements. Any additional information collected would be used for external requirements. If the funding is provided then the information will be collected, "but don't take the money away from our teaching staff- some of our teacher assistants make only $6.00 an hour!" Outcomes: The State Department of Education is looking at providing agencies with some sort of fiscal support to begin collecting outcome data. The type of outcome data that could come from long-term funded studies does not exist right now. "Our funding does not require nor provide resources for data collection along the lines you have described. We have an 8% CAP on indirect administrative expenditures on each contract we receive from the state. We simply could not justify the use of contract funds to collect data that is not mandated by the contract and would further erode our ability to provide for the eleven hundred children under our care." "We do not collect data on how many successes that we have,just like we do not collect information on how many failures that we have; not because it couldn't be done, but because it has never been required by the funding source." The program manager does not feel that 38 collecting this data would change program operations, mostly because no flexibility goes into program design. "I do not have any flexibility for redesigning the program. We have set operating parameters if we want to collect this money from the state." However,this information would be useful as a tool for advocating for more money and policy changes. "If outcomes are set up fairly they should show that preventative programs work." Program operations are heavily regulated and collecting more data would not change the way the centers are run. Some process data are collected but outcome data are not. Staff, computer and technical assistance resources would be needed to begin collecting outcome data. 39 2. Cooperative Extension,Youth Development Services: Cooperative extension is structured so that it gets funding from UC, USDA, and the County (which gives some administrative support). It is the smallest county department. As part of the University,this program is not typically in the business of direct service delivery. While it has some direct service programs that serve youth, like the after-school program, 4H clubs, and the nutrition program,these are unusual. Its primary role is to identify problems and discover solutions to those problems through research and training. It also provides workshops for people who work with children. Some current programs include: after-school classes, 4H clubs, nutrition programs, and educational workshops. Goal: Help youth grow up to be responsible and productive adult citizens. Current Evaluation Mechanisms: Since the programs Cooperative Extension provides are diverse, so too, are the evaluation mechanisms. Identifying evaluation measures is not new to this program, it is always working to define better evaluation components. Since it is part of the University, Cooperative Extension staff receive training in evaluation methodologies. One thing that staff have learned is that it is not always feasible to do extensive evaluation. Some of Cooperative Extension's programs are short-term and not repeated. Other programs are national models, like 4-H which have been well evaluated by other groups over the years. "If the literature already shows that this program design is effective, we most probably will not evaluate again unless we are typing to ascertain if it is appropriate for a new audience, etc..." Cooperative Extension's direct federal programs have the most rigid outcome measures and data requirements. The federal nutritional programs have been running for a number of years and as result they are stable and well evaluated. Evaluation has been a struggle in nutrition because while program participants can show a knowledge change through pre- and post-testing, it is harder to show a change in behavior. Cooperative Extension is making a transition to gathering information on behavior changes because a knowledge change may not make a lot of difference in program participants' health and well-being. Workshop work is more difficult to evaluate. Cooperative Extension gathers feedback from the participants asking them what they learned as a result of the training. Sometimes program staff are able to follow up to see if participants use the techniques that they were taught. In the after-school program it has been suggested that program staff look for academic impacts. The number of assignments turned in or improvements in attendance could be measured. The difficulty is some young people might be coming to the programs because they want something to do after school not because they really want to do their homework. In these cases a different outcome might be seen, such as those kids staying out of trouble. Cooperative Extension staff emphasize that huge leaps in academic improvement are not always the goal of the programs. "It is hard for us to get long-term data. We are always looking for better ways to evaluate. Since we deliver many workshops that are "one shot", I wish that there was a way to measure the 40 impact of those limited duration programs. We are also delivering television programs and we do not even know who is watching. We could collect how many people view the program but what are they doing with the information,what is the impact?" Data/Resources: Data is collected for each program based on the outcomes that Cooperative Extension hopes to measure and the requirements of the funding source. Limited duration programs receive less data collection. Some difficulties of data collection are: need for human subject protocol, restricted access to school records,the transient lifestyle of some clientele. "In a year we probably spend about 25% of our time on data analysis, including needs assessment and goal definition. This is greater for pilot programs." Outcomes: Cooperative Extension started moving towards outcome evaluation 15 years ago, but it took 10 years of the administration pushing to see an new direction in evaluation. "I am very humbled by how long it took to get this. I think the Board should know this. When a knowledgeable higher authority (Federal and Local administration) gives a mandate with support to go along with it, it moves people. If a non-supportive government entity does this without funds...it will not be a great source of motivation." Outcomes are identified by project and based on the clientele that a program is serving. Need assessments are always conducted before Cooperative Extension begins a program. If outcome measures had to be provided for all Cooperative Extension programs, it would change what types of programs are offered. "The reason that we can offer a variety of programs is because we are selective about when we evaluate. It takes us 25%of the time now. If we had to evaluate all of our programs,then it would take 50%of our time and we simply could not offer all that we do. Some of the outcome measures that would truly prove our effectiveness would be so difficult to get that it would not be worth the time to offer the program to begin with." Some of Cooperative Extension's programs are intentionally geared for longer-term change. Cooperative Extension does not plan on seeing an immediate impact. Also, other pLograms/persons interact with the participant between when services are delivered at Coe Perative Extension and when the impact is felt. It is hard to show cause and effect. Outcomes can be a good thing as long as they do not create a bigger bureaucracy by applying outcomes to everything. "Maybe outcomes will help politicians realize that change takes a long time and that when they look at funding certain programs, they will recognize that a lot of things interact to produce that outcome." Outcomes could force programs to shrink expectations about the kinds of outcomes produced. "Let's hope that on the front end people realize that these smaller outcomes are alright." For example, after-school programs may have many indirect benefits that are not easily recognized as beneficial. End use of outcome data is extremely important to consider. It will also be difficult for the Board of Supervisors to interpret the complex set of answers that they receive through outcome measures. "We did not elect politicians because they know about program evaluation." There is 41 no one way for outcome measures to be packaged that explains all that this program does. Over the years program staff have provided impact information to funders knowing that the information did not truly reflect all that goes on. This program already does extensive evaluation. Cooperative Extension finds that it is not always advantageous or feasible to do outcome measurements. Often the program goals are geared towards long-term progress and small incremental changes. Adding outcome measurements to Cooperative Extension programs has required a great deal of institutional support and time. 42 3. Department of Health Services,Born Free Services: Born Free provides outpatient treatment for pregnant or parenting women who have problems due to alcohol and other drug use. Services include: recovery groups, parenting, child care, individual sessions for women, massage, smoking cessation, intervention(non-Medi-Cal billed), brief substance abuse prevention with pregnant and post-partum women (15 to 30 minutes), counseling, assessment of a women's level of addiction, education, and other offers of help. Born Free has 120-130 participants at any given time. The program has been recognized nationally since its conception in 1988. Goal: Born Free programs focus on meeting five goals: (1)to provide a safe environment for women to be able to make healthy choices for themselves and their families, (2)to promote a recovery lifestyle, (3)to provide a learning opportunity for students in the maternal and child health field, (4)to educate women in recovery about health issues, and (5) to ensure quality of care to women in recovery at Born Free. Current Evaluation Mechanisms: Each of the goals identified above are linked to objectives, methodology, person or persons responsible, time lines, and evaluation measures. Each of these goals and accompanying measures are related to process. Managed care is creating a new evolution in health care and Born Free has been impacted by this trend. However, in the past Born Free has talked to funders about the potential cost savings of this program. To demonstrate cost savings, Born Free has relied on research conducted by other organizations. Internally, Born Free can provide qualitative data by using specific client success stories. (Note: the Community Substance Abuse Services Division is in the process of developing quantifiable measures.) Data/Resources: "We have all the information we need to know if we are successful but we do not have the staff time, resources or computer database systems to analyze that information." Born Free collects monthly data and gives that information to the County Health Services Administration. Numbers on intakes, discharges, women currently active in their program, number of clients that are pregnant, number that have children, how many group counseling sessions were conducted, number of individual counseling sessions, and toxic screens of women who have given birth(either self-reported or taken from the hospital (harder if not the local hospital). Since program staff do not analyze outcome data themselves, they utilize peer- reviewed research studies that say which interventions work best. Born Free staff complete the "California Alcohol and Drug Data System" (CADDS) on each client. It is easy to request aggregate CADDS data from the state but it takes a couple weeks to request and receive. This data would provide program staff with average length of stay, number who graduated from the program, type of services received, and problem that the client was seen for(alcoholism or drug addiction). Besides this data program staff do make some follow-up calls to women and those contacts are kept with the individual charts. However, making these calls on a regular basis takes a lot of time (Born Free calls every woman who has been a client) because hale of the phone numbers are generally wrong. "I think I have all the information to show that we are successful but I do not have the ability, in terms of time and capacity to do the analysis. I 43 could collect more data if necessary, it is just a degree of staff time." Outcomes: Born Free would benefit internally from greater analysis of existing data. "I see the importance, I know that I have information that would be helpful, but if I am made to do more work without the clerical staff, without the support,then it becomes, I do not care how much money you take away from me, because I can't do what you want." Born Free is short clerical staff and does not have oversight over the budget. The director has never seen an itemized budget breakdown and does not have budget oversight. Born Free finds that for pregnant women who use drugs, and for women with children, a range of successes are possible. "How we define success is important when we need to prove the effectiveness of our program." For example if a heroin user ends up smoking pot after the program, this could be considered a success. Also if a woman who has traded sex for drugs reduces that behavior, or now uses a condom, this too could be considered somewhat of a success. A range of successful outcomes is needed: from complete abstinence and recovery skills to decreased use; from giving birth to a drug-free infant to being sober, reunifying with children and getting a job. Even though most of the work in the program is very individualized, administrators and funders want to see data aggregated. Currently Born Free does not have aggregated outcome data,just anecdotal success stories. The new MIS computer program which does Medi-Cal billing is the first push for results data. However, Born Free has already been collecting information on the number of drug-free babies born. The director at Born Free believes that it would require more than a number to explain what this data means. "We maybe have one birth a month, that would probably not justify our budget to someone who is looking only at that outcome. It is providing that rich picture of what we do, from the process point, as well as from the outcomes perspective." The program is designed to impact women for the long haul. Sometimes a woman will not feel the impact of the program for some time. And "if she recovers much later, many people will have intersected with that person, influencing her experience." If outcomes are to be adopted money should be connected to the process. The sources requesting the information would also ideally understand what the information means. "Outcomes are good and if I could say that January through December we had 12 drug free babies and someone thought that was great, fine, but if they are looking for numbers like 300 drug free babies..."then they (someone) would be disappointed. Born Free uses cost-benefit studies performed by other health educators/researchers to justify the benefits of the program. Would like to collect more data, but would need more staff. Confident about the strengths of Born Free but fear that outcomes would not give a clear picture of everything that goes on within the program. 44 4. Department of Health Services, Child and Adolescent Mental Health Services: Case management is provided to children and adolescents to assess and treat mental health conditions. Goal: The goal for clients in child and adolescent mental health is to impact young people's mental health by keeping them at home, in school learning, and out of trouble. Current Evaluation Mechanisms: This division just received a grant that will allow it to do extensive program evaluation. Prior to this grant Child and Adolescent Mental Health did not measure the effectiveness of its programs based on the impact it had on the clients served. However, even if this grant had not been funded, Child and Adolescent Mental Health was still under a state-wide mandate to begin collecting outcome-based measurements by January 1998. Now, there are a series of tests that all youths entering the program take at intake, six months, and at a year. The results will come back to the therapists and then the data received will be compiled for system-wide use. The division has begun collecting data on each of these measures: keeping kids at home, in school, and out-of-trouble. Since Child and Adolescent Mental Health just started collecting these numbers, and the computers are not fully up and running, the impact of the new system is not yet known. One specific measure on which the program will be evaluated is reducing the number of kids in out-of-home placements. In order to track this data Child and Adolescent Mental Health is using national standardized methods which will be used by other mental health programs state-wide. The evaluation component was necessary since the contract with the state is performance-based. If the numbers do not go down in the areas that Child and Adolescent Mental Health has elected to measure, it will not be refunded. "Prior to this grant we didn't have performance outcome measures. Now we will test every kid and family coming into the system. These numbers will be reflective of all that we do in our programs. It will hopefully show that we are making a difference." By measuring external indicators of home, school, and arrests, the department avoids the difficulties of trying to directly measure change in the area of mental health. Mental health has gotten to trouble in the past by trying to measure mental health outcomes. For example, if you decide that the outcome of the program is increased self-esteem, how do you know when someone's self-esteem is higher? The field of mental health has the advantage of extensive research and analysis in the area of evaluation. California and UCSF Medical Center have been pioneering these evaluation techniques for the last ten years. Data/Resources: The grant provided sufficient resources to hire an evaluator and pull together the necessary computer technology. So far the staff have felt '.'_iat they are spending more time filling out more forms without receiving any direct benefits. Once the computer system is on line and staff members have more information on their clients, it is likely that paperwork will not 45 be seen as such a burden. Once the data can be aggregated,the department will use this information to see where the programs are and are not effective and then change programs when necessary. This is something that has always been incorporated in the program but it has never been done through formal mechanisms. The advent of managed care requires good feedback in order to stay competitive. Training was provided for 225 staff on the outcome measures. Also, a conference was held for over 200 people including staff from the Departments of Probation, Education, Social Service, and Mental Health on this system of care model. The conference addressed issues of collaboration so that the combined departments can begin working together to impact the system numbers. Outcomes: While this department is just beginning to collect outcomes, similar programs have been implemented in Ventura, San Mateo, Riverside, Santa Cruz, and others. The data collected so far shows savings in the millions of dollars by reducing out-of-home placement costs. So while this division is early in the process of collecting outcomes, it is confident that the model will work,that the necessary systems and resources are in place, and that the data collected will show that the program is effective at what it does. The head of Child and Adolescent Mental Heath believes that the County should move all programs in this direction. She recommends adopting the same measures that Child and Adolescent Mental Health are using, keeping kids at home, in school learning, and out of trouble. By simplifying the measures she believes that all County programs can develop outcome measures that relate to these common goals. In addition, by using a standardized system other programs state-wide can offer Contra Costa County advice. "If you look at the county's vision in terms of what it wants to do for kids, it is keeping families together, in school, out of trouble with the courts and learning. When we have had policy forum retreats, those goals have been part of conversations, those are the measures we care about. It would be great if the Board of Supervisors and the Department heads really bought off on it..." Training, resources, and a plan of action are needed by the Board of Supervisors. Government must move in the direction of outcomes if they want to continue to exist. As a mental health provider Child and Adolescent Mental Health feels the imminent threat of the private sector waiting to take over. "If people don't get into end results, then they won't exist. So there are some real external motivators out there." Child and Adolescent Mental Health is just beginning to collect measurable outcomes for all of its programs. Funding will be based on the programs ability to impact quantifiable changes in the number of out-of-home placements, including group homes and state hospital utilization. Resources are already in place. 46 5. Department of Health Services, Child Health and Disability Prevention Program _ Services: The Child Health and Disability Prevention Program (CHDP) is a State and Federal program aimed at providing wellness care examinations and follow-up on conditions detected for youth on Medi-Cal and at 200% of the poverty level. Youth on Medi-Cal receive services up until the age of 21 and youth at 200%of the poverty level receive services until age 18. Outreach work is done so that families know what services are available and access is ensured to a physician and dentist through a CHDP provider network. Approximately 1500 referrals come in a month through Social Services. A copy of the report of the health assessment is returned to CHDP after the client has seen the doctor. If health problems are identified that need further evaluation for treatment and/or diagnosis, follow-up must be completed within 120 days. (This time frame is a federal program requirement). Within the dental health component a provider network is maintained and the process for receiving treatment is streamlined for CHDP clients. Nutrition services and health education are also provided. The anti-tobacco program is currently being run through the health education component. CHDP also ensures that every child going into the first grade receives the school health assessment. The program is at 96%compliance with this requirement. Guidelines are given by the State and Federal government for the types of services that need to be,provided, but CHDP also conducts a needs and health data assessment which is used to develop the"scope of work plan" for the year. This plan includes outcome objectives responding to the data trends observed. Goal: The goal is to ensure access to health care for the specific population served and provide treatment and follow-up on health conditions detected. Current Evaluation Mechanisms: Every quarter the program does an internal audit to assess the system of tracking, and to see if the youth received the types of referrals that were needed. An internal assessment is also conducted to see if the services needed were provided by appropriate level personnel (e.g., nurse v. nurse practitioner). Within the scope of work plan each service component has specific objectives that must be met. For example, one objective would be to maintain a current physician and dental provider network. Each professional staff member with program component responsibility creates a work plan for the year. Process measurements are used to track the number of client appointments made, clients seen, information provided, and treatments given. "This is my roadmap because I hold people accountable for what they say that they will do." The aim of all evaluation mechanisms is to ensure that the program is run efficiently,that the same level of funding can be guaranteed for the next year, and that children get needed services. = Data/Resources: Extensive amounts of data are collected and tracked. Some of this data includes: number of clients referred for services, number that received services, number that refused services, number on AFDC, number on Medi-Cal, medical conditions detected, follow- up services provided(about 30% need follow-up), number of clients who are pregnant, number 47 of clients in foster care, number of home visits and phone calls, and health conditions discovered. "Between all of the data sources, both internal and that which comes from the state, we have enough information to know the results of our services." The data collected are used to plan services for the next year. Outcomes: In conjunction with the guidelines provided by the federal government, CHDP determines which service objectives will be accomplished each year. While many of these objectives concern process -providing access and giving treatment-the overall outcome for the program is improved health for this population of youth. One evaluation measure is: "if you have a health problem identified did you receive treatment for it?" While the program director will not say that this program directly makes children healthier, the program does target a population at greater risk for many psycho-social health conditions, and because these children get services that can help ameliorate those problems they should be healthier. "In the long run they may be healthier as a result of receiving these services. If a child had a medical condition and it was treated then that is improved health." "If you are required to develop a systematic system of evaluating your program, and it is imposed on you by your funding source, then you will comply." It is beneficial to have this information because in order to impact the population that this program is working with it is best to know what is effective and what happens as a result of services provided. "As a manager you have to know what you are spending your money on and what you did with that money." Extensive data is collected on the services provided and the health conditions of the community served. This information is used to establish work plans for staff and determine future services that should be provided. Many measures are process-based but by providing treatment and preventative care a larger outcome of improved health is achieved. 48 6. Department of Health Services, Prevention Programs _ Services: The Department of Health Services provides a variety of community-wide, health prevention programs. Some of those programs include: head injury prevention in youths, lead prevention, breast cancer prevention in uninsured women, violence prevention, tobacco and drug prevention. Goal: The goal varies depending on the intervention and scope of the program. Sometimes the goal of the prevention program is to reduce harm, change behavior, and/or increase knowledge. Current Evaluation Mechanisms: The Heath Services prevention programs have always built evaluative components into the grants received. "We certainly want to know if what we are doing works." Within each program, staff must match the level of evaluation with the level of services provided. Some programs are small and short-term and do not provide the time or resources to do big surveys that will document changes in behavior. "To do good evaluation often takes money away from the intervention." When the programs do evaluation, staff want to know what components of the program work well and who it works well for. All evaluation should be used to provide feedback to the program managers so that they can improve the functioning of the program. "People have bought in on the importance of doing evaluation in their programs." Sometimes funders create measures that are beyond the scope of the project. The problem that the program is trying to address must be defined before the measures are put in place. This sets the stage for good evaluation. The measure should be as close to the actual intervention as possible. One area of evaluation that can be hard is the development of collaborations. If the goal of a project is to bring people together to solve problems in the community, then is the formation of the collaboration a process or an outcome? Da,a/Resources: Some of the available data sources, like suspension rates and homicide rates, might not directly link with what a particular prevention program is trying to do. In that case new data collection mechanisms must be established. For example, with the bicycle head injury prevention project useable data was not available. The desired outcome of this program is reduction of head injuries but since so few occur a year this measurement would not give good information back to the program staff. So instead the data collected was an observational study of the number of times evaluators saw youths wearing helmets in the city pre- and post- prevention program. The Center for Disease Control funded this project and provided additional funds so that outcome data could be collected. The Breast Cancer project has also included resources for collecting data. The goal of the program is to screen uninsured women for breast cancer. The residential drowning program has enabled staff to collect a snapshot of information about residential pool drownings. When this 49 type of data can be collected it establishes baselines for future interventions. Some barriers exist to collecting outcomes for all programs. One is that the program cannot always get access to the data that shows impact. Second is that not all staff are at the same level in terms of their knowledge or ability to do good evaluation. The amount of resources needed would depend on how many measures are put in place,the type of measures used, and whether baselines exist to compare data to. Outcomes: Identifying outcomes in prevention programs presents some challenges. For example with a violence prevention program- is the outcome reduction in homicides or is it a change in behavior which is more likely? Some of the direct screening programs, like Lead Prevention, are easier to measure with outcomes. Program staff can test to see if the community-wide lead levels have gone down as a result of the prevention services provided. The linkage with lead is much more direct that the linkages with violence prevention. It is harder to say that a specific prevention program caused a decrease in the homicide rate. It is easier to make that case with lead since fewer variables interact causing change to occur. Currently program evaluators are trying to find out what each programs does and develop tangible objectives that can be measured for each program. Some of the projects undertaken in this department reach out to a small percentage of the population. For example, the gun education project was designed to decrease the accidental shootings of youths who come into contact with a gun in the home. Since this is only 2%of gun deaths the change as a result of this program cannot be great. "You would hope that you see this dramatic drop in injuries and deaths as a result of one of our programs, but in reality with the limited resources put in is that a realistic expectation of, for example, an hour long assembly?" Staff can come up with a measure for almost every program under way,but sometimes these measures are not what the funders hoped to see. Funders would ideally like to take responsibility for reducing the number of youths who die. However, few prevention programs can show that result directly. Funders need to be realistic. "We have an obligation to see if what we are doing is helping anyone. If we are not helping anyone we should do something different." While outcome measures provide that insight it is difficult to find one model to push all programs toward. Also it is important to have the outcomes tied to the program instead of creating outcomes that are wonderful in theory but impossible for any program to live up to in practice. It is difficult to identify what prevention programs are doing and how to measure those results. However, funders are pushing in that direction and most of the prevention programs are measuring some sort of results. Evaluation components are generally included as part of a project even though it is not always evaluating outcomes. 50 7. Department of Probation, Court Ordered Residential Placements Services: The Probation Department does not decide who to serve, the court orders that. Based on very specific findings, the welfare of a minor may require that a youth be removed from the home. In these cases, the minor has already received diversion and pre-placement services, and those interventions indicated that the minor must be removed and placed in another setting. The mandate is that minors be returned to their families at the earliest opportunity and at every six months, while the minor is in placement,the case is reviewed before the juvenile court. Progress, treatment, and participation of the family are all looked at to see what goals have been accomplished. If after 18 months the minor is still in placement, then a permanency planning hearing is held, and case managers reexamine the goal of family reunification to see if it is still possible. A placement diversion program made up of a probation officer, a counselor, and Families First(a family preservation program) was established in 1990. These probation workers take on small case loads and do some "old time probation work". By going into families' homes, spending some extensive time with families, kids, and the schools, the program has been very successful at taking kids that were identified as otherwise going into placement, and keeping them in their homes. This program also impacts the family structure in such a way that younger siblings may not end up in the system. Court-ordered Residential Placement has found that the placement diversion program not only pays for itself but also provides money for other probation services. Last budget cycle this department had 2300 minors under formal supervision, 181 minors in placement, and roughly 2500 kids booked into juvenile hall. The department gets roughly 240 new cases a month and nine supervision deputies are on staff in a county of 800,000 people. Goal: The goal is to keep the youths at home. "Unless you are talking about serious abuse in a family, outside of that situation,the worst family is better than the best placement. If you can keep the kid at home, the chances of keeping that kid from going further into that system or other systems is enhanced." Data/Resources: Monthly statistics are collected that show the status of each youths' case, where the case is going, and when how soon the next step will happen. Since Proposition 13 the program has seen a reduction in resources. If it had more resources to improve research capabilities it could do some longitudinal studies of the youths that go into placement and the constellation of services that were received. Currently the program has little ability or resources to dedicate to this type of research. To be effective, technological resources would be needed to link Court-ordered placement systems with the juvenile and adult information systems. Also additional staff resources would be needed to be able to track cases over long period of time. "If the result of this is that resources are going to be committed to help us develop data, to develop useful outcome measures, and help us figure out what is effective and what is not, then that is one thing. If what comes out of this is that existing resources will be reallocated, then what will su,fer is the service delivery in an already strztched system." 51 Current Evaluation Mechanisms: If the measurement required of the program was whether youth were returned to their homes,then program staff already collect that data. But to show the overall effects of the program would require some sort of point of comparison, a control group who does not receive services. The placement program has an ideal goal of family reunification but in some cases that is not possible because the child would be returned to the same situation that caused the problems in the first place. The program is designed to find something better for the child and create a stable situation. No formal evaluation systems are in place right now. Court-ordered placements would like to look longitudinally at the success of the program but it is difficult without a control group. It also becomes more complicated because even if a minor goes through a diversion program and has to be placed, studies have shown that this case may still be successful. The measure of success could then be that when the minor was placed he/she did not require the highest level of care, or the length of care was reduced as a result of diversion programs. "If you could tell us how to measure that, that would be wonderful." Outcomes: As an arm of the court this program provides services that are mandated by law. "So any time there is abuse, neglect, delinquency of the minor, drug use, a whole gamut of things, you must find the most appropriate placement for the minor." Some things within public service "don't lend themselves to bottom-line outcomes because we don't know what the baseline is". The differences with each case makes it difficult to establish measurements. "The outcomes for one youth may be that after a period of stabilization, we are able to lower the level of care to a point where a minor resides in a foster home until emancipation. For another that they overcome drug issues and return home and then be able to function in their family structure." "It is also very clear that because of the individualized success measures of each case it would be extremely difficult to have one formatted outcome that could be applied across the board." "Too many of these things are trying to find a quick fix to problems. If you are looking at any number of social service issues, poverty, drugs, gangs etc.., these problems did not just occur yesterday and they are not going to be fixed tomorrow, they require a long-term commitment to turn around. We do not have the resources to turn these things around independent of the community where they are occurring. Change is going to take place over a long period of time. The recognition that there is going to have to be a real working partnership with people in the community is fairly recent. Short-term restructuring is not going to be very effective, we need long-term solutions." Believes that cases are too individualized to adopt overall measures of success and that comparison control groups would be needed to do proper evaluation. It would also be difficult to establish baselines. Outcome-based evaluation systems are not currently in place and computer and staff resources are limited. 52 8. Department of Probation, Electronic Monitoring and Home Supervision Services: The electronic monitoring and home supervision program was created so that clients could stay home while they were awaiting trail, awaiting sentencing, or serving out their term. Clients are assigned to the program for a period of time and most clients are assigned to home supervision; electronic monitoring tends to be the exception rather than the rule. This program helps alleviate some of the over-population problems in juvenile hall. The program has doubled over the last few years. Further expansion is limited by the number of operational monitors in the County. The mandate is to enforce the request of the Juvenile Court Judge. Clients must stay at home, not commit any more crimes, go to school, and go to drug treatment. Anything that the court requires, like AA meetings or counseling, this program watches to make sure that it happens. "We are the eyes of the court." Contact is made with the families daily; the court orders one contact per day per client. Currently the program monitors 93 clients. "If a kid does not comply with the court order we can reprimand, bring the agreement back to court, or arrest them. This is the most stringent form of probation." When program staff have time they provide referrals to programs like Families First or Tough Love. Staff members are not required to intervene but if they observe something in the home they try to address those issues. "If they see a lot of family trauma that has never been dealt with they will make the proper referrals." The longer a client is sentenced to the program, the greater the opportunity for staff to provide more in-depth services. Goal: The goal of this program is to enforce the court order and protect the community by not allowing the participants to commit more crimes. Current Evaluation Mechanisms: The current measure of evaluation is to gather violation reports. Quarterly reports are completed that show how many kids have completed the program successfully and how many have failed the program. However, failing the program is not necessary a failure because it means that staff detected a situation where a youth was not doing what the court asked and the staff member in charge of the case brought the client back into custody. "This is the only quantitative measure that we have." "If I only have a 5%rate of failure, that does not mean that our program is successful. It could mean that staff are not doing their job well enough, that they are not following up on problems they see." The failure rate has always been between 19-27% for home supervision. Electronic monitoring is more stringent and the failure rate is closer to 30%. "If you contrast this with the types of things these clients have done, we are doing a pretty good job." Data/Resources: This program is extremely short of resou.ces. Since March of 1995, when this director came on board, one computer has been donated. The program is also short of vehicles; staff wait for other staff to return so that they can go out into the field. The program director has 53 also requested an additional staff member. All reports are done by hand. Data is collected on the number of home visits, visits with parents, contacts with police and schools, number of phone calls to each residents. The goal is to see 100% contact with each person in the program. This is not always possible but should be within 80-90%. Currently the program is at 77%because it is short resources and staff. The contact rate is used to let the director know if the staff are overburdened. In the main office, Electronic Monitoring and Home Supervision has three computers that are linked to the probation department. The main database is used to get impact records, previous histories, and court dates. As far as collecting outcome measurements, the data could be retrieved from database, but"frankly we are so stretched right now that just doing these statistics is a major endeavor. Even though I am a supervisor I still am considered a staff person and I have to work on the line three days a week. I am requesting more staff because it doesn't really function effectively this way." The program director would also like to have a computer linkage with Social Services. That would let staff see records of Child Protective Service's contacts in the homes as well as animal control contacts (households that abuse animals also tend to abuse children). This would give staff better insight into their clients. "This is something we would like but we have been told that there are no resources to do this at this time." The program director would also like to share a database with the police department. Right now staff make a phone call or send a fax to tell the department that someone is on house arrest in their area, "but I think that the police departments tend to throw away this information because they too are overburdened. If we had this linkage we could do daily updates." These linkages would also reduce duplication in computer entries. This program is evaluated by the number of new crimes committed by their participants while in the program. These numbers are tracked and tabulated quarterly. "Last quarter I had six new crimes out of approximately 200 cases. Sometimes you cannot help a new crime being committed, since we are not there 24 hours a day." The type of crime committed is also tracked. "If a kid commits a violent crime while on the program I look back over their record to see if we could have predicted that. Usually we couldn't have." Outcomes: When this program can work with a participant for a longer period of time it is able to influence some outcomes. For example, staff members collect attendance records and if the participant is not going to school, then that staff member can take the participant back to court. Contact with the client ends once the program is completed. "Contact post program is more the - exception than the rule, it is not what we are paid to do." The program director sees a use for outcomes especially around budget time when he is trying to get funding. "It is those programs that can't show any outcomes that will be cut. It would be nice if I had more statistical evidence that shows our impact. It would also help us show other groups in the community what we are doing. I wouldn't mind doing it, if I could. It would just take staff. I could probably do it with one full time clerk." 54 Would like to have more measures, can identify some outcomes, but does not have the staff time or technology to do it right now. 55 9. Department of Probation, Orin Allen Rehabilitation Facility Services: Orin Allen is a 74-bed facility. It operates as an arm of the court within the probation system. Programs are offered on drug abuse, victim awareness, gang diversion, and year around schooling. Teaching socialization skills is an important part of the process as well as individual counseling and case management. To supplement current services provided at the facility, Orin Allen just received a grant to provide after-care for its participants once they leave the facility. Two probation counselors are in charge of the youths' cases for 45 days after discharge in order to ensure that the participants go back to school, get access to resources in the community, are monitored closely, and receive drug testing. As an arm of the court client participation in the program is legally mandated. The focus of rehabilitation is placing responsibility on the individual; "we can get you back in school, we can help you get a job, we can refer you to counseling, but you have to carry the ball and go with it." Goal: Goal is to protect the public, carry out the court's mandates and offer the youth opportunities to change. The goal of the after-care program is to maximize the youths' initial adjustment upon leaving the facility. Current Evaluation Mechanisms: "We have a situation here in our particular field where we have some things that are very difficult to measure and some things that are relatively concrete and easy to measure." Each of the youths sentenced to the facility are at a different stages of rehabilitation. As a result, success measures must be individualized. However, curing delinquency in all kids is not possible. For some it is"going to take a lot of intense work, guidance, and a whole lot of resources to improve their chances of success." An evaluation component is currently being added to the after-care program since it is required by the funder. Measures are now being designed that will evaluate the program's ability to improve Orin Allen participants chances of success. One of the measures now being used is whether the clients return to school in the first five days after leaving the facility. The two probation officers are checking to see if the youths are in school and help expedite their enrollment. The program director believes, and studies have shown,that school attendance will help prevent future delinquency. The number of referrals a :.outh receives to resources in the community is also measured, like mental health services, drug treatment, and jobs. In addition, within the aftercare component, recidivism rates during that first 45 days are tracked. Both technical violations (i.e., violation of their probation status, drug usage, incorrigible behavior at home, staying out late) and new criminal violations are tracked. Prior to the grant extensive evaluation was not conducted. Orin Allen previously had one probation officer at the facility who handled all cases and when the youths left their cases were transferred to a probation officer in the field unfamiliar with their background. The after-care grant will last for three years and is renewable on a year to year basis. "People who have spent time in the field know that this after-care program makes sense. However, it is often difficult to translate that in some sort of number crunching way. We know that getting a kid into school and referring them to outside resources, giving intense supervision and holding them accountable for 56 their actions by letting them know that someone is out there watching, will increase their chances of success. It will prevent them from falling into bad habits in the first few weeks - which can happen." ' Data/Resources: "I think all of the data is there, in the overall system, but it is scattered. It is like finding a tree in a forest." It would be very labor intensive to follow up with past participants. Some leave the county after the program. No systems are being established past the 45 day follow-up period. The measurement of returning to school will be examined as part of the evaluation. Through the Safe Futures grant an outside evaluator has been hired to develop the systems for the after-care component. This resource has been very helpful. Without the resources provided in the grant it would be very difficult for the program to establish good working measurements. Outcomes: Since the Safe Futures grant was received, staff members have begun to identify quantifiable measurements that wilt' reflect the impact of the program on the youths that they serve. "Hopefully we will positively impact these kids while they are in the system and after their release." Prior to this grant the funding was not available for evaluation and the funding sources did not require measurable outcomes. Since the program is early in the process, implementation is not yet complete, and results are not available. The outcome measures identified so far include school attendance, recidivism rates, and community referrals. Some outcome-measurements have been defined but are just beginning to be tracked. A three year grant is providing the necessary resources and technical expertise to make this possible. 57 10. Department of Social Services, Emergency Response Services: Emergency Response is a federally funded program and the goals are established by the federal government and reinterpreted by the state. There are a variety of services that Emergency Response provides in order to meet its mission. Emergency response receives calls that describe a potential situation of abuse. Over 4000 calls come in a month and by year end Emergency Response will have made 8000 physical contacts with clients. When a call comes in ER decides what intervention is needed based on the degree of risk to the child and set guidelines that must be followed. The call could have insufficient grounds for intervention(i.e., it is not petitionable by the courts) and the case be closed, or the call could lead to voluntary or court-ordered services like family maintenance, family reunification, permanency planning and adoption. Program staff have 12 months to reunify the family before moving to permanency planning. Goal: The ultimate goal of the program is to protect the child. This is done by measuring the degree of risk and making a determination as to what is best for the child. Current Evaluation Mechanisms: There are two ways in which ER could be evaluated. One would be whether the right intervention was selected based on the information heard on the phone. The second would be to follow-up with the child after the intervention and determine if the child is better off as a result. This type of evaluation is not currently done at ER. "We don't have a way of measuring the effect of our work in the long run to see whether our intervention has been successful...I do not even think that we have a mechanism in place that tells us the number of referrals that come in a month,that are assessed immediately or within 10 days which is the state requirement." ER provides single interventions. Cases are closed or transferred to other services as soon'as ER's role is legally fulfilled. Future impact on the child is not monitored. The speed and accuracy of how cases are handled is evaluated by internal quarterly reviews and by State audit. Data/Resources: The State of California Health and Welfare Agency requires a monthly statistical report from the Emergency Response programs state-wide. This form, the SOC 291, is the sole data source in this department. It takes a monthly aggregate of referrals received, cases closed, referrals made to other programs, and reasons for emergency response (i.e. the type of abuse). It does not identify processing times or any information that could be linked to individual client outcomes. ER compiles the information needed for the SOC 291 and then forwards that information to the State. The information collected on that form is minimally used internally within the department. With more time and staff more data might be collected. Outcomes: As a federal program that is run state-wide, any innovations in evaluation, like outcomes, would be more likely to come from the State or Federal government than from the county. The State has devised a new computer system that will allow records to be maintained case by case. This new technology holds some promise for this department to achieve outcome data. The department is short of staff time and resources. Spending time collecting data is considered 58 a luxury. At times they are even unable to meet the basic mission of seeing every case that is referred into the department. In addition, because other programs are short resources ER cannot always make referrals to the type of services that clients need. Having other agencies provide services in the community is vital to the success of the program. "We don't have enough staff time to do the initial screening process which is a highly scrutinized process. And then people are going to stop and figure out what outcomes are? I don't think you are going to find anyone with the time to just sit around. I think on a case by case basis to have that information would be helpful. If a program was created with sufficient funding I think that we could do something." If outcomes were to be collected, it should be done for a sample of a cases, and they should be tracked individually as they move through the system. The reason that this is not being done now is that the resources are not available and families move out of the county. No evaluation mechanisms are in place either in relation to outcomes. Outcomes have not been identified and clients are not tracked once they leave this program. Staff time is limited and while computer hardware is prevalent, computer systems useful for collecting outcome data are not. 59 11. Department of Social Services, Independent Living Skills Program Services: Using Federal dollars, this program works with foster care youth between the ages of 16-18. Workshops are provided over a two year period in money management, health and hygiene, career and job development, educational development,prevention programs, and self • esteem/communication/positive relationship building. The program is designed to prepare the youth for the transition to self-sufficiency upon leaving the foster care system. The program is voluntary, so youth may enter and leave whenever they decide. Youth are informed through a newsletter about the services that are available and it is up to them to decide to attend. Cash, transportation, and school credit incentives are given to the youth to encourage them to use these services. Goal: The goal of this program is (1)to give all participating youth pride in being part of the foster care system, (2)to help those youth become self-sufficient when emancipating from the system, and (3) expose youth to opportunities in the community and in society. Current Evaluation Mechanisms: The youth who participate in the workshops complete questionnaires that ask them to identify the benefits gained as a result of the workshop, their opinion of the facilitators, and their overall feeling about the program. Staff members, advisory boards, and youth meet regularly to talk about how the program can be improved. Program staff like to get feedback on how the program can attract more youth and improve services to the youth who are currently participating. The results of the program are often seen after the youth leave the program. Some youth, on leaving foster care, are not yet ready to be on their own and self-sufficient. Program staff find that after one or two years the youth contact staff members to let them know the positive impact the program has had on their lives. There is no formal mechanism in place to track the youth once they leave the program. However,the relationship built with these youth while they are in the program helps administrators stay in touch with graduates. Client success is evaluated on an individual basis. For many clients developing greater communication skills is very important. These outcomes may not be considered important to people outside of the program. "People coming a long way in terms of their personal development" may not be considered an outcome by some, but"to me it is a lot in terms of achievement." So much of the program is about the process getting to the end result; outcomes might have a hard time capturing all that goes on in the program. This program prepares clients for what will come next in their lives. The program director feels that these lines of communication with graduates are sufficient to help him improve the functioning of the program. Data/Resources: Data collection/analysis is not a large part of staff members' time. Every month, staff members tabulate the number of;outh that they meet with one-on-one. If Federal dollars started to shrink and the program had to show proof of its effectiveness, then staff would start collecting data. Once a year data is gathered on the number of participants leaving the 60 program that are going on to college, into the military, working, and the ones with no plans. Statistics are calculated based on this data. Last year S 1%of their graduates went onto community college, a UC school, or another university. So far none of these graduates have dropped out of school. Follow-up data again is collected on an informal basis. An annual report required by the Federal government asks for the number of youth who participated in the program, number of teenage mothers and fathers, number in special education classes, and number going into transitional housing programs. The program director wishes that the program had greater database capabilities. Most contact information with youth is kept manually, it would be better to get this information on the computer. Other people in Social Services and counselors on staff would benefit from having that information accessible. Databases could include information on the conditions facing the youths when they entered this program and what their plans were on leaving. Through the help of Social Service clerks program staff are able to access information on their clients. In addition, sufficient computer hardware is available in the office. While the youth are in the program their home situation and school performance are monitored. The program also provides tutoring to all the youth in the program. Focusing on academics is a new component to the program. Outcomes: If this program was to start measuring outcomes, it would require more help from the clerks to do that kind of tracking. The program might also need to bring on another staff member. In addition, some sort of user-friendly software would be needed to hold this data. Staff would also need to sit down and decide exactly what measurements could be used that would show outcomes. One possible tool could be pre- and post-tests of the youth in the program. Some outcomes could be did the youth get a summer job, were the youths informed about STDs, and are the youths able to approach the relationships in their lives in a healthier manner. But"we would still be missing a lot of important elements in terms of the program." Especially in terms of the long-term changes that may be difficult to capture within this model. "I am all for learning how to measure (some of these intangible things), if someone comes up with a way, I will incorporate it into our program." One way that outcomes would be helpful is in providing justification for making large changes in the program. For example, staff have not felt that 18-19 year olds leaving the system are ready to be on their own. Perhaps outcome data would justify extending this program until the youth are older, or even to start the program earlier. Right now it is assumed by funders and the department that this program is producing positive outcomes. The program director has complete flexibility over designing the components of the program. One challenge that this program would face is :Ls voluntary nature. Youth can enter or leave the training at any time. 61 "There are times when I would like to see how we stand or how we have truly impacted a young person." Obtaining information from a variety of sources, foster parents and youth for example, would show the benefits of the services provided. Outcome measurements would be helpful if this program begins look for private funding. Private funders want to see measurable outcomes. "I would just wish that if a statistical component is added to our program, that it comes with some help. Enough training given to our staff so that they can grasp the concepts so that we can make this work on our own." Program participant data is kept in files and not on the computer. Outcomes may not be able to capture all that goes on in the program. Outcomes could be beneficial if the program had user- friendly computer software, staff, and training on how to use outcomes on a day to day basis. 62 12. Housing Authority, Section 8 Assisted Housing and Public Housing Services: The Department of Housing and Urban Development (HUD) funds the Housing Authority. In both Public Housing and Section 8 residents pay 30%of their income in rent. The Housing Authority is landlord to residents at eight family public housing developments and at six developments serving elderly and disabled persons. "Beyond Shelter"programs are provided to residents. These programs include: computer learning centers, child care centers, community- oriented policing system, recreation/youth prevention programs, resident participation, outside work experience program, and Arts Alive. The Family Self-Sufficiency program is also available to Section 8 participants. This program provides an incentive to increase household income. (See Attachment 1 for more information on these programs) Goal: "The mandate of the Housing Authority is to provide safe, decent housing to Contra Costa County residents who are low-income, elderly, or disabled. Two programs, conventional public housing, and Section 8 housing assistance, are the major means to accomplish the mandate." There has also been some movement towards incorporating self-sufficiency as a goal within a variety of Housing Authority programs. Current Evaluation Mechanisms: Most evaluation methods are linked to a grant in which the grantor requires certain things to be measured. However, in writing the grant the Housing Authority can be creative about what will be measured. These grants provide funding for services generally outside of the basic mandate of providing safe and decent housing. These services are the Beyond Shelter programs. The philosophy behind these programs is a concern for quality of life issues, scarce resources, and self-sufficiency. "Long before welfare reform became a household word, we were concerned about movement to self-sufficiency, probably because housing assistance is not an entitlement. We know that we can only serve a quarter of the need and that Congress is not renewing all replacement funding for assisted housing. We need to move people out of public housing and move other people in off waiting list." For example, our OCJP (State Office of Criminal Justice Planning) grant has very meticulous evaluating criteria. Attention is paid to establishing a baseline and providing statistical analysis. Data/Resources: Quantitative and qualitative data are collected on Housing Authority residents. Public housing residents are surveyed a great deal, Section 8 participants do not get surveyed as much. Partly this is because the philosophy with the 5500 Section 8 housing residents is that your life is made better as a result of assisted housing but no one else knows that. When Section 8 participants meet yearly with a worker, data is collected on family composition, income, employment, and general information on how the resident is doing. The housing inspectors actually have a closer relationship with the residents because they go out to the sites, meet with the residents and resolve conflicts with the landlords. The Housing Authority would like to see more information collected that would help move Public Housing and Section 8 clients to self-sufficiency. "You have to know where the client is, to know where the client might be going." Some of this information would be baseline 63 indicators and information on what their prospects are for getting a job. The information gathered now for HUD helps the Housing Authority see if the programs are in compliance with HUD regulations and to see if people are housed decently. Information gathered on family characteristics and similar data is also used to apply for grants and for developing new programs. Those grants require additional information to be gathered that gives more information about how clients function. The Housing Authority tries to leverage resources by working with other organizations in the community. By working with the community, program staff are trying to meet the goal of a computer center in every family development. Outcomes: The Housing Authority believes that programs have to be outcome-oriented. It is a positive concept taken from the private sector that the public sector should adopt. When grants are written for the Beyond Shelter programs, funding sources ask for specific measurements. However, the grantors provide enough flexibility so that the program manager can design the program to truly impact the community served. However, "anything with people is not totally predictable. Measurement and outcomes require flexibility and need to be adaptable to see what is actually going on. That is part of the decision-making of management." "If you start with a vision of what it is that you want then probably you are going to make the money go farther because you will make better decisions about structuring your program and you will be creative about leveraging money through collaboration. When designing outcomes programs should start with what outcome it is they want, then work backwards with great flexibility. The more rigidity built into job descriptions the harder it will be. Private and public funders have helped the Housing Authority move towards more outcome measures. Each grant has different measurements like employment goals, education, and intervention outcomes. Resources are in place to collect measurements of program success. The basic mandate of providing safe and decent housing to eligible participants generates primarily specific measurements of client eligibility and housing quality. 64 13. Mt. Diablo Unified School District, Foster Youth Services and Bay Point Healthy Start Services: Healthy Start started in 1989 in concert with the County Administrator's Office. It was a new way to provide integrated services with partners from Social Services, Probation, and Mental Health in Bay Point. A history of integrated projects in Bay Point and in the Mt. Diablo ` Unified School District laid the ground work for the Healthy Start program, along with two planning grants that provided start-up funds. This program provides County services at the community level. Services include: adult education, after-school classes, English-as-a-second- language, health services, mental health services. While the planning process for Healthy Start helped the community focus on providing integrated services to families in the area, initiatives like AB 1741 kept the idea of integrated services on the political agenda. The Bay Point Healthy Start program was funded in the second round of funding from the State. Since the Healthy Start funds were received, families who have chronic, long-term problems that are also eligible for services through the County Service Integration Teams (SIT) are referred by the schools to receive intensive case management. Family Resource workers work in the community collecting these referrals and handling a variety of family needs. Local adult education classes are provided at the Healthy Start site, as well as medical services. The health clinic is open to the entire community. Linguistically diverse outreach workers increase access to and use of services by the community. The Foster Youth Services Program has been around in the school district for a number of years. It serves youths in out-of-home-placements from Probation and Social Services who are in kindergarten through the 12th grade. Services include: tutoring, employment support, preparation for emancipation, a variety of social work services. The goal is to provide on-going support for foster youth so that they can be successful in school. This entails working through behavioral and social issues as well as providing independent living skills. This program also attempts to collect and maintain the education and health records of youths in foster care so that these records are not lost when foster youth transfer schools. Goal: The goal of the Healthy Start program is to ensure that the services established in the community are still there when the initial program funding runs out. The services that Healthy Start is well on its way to establishing in the community are adult education classes and local health services including WIC, screenings, and immunizations. The goal of the Foster Youth Services program is to provide on-going support for foster youth so that these youth can be successful in school. Current Evaluation Mechanisms: The impact of the programs in terms of results are not measured in any sort of systematic way. Attendance records are compiled and program staff have found that attendance rates have been going up as a result of the programs offered in Bay A Point. The health services component will get tracked but the systems are not yet in place. Eleven different data systems exist in the Health Services Department that are not interconnected and this creates organizational barriers for collecting data in an integrated, multi-department program. "The actual tracking now is not something that I can just pull up on the computer. If 65 someone asked me I could get projections from Health Services on, for example, the number of client visits, but the actual medical data is not there." However, the appreciation expressed by ' clients and rate of visits to the clinic by members of the community demonstrate the success of the health program. Once community outreach and marketing are underway for the clinic usage } and data collection will increase. Data/Resources: The Healthy Start grant allowed program staff to go out into the community and survey people who have received services through Healthy Start. Surveys are now being collected and the initial results show that the program has been successful in impacting the lives of people in the community. It would be helpful to also collect this information from the people who visit the clinic. In addition, this program would like to have medical data that shows the types of chronic diseases and infections prevalent in this community. This would give baseline numbers on the health status of the community. Program staff would also like to track utilization rates and the impact utilization of services has on certain types of health conditions. The Healthy Start grant provides funding to collect school-based data only (like attendance rates). "We have some basic information that we provide to the State for Healthy Start, but it is geared towards the educator's mind, and it is used to justify expenditures to the Legislature for Healthy Start." "Our funding has not gone into providing a real sophisticated evaluation system. One of the things that I am aware of as a result of looking at outcomes is that it is very expensive to be able to pull this off, and I am trying to sell the idea to the school district that we invest in a full-time evaluator who can look at identifying outcomes cross programs. It is not something that I can do, and it is not something that can be done by any small percentage of one person's staff time. It really takes a concerted evaluative mind and some financial resources to pull that off." Outcomes: Program staff have identified the types of outcomes that they want to measure. Outcomes have been looked at in the development of the Healthy Start program in Concord and in relation to the Foster Youth Services program. "It is one thing to talk about outcomes, because intellectually everyone says that if you are spending money you better be getting some results, but when you begin planning to collect this information and bringing together the people involved in the process, you realize the number of barriers that exist." Outcomes are often discussed within the programs and the value of using outcomes is recognized by program staff. "Outcomes help you focus your thinking and actions on getting results that you believe are good expenditures of time and money." Attendance is always an outcome that will be tracked by this program. Poor attendance rates are often an manifestation of negative family functioning. This measure becomes an indicator of a need for services within that family system. The program director believes that health status is one of the main outcomes that this program should be looking at. Some of these measures could be: did children get immunized at the right times in their childhood?, did the child get the recommended number of well-,:hild visits?, when children k:e screened and a problem is discovered do they get the appropriate referrals?, are follow-up services provided when a 66 problem is detected?, are we helping parents to provide good at-home treatment?, and can a reduction in emergency medical attention be seen as a result of our services? Collecting this data would help convince the medical establishment and funders to shift expenditures to the types of services provided through Healthy Start. 1 Getting the data necessary for outcomes will be the challenge because data will have to be collected from multiple databases. Front end work and time will be required to create data systems that are interconnected and can track the kind of outcomes that this program wants. "I think that outcomes are really labor intensive and expensive. Its an easy expectation to say that we should be collecting outcomes, but it is a high expectation to have programs start doing it without any resources." The program director believes that it is a good use of the County's time to push towards outcomes but they should first implement pilot projects that will give credibility to the system of outcomes. Once a system of outcomes are in place, Contra Costa County programs will receive notice from groups outside of the County and this will help bring in outside foundation funding. Have identified outcome measures to collect. Attempting to design systems now that will link multiple programs' data sources together in order to collect the information needed for outcome measures. Believes that outcomes are expensive in terms of time and money. a 67 14. Private Industry Council, Summer Youth Employment Training Program (SYETP) Services: SYETP is intended to be a summer employment and training program for youth 14-21. ' The money comes from the Department of Labor, Job Training Partnership Act (DTPA) and it is allocated to states by formula. Once the money comes to the state, 100%passes through to the Service Delivery Area(SDA), e.g. Contra Costa County. There are general Federal guidelines for permissible and mandatory activities. The local Private Industry Councils (PIC)which manage the JTPA funds receive Department of Labor guidance which is just short of regulation. In the past the goal of SYETP was simply to put kids to work for the summer. A new emphasis in the past few years has been to include educational enhancement components in the programs. The goal is to link classroom training and academics with the summer worksites. Goal: Provide summer work experience and education to low-income youth ages 14-21. Current Evaluation Mechanisms: Every other year each Private Industry Council is required to develop a plan that outlines its goals and objectives. Summer Youth is included in that plan and a stated objective for SYETP is to help youth achieve occupational goals. Also written in the plan is the objective of enrolling youth in appropriate vocational classes. The measurement provided is that at least 80% of youth receive a grade and credit. The Contra Costa PIC is measured on how well it fulfills this requirement, and the PIC in turn evaluates the service providers in the community who they contract with for direct services. Much of what the PIC is evaluated on involves process and eligibility requirements. For example, did contractors enroll the proper number of students?, the appropriate ethnic breakdown?, and provide sufficient hours of training? The SYETP is evaluated on a few specific outcome measures. The most pronounced is whether the youths leaving the program return to school. Baselines are set, generally between 80-90%, for successful performance in this area. Other possible outcomes are getting a job or enrolling in a vocational training program. However, this program is short in duration and the measures only examine the immediate results of the program. They are not able to know if the classes and work experience provided impact the youth in the long-term. Data/Resources: Data collection for this program is extensive. A file is kept on every participant and data is collected on income levels, reading and math scores, and general attitudes. PIC does wish that it could collect some follow-up data. For example, it would be nice to see if SYETP participants' performance in school is better the next year. However, this would be an expensive undertaking. Some of these studies would be done better on a national scale. While much of the contractors' time goes into data collection, significantly less goes into data analysis. This is in part because of the short duration of the program. Outcomes: As an outcome driven program (year-round adult JTPA funded programs are even more outcome-driven), SYETP has certain mialmum standards that must be met. This also creates relatively consistent programs nation-wide. "If DOL did not at least put certain minimal standards you would really have 50 different SYETP programs. DOL builds in a minimum level 68 of quality. In general it makes for high quality programs." However, because PIC collects so much information, a lot of time is spent ensuring that the paperwork is completed properly, sometimes at the expense of developing a creative program. ! Because PIC funded programs, like SYETP, are heavily regulated by DOL, it is explicitly clear how money can be spent and what outcomes are expected as a result. But there have been times when developing measurements have created consequences that the DOL or PIC did not intend. For example, in an effort to bring down the cost of the programs, a cost per placement incentive measure was added. As a result, contractors started designing cheap programs and quality was sacrificed. While this was not what was necessarily intended, the measurement made cost the major determinant of a good program instead of quality. PIC has extensive data collection systems in place. The majority of their programs are outcome- based and funding is linked to performance. While many of the indicators relate to process, all programs are tied to quantifiable outcomes.