For a shorter overview, read “Evaluation Answers the Question ‘Are We There Yet?’”

This article is about an initiative that impacts the lives of every child in New Mexico and the role of evaluation in ensuring its success because, at the most fundamental level, it cannot afford to fail. For an overview of the 100% New Mexico initiative’s evaluation process, please visit our evaluation page.

With hundreds of smart, dedicated stakeholders working in the county-based initiatives, there are very important questions we need to ask in order to help them reach their goals.

“Are we making a difference?”
“What are the key takeaways from the evaluation?”
“What does the evaluation data reveal about progress made?”
“What type of progress impacting children and families can be identified by evaluation?”

First, why are we doing this work?

Every day, our children endure abuse, neglect, and trauma, creating a public health crisis that is mostly ignored. The goal of the 100% New Mexico initiative is to prevent adverse childhood experiences (ACEs) and family trauma by ensuring ten vital services for surviving and thriving. The challenges our children and their struggling families endure, can be understood by reviewing the county-level data from the New Mexico Department of Health, which includes reports on:

1. Child physical, emotional and sexual abuse
2. Child emotional and physical neglect
3. Middle and High School students feelings of hopelessness, suicidal ideation, attempted suicide and suicide, and adult mental distress
4. Rates of domestic violence and sexaul assault
5. Substance use disorder-related illness, injury, and fatalities
6. Teen parents (who may not have the resources to support a family)
7. Children living in poverty
8. Housing insecurity and homelessness
9. Unemployment
10. Food insecurity, adolescent and adult obesity, and degree of access to healthy food

The data above reveal that a troubling percentage of our children in each county are growing up enduring adverse childhood experiences (ACEs), which include traumatic forms of maltreatment and growing up in households where parents and adults household members are misusing substances, have mental health challenges, and are violent toward one another. A complicating factor for those seeking to prevent ACEs is that they represent ten distinct, yet related forms of neglect and abuse, as illustrated by journal articles that can be found on our ACEs and Research page.

You can also visit any of our local 100% New Mexico initiative websites that provide to stakeholders within each local initiative with the data collected by the NM Department of Health. Simply click on a county from our map and choose “Action >> Spotlight on Health” from the county’s main menu.

The pay-as-you-go society

To answer questions about the evaluation of the 100% New Mexico initiative, we first have to provide some context. We live in a nation that does not guarantee anyone the right to access vital services like medical care, mental health care, stable and safe housing, a secure food source, or transportation to vital services. We are what might be called a “pay as you go” society, which means that if you can afford survival services, like health care, you can surely buy them or get them through one’s high-paying job via health insurance. (We add “high-paying” since most health insurance requires co-pays and additional fees to maintain one’s medical care. As for mental health care, that’s mostly an out of pocket service.)

As for the services shown to help a family thrive, one must also often pay for parent supports, early childhood learning programs, fully-resourced schools with the staff to ensure high academic achievement, mentoring programs, and a highly rated university to get that lucrative job. Yes, in some localities the government has been altruistic, supporting subsidies for certain services, if one qualifies, if service providers exist, and if hours are friendly for working parents. (That’s a lot of “ifs.”)

Neither our nation, nor any state within it, has a history of providing universal access for anything to support families, except education and even that leaves room for debate.

What our 100% New Mexico initiative is doing has never been attempted in the US before. We are advocating that all families have access to the ten vital services for surviving and thriving. In our countywide surveys in New Mexico, we know that anywhere from a fourth to half of a county’s parents seeking a particular vital service may struggle to access it. Reasons include cost, not qualifying, too far to drive, no transportation, lack of friendly hours for working parents, unaware of the service, or the service is not provided locally.

Fixing these barriers will require a data-driven process to pinpoint where in a county service disparities exist and why. Solutions may arrive in many forms including public-private partnerships, increased city and county support, and the use of technology, including providing access to the internet.

Creating a plan to evaluate progress

Our initiative and evaluation of each county’s work are guided by what is called our 100% New Mexico Framework for Change, which describes all the activities required to assess challenges to accessing ten services, identifying evidence-informed solutions, securing funding for projects shown to reduce service barriers, implementation of projects, and evaluating the short and long term effectiveness of each solution. This means that we ask questions, such as, “by establishing food pantries at public schools, did we decrease hunger and food insecurity among the student population and their families?” Or asking, “by funding school-based health centers, did we increase student access to medical, dental, and behavioral health care?”

As you can imagine, some projects may be done and evaluated quickly, such as creating a food pantry at a school. Other projects, such as securing funding and then building a school-based health center, plus recruiting staff, could take years to establish and evaluate. Strategies for developing vital services can be found on our Ten Sectors and Research page.

To be clear, Evaluation means evaluating local capacity-building which can take years, especially in rural and urban areas that endure significant pandemic-related disruptions in healthcare, education, social services, and jobs. Building the infrastructure, county by county, to ensure ten vital services is a long term project, with many benchmarks to evaluate along the way that include the following steps A through G:

[A] IDENTIFYING THE CHALLENGE
[B] RECRUITING THE LOCAL 100% NM TEAM
[C] EMPOWERING AND EDUCATING THE TEAM
[D] PROCESS
[E] ENGAGEMENT
[F] RESULTS
[G] OUTCOMES

Change: A thru G

Opportunities to measure each phase of the initiative follow. We are discovering that the process is not linear, and some counties might work on public awareness before forming action teams or develop resource directories before establishing a secure institutional base. The initiative was designed to be a customizable process, led by local stakeholders.

[A] IDENTIFYING THE CHALLENGE:

We begin the initiative’s local assessment process with our 100% New Mexico county-wide survey of parents and youth to identify local challenges and barriers to vital services. This survey is the first step in understanding the magnitude of service barriers and specifically which barriers make it a challenge to access local services. We can measure to what degree each county has the capacity to implement the survey through a core local team.

Indicators of progress to look for could include:

1. The results of the 100% New Mexico countywide survey, indicating barriers to vital services
2. Analyzing the data and what barriers to service mean to families and communities. For example, in the arena of food security programs, barriers include unfriendly hours of service, lack of transport to service, lack of qualifying for service, too far to drive, and lack of awareness of where services are located.
3. Local stakeholders are taking ownership of the problems and making security a priority.
4. Local service organizations are listening to reported barriers and improving the user-friendliness of the service.
5. Willingness of those in a sector, for example food pantries, to collaborate with one another to create a coordinated countywide system of food security

Countywide survey reports may be found on our Anna, Age Eight Institute/NMSU site.

[B] RECRUITING THE LOCAL 100% NM TEAM:

Local champions then mobilize and create a local team, including a variety of task forces and ten action teams, with each action team focused on building a seamless countywide system of a particular service (like food security or mental health care). We can measure the local initiative’s capacity to engage its action teams leaders in book clubs, and “getting on the same page” by reading the blueprint for the 100% New Mexico initiative called 100% Community: Ensuring 10 vital services for surviving and thriving, as well as the book Anna, Age Eight: The data-driven prevention of childhood trauma and maltreatment.

We can consider measuring to what degree each county has the capacity to recruit action team leaders and members. Most counties are able to create a robust local initiative team with leaders from local family-serving agencies, the local health council, local elected leaders, and representatives from local foundations.

Indicators of progress to look for could include:

1. Recruitment of those who take ownership of their particular sector and are committed to improving countywide service delivery
2. Recruitment of those with social capital who have relationships with elected leaders and agency leaders, all key partners in funding and supporting a countywide system of service.
3. Recruitment of those with an interest in project development, open to using continuous quality improvement to identify barriers and the evidence informed solutions to those barriers
4. Recruitment of allies and partners
5. Recruitment of those, within each of the ten sectors, with the capacity to turn the opposition into allies
6. Recruitment of those with the capacity to reach out to powerful authorities who control funding
7. Creation of a space within the initiative to provide room to hear from dissenting voice, even those who might be considered obstructionists and troublemakers

[C] EMPOWERING AND EDUCATING THE TEAM:

The local initiative commits to a shared vision and other components of the Collective Impact framework. The Anna, Age Eight Institute can provide workshops and seminars on creating a shared vision, as well as teaching action teams about the social determinants of health, public health social-ecological model, health equity, racial equity, Maslow’s Pyramid: Hierarchy of Needs and the four skills of continuous quality improvement: assessing, planning, action and evaluation. Training on Adaptive Leadership, a process that guides transformation and helps local stakeholders address push back to change, is also provided. To review the research and frameworks guiding the initiative visit our Adversity and Research page.

We can consider measuring to what degree each county local initiative team members report being satisfied with their involvement and the ongoing training offered that empowers them and provides a clear vision, goals, and strategies that can counter local leaders defending the status quo.

Indicators of progress to look for could include:

1. The local initiative is sponsoring and facilitating training on key skill-building
2. Interest among initiative participants in ongoing learning
3. Satisfaction among participants within the action teams
4. Empowerment of initiative participants that help them take ownership of the success of the action teams
5. Lack of overdependence on a single individual to move the initiative
6. Initiative participants applying adaptive leadership skills to deal with powerful stakeholders who are not interested in engagement or service capacity-building

[D] PROCESS:

Each of the ten action teams begins a process of continuous quality improvement (CQI). In this data-driven process, action teams identify the barriers to local services across their county. Then comes the process of informational interviews with services providers to identify why organizations might not be able to meet local needs. Another part of the assessment process is building the 100% Family Services Directory, filled with updated local services. From there, the action teams research the evidence-informed strategies/projects, policies and programs which show the most potential to remove barriers and increase access to quality services. Each team has access to 20-25 projects housed in our Eye on Solutions site to choose from as a starting point. We can measure to what degree each county has the capacity to initiate local projects and actively engage in local projects that will reduce the service barriers reported in the countywide survey.

The process of building vital services is a long-term project, requiring the funding and support for expanding medical care, increasing healthcare provider workforces, bringing in more behavioral health care providers to staff school and community-based centers, as well as identifying how to implement web-based therapy. Regular public schools will need to be transformed into fully-resourced community schools with health centers. The development of 100% Family Centers, serving as one-stop service hubs, will benefit parents of young children. All the ten vital services, including affording housing, accessible food pantries, timely public transportation, and increasing access to job training are multi-year projects requiring buy-in from local and state elected leaders. Each of the ten service sectors have their own unique development and funding process. (This part of the project will require evaluation over the next five to ten years at a minimum.)

Indicators of progress to look for could include:

1. Action teams that have changed their behaviors, from working on assessment and planning to the actual implementation of projects intended to decrease survive barriers
2. County workforce in local government or nongovernmental organizations prioritizing the work of removing barriers and committing part of their workweek to this effort (Note: This integration of initiative work into existing local staff work is our initiative’s preferred option.)
3. An acknowledgement among initiative participants that they lack the capacity to take on full-time project development work and have identified the need to hire full-time project developers
4. The local initiative’s 100% Community Schools Action Team has been able to prioritize key strategies which include increasing access to ten vital services by transforming schools into fully-resourced community schools with health centers serving students and parents.
5. The initiative’s 100% Parent Supports Action Team has developed another priority strategy in the form of launching 100% Family Centers: One Stop Service Hubs which link family members to the ten vital services for surviving and thriving, staffed with navigators who provide warm handoffs to family member, ensuring their access to local service organization on site, within the county or online.
6. The initiative’s 100% Housing Action Team has been able to work with local and state partners in the public and private sectors to ensure internet access for all households, increasing access to web-based services, public education including vocational education and higher ed, and public health information. This process should also ensure that each household has electricity and plumbing.

[E] ENGAGEMENT:

Community engagement is a vital part of the initiative process, securing buy-in from local residents and elected officials for working to ensure vital services. This is being done with 100% Mural projects that are part of public awareness focused on the need to ensure local services, as well as for recruiting initiative members.

We can consider measuring to what degree each county has the capacity to increase public awareness of the challenges facing children and families, as well as promote engagement with the initiative through community cultural events, plus sharing and discussion of our 100% BLOG articles. We can enlarge our sphere of engagement to include cultural workers like writers, artists, poets, filmmakers, and musicians. We can measure engagement of elected leaders who control the local budgets that impact service delivery and stakeholders with all ten service sectors.

Indicators of progress to look for could include:

1. Increased participation in the initiative with local stakeholders, elected officials, and the cultural organizations and collectives committed to the vision of 100%
2. Initiative participants trained in adaptive leadership to have an effective strategy when encountering powerful stakeholders who are not interested in engagement
3. Development of an initiative process that can build relationships with potential funder that include the following:
   • Making strategic investments in action team projects based on community feedback
   • Community members setting expectations and accountabilities for and with potential funders
   • Communities provided with unrestricted, multi-year funding
   • Communities and community-based nonprofits deciding who to fund (using funding from external sources)

[F] RESULTS:

Our focus is on the results in our measurable process. We can evaluate if our projects meant to reduce service barriers are being implemented and are increasing access to services, as well as improving services in all ten sectors. Projects like building the 100% Family Center: One Stop Service Hub and transforming schools into fully resourced community schools require a multi-year planning and action process.

We can consider measuring local project development in all ten service sectors. We can also measure to what degree local initiatives were able to create funding for staff to coordinate all local activities and track all stages of continuous quality improvement as projects focused on improving access to services were developed.

Indicators of progress through the initiative to look for could include increased capacity on the county level to:

1. Ensure that 10 vital services are being provided more than they were and for more people, with a priority on our three key strategies, developing community schools, 100% Family Centers: One Stop Service Hubs, and ensuring all household have access to the internet to access web-based services
2. Conduct regular countywide surveys
3. Analyze data and what “barriers to service” mean to families and communities
4. Take ownership of the problems and make security a priority.
5. Listen to reported barriers and improve the user-friendliness of the service
6. Recruit those who take ownership of their particular sector and are committed to improving countywide service delivery
7. Recruit those with social capital who have relationships with elected leaders and agency leaders, all key partners in funding and supporting a countywide system of service
8. Recruit those with an interest in project development, open to using continuous quality improvement to identify barriers and the evidence-informed solutions to those barriers
9. Recruit allies and partners, including local elected officials, socially engaged business leaders, and community change agents of all ages
10. Recruit, within each of the ten sectors, those with the capacity to turn the opposition into allies
11. Recruit those with the capacity to reach out to powerful authorities who control funding
12. Create a space within the initiative to provide room to hear from dissent voice, even those who might be considered obstructionists and troublemakers
13. Sponsor and facilitate training on key skill-building
14. Grow interest among initiative participants to engage in ongoing learning
15. Ensure satisfaction among participants within the action teams
16. Empower initiative participants that help them take ownership of the success of the action teams
17. Decrease overdependence on a single individual to move the initiative
18. Apply adaptive leadership skills to deal with powerful stakeholders who resist change
19. Identify to what degree action teams that have changed their behaviors, from working on assessment and planning to the actual implementation of projects intended to decrease survive barriers
20. Empower a county’s workforce in local government or nongovernmental organizations to prioritize the work of removing barriers and commit part of their workweek to this effort. (Note: This buy-in from local organization staff is our initiative’s preferred option, reducing the need for new hires.)
21. Acknowledge among initiative participants that they lack the capacity to take on full time project development work and have identified the need to hire full time project developers
22. Increase participation in the initiative with local stakeholders, elected officials, and the cultural organizations and collectives committed to the vision of 100%
23. Train local participants in adaptive leadership to have an effective strategy when encountering powerful stakeholders who are not interested in engagement
24. Develop an initiative process that builds relationships with potential funders that include making strategic investments in action team projects based on community feedback and creating unrestricted, multi-year funding
25. Create local sustainable and institutionalized funding to support the initiative

[G] OUTCOMES:

We work to achieve our vision and goal of 100% of families having access to the vital services, and with it increased health, safety, resilience, school achievement, job readiness, and employment. This is not a quick fix, as it may take many years to create a seamless countywide system of care, safety, and empowerment, represented by easy access to the ten vital services for surviving and thriving. Unlike most government funded community-capacity building projects, 100% New Mexico represents systemic change that requires many years of support once buy-in is established.

We can consider measuring outcomes by returning to the public health data on adverse childhood experiences, including all forms of abuse and neglect, as well as other indicators of health that include family’s improved access to all ten vital services, increased school engagement and student achievement, and increased job readiness and employment.

Indicators of progress to look for could include:

1. Reduced ACEs and family trauma
2. Decreased family involvement with child protective services and juvenile justice
3. Increased family self-sufficiency
4. Increased access to ten vital services within a county’s border
5. Increased access to the internet to engage with web-based services
6. Ensuring that each household within a county’s borders has indoor plumbing, electricity, is considered safe, and meeting housing standards and regulations.
7. Alignment of education with present and future workforce needs
8. Changing the attitudes of local government to holding the value that the role of government is to prioritize the health, safety, and education of children and their family members
9. Increased awareness of social adversity in the form of racism and descrimination and a shared community commitment to addressing historical trauma and injustice
10. Ongoing cross-county sharing of successes and challenges within initiatives to strengthen the statewide network of change agents focused on ensuring vital services

Making the long-term commitment to our children

It is not uncommon to hear that New Mexico has broken systems. We believe a more accurate statement about systems in all fifty states is that systems were never built in the first place. What the initiative is working toward is the design of a countywide system of empowerment that includes timely access to ten vital services, a process never taken on anywhere in the nation… until now.

We believe this groundbreaking initiative is worth watching very closely, with robust evaluation to identify meaningful progress that means an eight-year-old’s home, school, and community environment is well-resourced and designed to ensure a trauma-free childhood. We also can track how safe childhoods, starting with a newborn and mother returning home from the maternity ward, can lead to positive outcomes in the future as children, students, and adults. We can, by ensuring access to ten vital services through a local collaborative process, break the generational cycle of ACEs and a host of related challenges.

Are we there yet?

Each county will “arrive” at our 100% New Mexico long-term goals and desired outcomes when ten vital services exist for every resident with medical care, mental health care, food security programs, affordable housing security programs, and transportation empower a mom with her newborn from day one. As the newborn grows, vital services that need to be available include parent supports, education, home visitation programs, early childhood learning programs, fully-resourced schools with health centers, mentorship programs, and job training. We can evaluate progress made within a county’s borders toward ensuring ten vital services within each community, both rural and urban.

The ultimate measurable outcome in each county will be healthy children, self-sufficient families, students empowered to be job ready, engaged and altruistic communities, and local policies and institutionalized funding streams that support a countywide system of services that promote a culture of caring where all residents thrive.

100% New Mexico is a groundbreaking and comprehensive strategy – a bold “build back better” plan. It is not an initiative that can afford to fail. To be clear, in the urgent quest to protect all of New Mexico’s children, there is no PLAN B.

Please visit our Evaluating Progress page to review our latest initiative evaluation reports.

Note: We are grateful for the input of our colleague Eric Martin, Managing Director at Adaptive Change Advisors and author of Your Leadership Moment, for his contribution to this article. We also are grateful for the insights of the evaluation team at Chapin Hall at the University of Chicago.