Questions
Over the last couple of years, I've been working with one of the most compelling organizations in Canada, Intercordia. The organization is a thoughtful off-shoot of the international L'Arche movement, steeped in the philosophy and learnings of L'Arche founder, Jean Vanier. Intercordia is a radical model of service learning/study abroad programming for Canadian university students. Among many areas of engagement, the folks at Intercordia came to me to help them answer the question, "How do we evaluate transformation of the heart?"
I must admit that I am not an evaluation guru, though I've been forced over the years to be thoroughly engaged in the practice of program outcome evaluation (thank you, United Way). Sure, I took methods courses in graduate school and know my way around a randomized trial. What I am known for is exploring and applying creative solutions in spaces with limited resources. You might say I can turn an egg and a beer into a champagne brunch.
Over the last couple of years, I've been working with one of the most compelling organizations in Canada, Intercordia. The organization is a thoughtful off-shoot of the international L'Arche movement, steeped in the philosophy and learnings of L'Arche founder, Jean Vanier. Intercordia is a radical model of service learning/study abroad programming for Canadian university students. Among many areas of engagement, the folks at Intercordia came to me to help them answer the question, "How do we evaluate transformation of the heart?"
I must admit that I am not an evaluation guru, though I've been forced over the years to be thoroughly engaged in the practice of program outcome evaluation (thank you, United Way). Sure, I took methods courses in graduate school and know my way around a randomized trial. What I am known for is exploring and applying creative solutions in spaces with limited resources. You might say I can turn an egg and a beer into a champagne brunch.
Because Intercordia has a unique model, it's important to lay it out before we move on to the question of evaluation. To quote Intercordia's Web site: "The goal of this innovative learning program is to encourage moral
responsiveness, develop respect for diversity and a valuing of other
cultures, religions and socio-economic backgrounds that will enable
Canadian students to attain a well-educated solidarity with others who
are different." Through this type of inner-transformation, Intercordia enhances the capacity of young Canadians to thoughtfully engage in their own communities, now and in the future. Sounds great, eh? So what does the program actually look like?
Students enrolled in the program take an academic course designed by their university to provide them with critical tools and lenses to engage and begin to understand an unjust world. Simultaneously, students participate in seminars that cover everything from the practicalities of travel in the majority world to Jean Vanier's philosophy concerning encounters with the "other." Students then spend 3 months living and working in community with marginalized people in Ghana, Swaziland, Bosnia, Ukraine, Ecuador, Honduras, Nicaragua, Dominican Republic or Brazil. Sounds like any other service learning/study abroad program, eh? Well, the key difference is the radical formation and re-integration that the students experience.
Intercordia encourages students away from the need/desire to DO, which is a key characteristic of most service learning programs. Intercordia helps students recognize that at this juncture in their lives, the only thing that they really have to give is their time, presence and openness to learning. Sure, they have volunteer jobs while they are in placement, but the main focus is on BEING in community with the people. Intercordia students live in often extreme circumstances to learn about people, community and, dare I say it? love. The most important thing the students experience is the formation of honest relationships with people. The breaking down of barriers and stereotypes in this way encourages future moral responsiveness in the face of widespread injustice.
The experience isn't controlled in the ways that most service learning/study abroad programs are. Instead, Intercordia relies on the wisdom of partnering grassroots organizations to trouble-shoot, work with host families and community employers and set the terms by which the students are present in the communities.
Challenges
The key challenge Intercordia faces in evaluating the impact of their program is a common story for innovative organizations and projects - resources. While a UN-funded development project may have the funds and personnel required to engage in a well-rounded, long-term evaluation strategy, Intercordia does not. In addition, evaluation gets a little more tricky because Intercordia's partners differ in their approaches and offerings. For example, student A may be coming from the Christianity and Culture program at one university with certain theoretical frameworks and approaches, while student B comes from the Social Justice program at another university. Then we take student A and send her to rural Ghana to live in community and teach in schools where caning is sanctioned. Maybe we send student B to Honduras to live in community and work with families who live and work in toxic dump sites. While student B is in Honduras, the current president is ousted in a military coup and mass civil unrest ensues. So clearly, we have different approaches, frameworks and emergent challenges to incorporate into an evaluation framework.
Assets
Intercordia has a true culture of innovation. From the Board and executive leadership all the way to part-time program staff, there is a history of deep-listening and learning that translates into the ability to innovate, or to engage in what I call some honest-to-goodness trial and error. The program itself is small and nimble, with no more than 50 participants in a given program year. An additional asset is one not to be underestimated - Intercordia has some pretty fantastic, innovative funders who believe in the uniqueness of Intercordia's model of transformation. Finally, Intercordia is an organization that loves stories. I remember when they contacted me to have our first meeting. In that session I was asked repeatedly by the Director and Board members to "Tell us a story of a time when..." This last asset is the key to my evaluative approach.
Actions
In 2007, I had the privilege of joining a 2-year working group convened by the Millennium Scholarship Foundation and the Centre for Social Innovation to think about and document strategies for evaluating networks. It was through this group that I met Jamie Gamble from Imprint, Inc. Jamie had recently authored "A Development Evaluation Primer" with the support of the J.W. McConnell Family Foundation (also one of Intercordia's funders). Prior to the working group, I was already smitten with the possibilities for Development Evaluation (DE). It was through my exploration of DE that I came to know an evaluation/monitoring framework often used in sustainable international development projects known as "Most Significant Change (MSC)."
According to the framework's primary authors, Rick Davies and Jess Dart,
In the case of Intercordia we ask the participants a series of open-ended questions while they are in placement. The questions are designed in collaboration with the university partners to encourage personal reflection, the telling of personal stories, within a critical framework. Students respond to the questions in a password-protected "member zone" of the Intercordia Web site. The responses are monitored by Intercordia staff as an additional risk-management layer (looking for "red flags" in the experience of the student). The responses to the questions become the focus of the reintegration seminar - a way of highlighting the common threads of experience for all of the participants and deepening their understanding of their stories in the larger context of the placements. The students are the engaged stakeholders who review the stories with each other and staff, discussing the experiences and reported impacts. These selected stories are then passed on to the broader community of stakeholders (Board, funders, etc.) for review and discussion. The responses to the questions offer a way to assess the extent to which Intercordia's desired outcomes are met. The MSC framework allows real learning of the positive, negative and unintended impacts of the Intercordia program on the lives of the participants, and as Intercordia has the resources to expand the framework, on the participating partners and communities.
I must be clear that due to limited resources and numerous areas of inquiry, the piloting of the MSC program evaluation/monitoring framework was not intended to be a comprehensive program evaluation strategy, but rather one framework that builds on the current culture and systems of the organization to engage in deep-listening concerning the program impacts. It also enables the participants to deepen their reflections while in placement and beyond - which is a key component of inner transformation and action beyond the boundaries of the program.
Learnings
I will admit that what I find so amazing about this project are the MSC stories that emerge from the program participants. It's hard to maintain objective distance when the stories make me laugh, cry and bring me closer to the struggles and joys the students experience in placement. Below I've highlighted a handful of key learnings in the refinement of the MSC process.
*Stay tuned for updates on the expansion of this compelling framework. If you have any questions or would like to offer suggestions, please feel free to contact me.
Students enrolled in the program take an academic course designed by their university to provide them with critical tools and lenses to engage and begin to understand an unjust world. Simultaneously, students participate in seminars that cover everything from the practicalities of travel in the majority world to Jean Vanier's philosophy concerning encounters with the "other." Students then spend 3 months living and working in community with marginalized people in Ghana, Swaziland, Bosnia, Ukraine, Ecuador, Honduras, Nicaragua, Dominican Republic or Brazil. Sounds like any other service learning/study abroad program, eh? Well, the key difference is the radical formation and re-integration that the students experience.
Intercordia encourages students away from the need/desire to DO, which is a key characteristic of most service learning programs. Intercordia helps students recognize that at this juncture in their lives, the only thing that they really have to give is their time, presence and openness to learning. Sure, they have volunteer jobs while they are in placement, but the main focus is on BEING in community with the people. Intercordia students live in often extreme circumstances to learn about people, community and, dare I say it? love. The most important thing the students experience is the formation of honest relationships with people. The breaking down of barriers and stereotypes in this way encourages future moral responsiveness in the face of widespread injustice.
The experience isn't controlled in the ways that most service learning/study abroad programs are. Instead, Intercordia relies on the wisdom of partnering grassroots organizations to trouble-shoot, work with host families and community employers and set the terms by which the students are present in the communities.
Challenges
The key challenge Intercordia faces in evaluating the impact of their program is a common story for innovative organizations and projects - resources. While a UN-funded development project may have the funds and personnel required to engage in a well-rounded, long-term evaluation strategy, Intercordia does not. In addition, evaluation gets a little more tricky because Intercordia's partners differ in their approaches and offerings. For example, student A may be coming from the Christianity and Culture program at one university with certain theoretical frameworks and approaches, while student B comes from the Social Justice program at another university. Then we take student A and send her to rural Ghana to live in community and teach in schools where caning is sanctioned. Maybe we send student B to Honduras to live in community and work with families who live and work in toxic dump sites. While student B is in Honduras, the current president is ousted in a military coup and mass civil unrest ensues. So clearly, we have different approaches, frameworks and emergent challenges to incorporate into an evaluation framework.
Assets
Intercordia has a true culture of innovation. From the Board and executive leadership all the way to part-time program staff, there is a history of deep-listening and learning that translates into the ability to innovate, or to engage in what I call some honest-to-goodness trial and error. The program itself is small and nimble, with no more than 50 participants in a given program year. An additional asset is one not to be underestimated - Intercordia has some pretty fantastic, innovative funders who believe in the uniqueness of Intercordia's model of transformation. Finally, Intercordia is an organization that loves stories. I remember when they contacted me to have our first meeting. In that session I was asked repeatedly by the Director and Board members to "Tell us a story of a time when..." This last asset is the key to my evaluative approach.
Actions
In 2007, I had the privilege of joining a 2-year working group convened by the Millennium Scholarship Foundation and the Centre for Social Innovation to think about and document strategies for evaluating networks. It was through this group that I met Jamie Gamble from Imprint, Inc. Jamie had recently authored "A Development Evaluation Primer" with the support of the J.W. McConnell Family Foundation (also one of Intercordia's funders). Prior to the working group, I was already smitten with the possibilities for Development Evaluation (DE). It was through my exploration of DE that I came to know an evaluation/monitoring framework often used in sustainable international development projects known as "Most Significant Change (MSC)."
According to the framework's primary authors, Rick Davies and Jess Dart,
"The MSC technique is a form of participatory monitoring and evaluation. Essentially, the process involves the collection of significant change stories emanating from the field level, and the systematic selection of the most significant of these stories by groups of designed stakeholders or staff. The designated staff and stakeholders are initially involved by "searching" for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes."
In the case of Intercordia we ask the participants a series of open-ended questions while they are in placement. The questions are designed in collaboration with the university partners to encourage personal reflection, the telling of personal stories, within a critical framework. Students respond to the questions in a password-protected "member zone" of the Intercordia Web site. The responses are monitored by Intercordia staff as an additional risk-management layer (looking for "red flags" in the experience of the student). The responses to the questions become the focus of the reintegration seminar - a way of highlighting the common threads of experience for all of the participants and deepening their understanding of their stories in the larger context of the placements. The students are the engaged stakeholders who review the stories with each other and staff, discussing the experiences and reported impacts. These selected stories are then passed on to the broader community of stakeholders (Board, funders, etc.) for review and discussion. The responses to the questions offer a way to assess the extent to which Intercordia's desired outcomes are met. The MSC framework allows real learning of the positive, negative and unintended impacts of the Intercordia program on the lives of the participants, and as Intercordia has the resources to expand the framework, on the participating partners and communities.
I must be clear that due to limited resources and numerous areas of inquiry, the piloting of the MSC program evaluation/monitoring framework was not intended to be a comprehensive program evaluation strategy, but rather one framework that builds on the current culture and systems of the organization to engage in deep-listening concerning the program impacts. It also enables the participants to deepen their reflections while in placement and beyond - which is a key component of inner transformation and action beyond the boundaries of the program.
Learnings
I will admit that what I find so amazing about this project are the MSC stories that emerge from the program participants. It's hard to maintain objective distance when the stories make me laugh, cry and bring me closer to the struggles and joys the students experience in placement. Below I've highlighted a handful of key learnings in the refinement of the MSC process.
- Early in the implementation of MSC, I discovered a vibrant MSC community of practice. This group of global practitioners allows me to challenge my assumptions and refine the process.
- During the first year of implementation, it was clear that the MSC reflection process with the students would be greatly enhanced through the presence of a mentor. In year two, we piloted a mentor program that placed mentors in 3 of the placements (Swaziland, Ghana and Dominican Republic) to live and work in community with the students in order to faciliate deeper reflection. The program has proven to be incredibly valuable in supporting students in their MSC reflection. Intercordia plans to expand the pilot mentor program in the coming years.
- It is clear that the MSC technique can be expanded to include the collection of stories from community organizations and family members in the international placements. We are currently crafting a strategy for years 3 and 4 that captures stories from these important stakeholders.
- Several MSC practitioners world-wide have had great success combining the MSC technique with Participatory Video. We believe this marriage is appropriate for Intercordia moving forward as a way to facilitate additional MSC participation in placement communities. We are currently exploring the implementation logistics.
- MSC stories are fantastic marketing tools where appropriate.
*Stay tuned for updates on the expansion of this compelling framework. If you have any questions or would like to offer suggestions, please feel free to contact me.
Leave a comment