Brief overview of monitoring and evaluation of community mobilization initiatives
Key lessons learned about community-level evaluations
The lessons below come from the experience of evaluating Stepping Stones in South Africa and are generally applicable to most community-level evaluations.
MEASURE Evaluation, at the request of The United States Agency for International Development and in collaboration with the Inter-agency Gender Working Group, compiled a set of indicators for community mobilization. The indicators have been designed to measure programme performance and achievement at the community, regional and national levels using quantitative methods. Note, that while many of the indicators have been used in the field, they have not necessarily been tested in multiple settings. To review the indicators comprehensively, including their definitions; the tool that should be used and instructions on how to go about it, see the publication Violence Against Women and Girls: A Compendium of Monitoring and Evaluation Indicators.
The compiled indicators for community mobilization are:
What It Measures: Knowledge of women’s constitutional and legal rights remains low particularly among women in many countries. Given the situation, human rights and women’s rights education programs are implemented in communities in a number of countries. Program managers and evaluators may be interested in knowing the extent to which men and women are aware of the constitutional and legal rights of women. This outcome indicator measures the extent to which the public are aware of such rights.
What It Measures: Knowledge of the legal sanctions associated with VAW/G is low, particularly among women in many countries. If women do not know of any legal recourse, they may not seek help, thinking that nothing can be done. Program managers and evaluators may be interested in knowing the extent to which men and women are aware of the legal sanctions for acts of VAW/G that exist in a country. This outcome indicator measures the extent to which the public are aware of such sanctions.
What It Measures: In countries (or regions within countries) where communication programs related to VAW/G prevention are implemented using IEC techniques, interpersonal communication channels and community outreach workers, program managers and evaluators may need to know the extent to which the intended audience is exposed to the communication programs. This outcome indicator measures the extent to which a population targeted by specific VAW/G programs and projects is exposed to VAW/G prevention messages through any means that they might be communicated.
What It Measures: This outcome indicator measures the level of acceptability of wife-beating in an area (region, country, community) for any reason, at the point in time that it is measured. A high proportion would indicate that most people in the targeted population feel that wife beating is acceptable under certain conditions.
What It Measures: Neighbors are often aware when a woman is being beaten by her husband because they can hear the incidents. Relatives (living outside the household) and friends are often told about these incidents by women. People’s willingness to extend help to the woman may save the woman from the pain, disability and death associated with IPV. A low proportion of individuals stating that they are wiling to help may indicate a general acceptance of IPV in the community surveyed. Women living in such communities are at higher risk for the consequences associated with IPV, and they may have little recourse.
What It Measures: People who feel that men cannot control their sexual behavior (i.e., that the reason they act sexually is because of a factor outside of themselves) also feel that on some level, men cannot be held responsible for what they do sexually. This places the blame for violent sexual behavior on the victim instead of on the perpetrator.
What It Measures: Cultural norms around the authority of husbands over wives may include that he has a right to have sex with his wife or partner regardless of whether or not she wants it. Alternately, there may be conditions under which it is acceptable (or not acceptable) for her to refuse. This outcome indicator measures how acceptable the idea of a woman refusing sex with her husband or partner is, and under which reasons people feel this is acceptable. In areas where there are no acceptable reasons or very low proportions of people agreeing with the reasons, women’s power of sexual negotiation may be very low and could leave them at risk for violence and exposure to sexually transmitted infections including HIV.
What It Measures: Cultural norms around the authority of husbands over wives may include forced sexual intercourse. Marital rape may be more likely to occur in places where both women and men ascribe to this belief. This outcome indicator measures how unacceptable marital rape is in the targeted population. In areas where the proportion is very low, women may be at high risk for marital rape.
What It Measures: In countries (or regions within countries) where communication programs related to the elimination of FGC/M are implemented using mass media, program managers and evaluators may need to know the extent to which the intended audience is exposed to the communication messages. This outcome indicator measures the extent to which the public (or population targeted by specific programs and projects) remembers seeing or hearing FGC/M elimination messages through various communication channels after exposure.
What It Measures: The practice of FGC/M is deep-rooted in culture, and is supported by beliefs that are in favor of the practice. International organizations, however, recognize the practice as a violation of international standards for girls’ and women’s rights, and that the practice has serious health consequences. Communication campaigns and community education programs are implemented in countries where the practice is prevalent to discourage support for FGC/M. This outcome indicator measures the level of public acceptance of FGC/M within a given population.
What It Measures: The practice of FGC/M is deep-rooted in culture, and is supported by beliefs that are in favor of the practice. International organizations, however, recognize the practice as a violation of international standards for girls’ and women’s rights, and that the practice has serious health consequences. Programs that aim to eliminate the practice of FGC/M will want a measure of women’s intentions regarding their own daughters. This outcome indicator provides a measure of the effectiveness of programs and initiatives that aim to reduce the practice of FGC/M.
What It Measures: The practice of child marriage is deep-rooted in culture and is supported by beliefs and customs transmitted through the generations. International organizations, however, recognize the practice as a violation of international standards for girls’ and women’s rights, and that the practice can result in serious emotional and physical health consequences. Communication campaigns and community education programs are implemented in countries where the practice is prevalent to discourage support for child marriage. This outcome indicator measures the level of public acceptance of child marriage within a given population.
What It Measures: The practice of child marriage is deep-rooted in culture and is supported by beliefs and customs transmitted through the generations. International organizations, however, recognize the practice as a violation of international standards for girls’ and women’s rights, and that the practice can result in serious emotional and physical health consequences. Communication campaigns and community education programs are implemented in countries where the practice is prevalent to discourage support for child marriage. This outcome indicator provides a measure of the effectiveness of programs and initiatives that aim to reduce the practice of child marriage.
For additional indicators on working with youth and working with men and boys, see the full compendium.
Programme descriptions and indicators for select community-based initiatives:
Monitoring and evaluation methods (including indicators) will vary depending on the programme’s goal and objectives. The descriptions and indicators presented below are examples that come from some of the most promising community-based initiatives that have demonstrated promise in reducing violence against women and girls.
The Rural AIDS Development Action Research (RADAR) Programme Intervention with Microfinance for AIDS and Gender Equity in South Africa (Pronyk et al 2006)
The programme Intervention with Microfinance for AIDS and Gender Equity (IMAGE) study in South Africa-- one of the few rigorous evaluations of community-level interventions globally related to gender-based violence-- showed that an intervention combining micro-financing to women with education and mobilization around violence reduced women’s experiences of violence by 55% over a two-year period.
The IMAGE study used a cluster randomized trial design to test the hypothesis that combining a microfinance-based poverty alleviation programme with participatory training on HIV risk and prevention, gender norms, domestic violence, and sexuality can improve economic well-being, empower women, and lead to reductions in inter-personal violence. The study was conducted between September 2001 and March 2005 in South Africa’s rural Limpopo province.
Outcome measures included past year’s experiences of intimate-partner violence drawn from the World Health Organization’s Violence against Women study instrument and nine indicators of women’s empowerment measured around three domains: 1) power within; 2) power to; and 3) power with. The nine indicators they used to measure these three areas of empowerment were:
In addition to the main outcome of reduced intimate partner violence, two other outcome indicators were measured because of their correlation to experiences of violence:
Qualitative data about changes occurring within intimate-relationships and the community were also collected through facilitated focus group discussions, contributing significantly to understanding the nuances behind the numbers. For example, discussions revealed that there was no equivalent for the term “empowerment.” Rather, women used phrases such as “the ability to claim personal power and use it to change for the better,” noting that “you can have money and still not be empowered.” Qualitative responses also highlighted that reductions in violence resulted from a range of responses enabling women to challenge the acceptability of such violence, expect and receive better treatment from partners, leave violent relationships, give material and moral support to those experiencing abuse, mobilize new and existing community groups and raise awareness about gender-based violence and HIV. [Pronyk et al 2006]
To view the IMAGE evaluation methodology, intervention materials, questionnaires (baseline and follow-up) for young people, women and households, and research reports visit the University of Witwatersrand, Johannesburg, Intervention with Microfinance for AIDS & Gender Equity website.
Stepping Stones, South Africa
Stepping Stones is a community-based intervention aimed at preventing HIV through more gender-equitable and more communicative relationships. Started by Alice Wellbourne in Uganda, the programme has been adapted for over 17 settings and translated to at least 13 languages for use in over 40 countries (Jewkes, Nduna, Levin, Jama, Dunkle, Wood, Koss, Puren and Duvvury, 2007). Though the programme did not explicitly set out to reduce the incidence of intimate partner violence, an evaluation of the programme in South Africa demonstrated that the programme did reduce violence against women.
In South Africa, Stepping Stones used a randomized control trial to evaluate the effectiveness of this well known educational tool for preventing violence and HIV. A cluster randomized-controlled trial (RCT) was conducted in 70 villages in rural South Africa to evaluate the impact of Stepping Stones, a behavioural intervention implemented in 35 communities in two workshops of 20 men and 20 women in each community. Individuals in control communities attended a single session on HIV and safer sex.
The impact was assessed through two questionnaire surveys at 12-month intervals. The primary outcome was HIV incidence. Secondary outcomes included changes in knowledge, attitude and sexual behaviours, including on gender-based violence. Qualitative research was conducted with 10 men and 10 women from two sites receiving the intervention (one rural and one urban) and five men and five women from one village in the control arm. They were interviewed individually three times prior to the workshops and then 9–12 months later.
This is the third randomized-controlled trial to be conducted in sub-Saharan Africa evaluating a behavioural intervention using HIV incidence as a primary outcome. It is of particular interest as the Stepping Stones intervention is used in many developing countries. There is good baseline comparability between the study groups, and the process (monitoring) data on the workshops suggested that the interventions were feasible and adequately implemented.
The areas of change that are measured (at baseline, 12 months and 24 months) in the Stepping Stones evaluation relate to these main areas:
The section on violence includes general questions and questions related specifically to emotional abuse, physical abuse, sexual abuse and abuse by a non-partner.
See the Stepping Stones website for more information on the programme, adaptations of the programme, resources and evaluations.
For a summary of the evaluation in South Africa, see the policy brief by Medical Research Council.
SASA!, East Africa
SASA! is a methodology developed by Raising Voices for addressing the link between violence against women and HIV/AIDS. Documented in a user-friendly Activist Kit, it is meant to inspire, enable and structure effective community mobilization to prevent violence against women and HIV/AIDS.
SASA! uses four strategies: local activism, media and advocacy, communication materials and training to reach a variety of people in a variety of ways and includes a variety of monitoring tools to help organizations assess and reflect on their efforts in the community. The tools, designed for activist organizations are simple yet provide meaningful information to feedback into programme design to make SASA! implementation stronger and more effective.
The SASA! methodology suggests that change happens in stages, and starts with awareness, preparation for action, implementation and then maintenance of change. They have therefore developed a set of indicators for each stage of the strategy, that measure changes in:
Each of these dimensions would be measured periodically, for example, every 6 months, using simple tools such as a quick survey of activists and community members, and sessions to reflect on the progress of the work. In addition, the programme has set up a monitoring system whereby NGO staff provide feedback and support to community activists, and at the same time document changes in the way people are talking about the issues of the programme (for example, do most people think that women deserve to be beaten or not?)
These tools do not provide data that can be used to measure impact, but they do provide powerful and meaningful information to community activists and programme staff, and complement the findings of more rigorous studies, such as the SASA! impact study.
In 2007, Raising Voices, the Center for Domestic Violence Prevention (CEDOVIP), the Gender Violence and Health Centre at the London School of Hygiene and Tropical Medicine, Makerere University and PATH collaborated to design and carry out a study to evaluate the effectiveness of the SASA! community mobilization initiative in Uganda. This study will provide evidence about the potential role and impact of the SASA! approach in addressing gender inequality, violence against women and HIV/AIDS – and is one of the few rigorous impact evaluations being conducted of community mobilization efforts focusing on violence and HIV.
The specific aims of the study are to:
What is particularly useful about this study is that, while the major aim is to measure the impact of SASA! using rigorous methods, it also aims to develop tools that can be used by other groups to plan, monitor, and assess their own community interventions.
To assess the impact of SASA!, surveys measuring knowledge, attitudes, skills and behaviors of men and women in the community are being administered in sites where the programme is being implemented as well as comparable control communities that are not receiving the intervention – at baseline before implementation, and again, three years after its initiation. In addition to quantitative research, complementary qualitative research is exploring how the dynamics of relationships, violence against women and HIV behaviors may change over time, including the impact the intervention on community activists, leaders and service providers. Focus group discussions shed light on community responses to the programme and experiences with promoting non-violent relationships. The qualitative data collection is being conducted twice yearly, and includes focus group discussions and in-depth interviews with community volunteers, local leaders, police and health workers.
Findings from the 2008 baseline study and focus group discussions highlighted high levels of violence and deep seated harmful gender norms. Almost half of currently or ever married women reported having experienced physical or sexual violence from a partner and half of the men reported using one or more forms of controlling behavior with their partners in the past year.
See the SASA! section on the Raising Voices website for more information on the programme and on the evaluation process.
Gender-equitable men (GEM) scale to assess norms and behaviours among young men in Brazil and India
Few interventions to promote gender-equitable norms and behaviours among young men have been systematically implemented or evaluated, and relatively little is known about how best to measure changes in gender norms, violence and their effect on HIV/STI protective and risk behaviours.
To address these gaps, the Horizons Programme and Instituto Promundo through their Programme H examined the effectiveness of interventions designed to improve young men’s attitudes toward gender-equitable norms, including gender-based violence, and to reduce HIV/STI risk.
GEM Evaluation Brazil: Set in Rio de Janeiro in 2001, the study compared the impact of different combinations of programme activities implemented in three different but fairly homogeneous low-income communities. One intervention was interactive group education sessions for young men led by adult male facilitators. The other was a community-wide campaign to promote condom use, using gender-equitable messages that reinforced those promoted in the group education sessions. A group of young men aged 14 to 25 was followed over time in each community.
To assess the impact of the programme, researchers from the Horizons Programme developed and validated the Gender-Equitable Men (GEM) Scale with a representative household sample. The GEM Scale, which measures attitudes toward gender norms, includes items in five keys areas: (1) violence, (2) sexuality and sexual relationships, (3) reproductive health and disease prevention, (4) domestic chores and childcare, and (5) homophobia and relationships with other men. Twenty-four (24) items were selected, 17 of which comprise the “inequitable” gender norms subscale, addressing norms that have been considered more “traditional,” and 7 which comprise the “equitable” subscale. Responses were categorized and scored as “least equitable,” “moderately equitable,” and “most equitable,” and collated for an overall score.
Changes in attitudes over time were also analyzed in order to correlate them with other outcomes. A variable was created to reflect changes in respondents’ GEM scores. If a respondent’s score increased after the intervention, it was reflected as a positive change, if it stayed the same or decreased, it was reflected as no or negative change. (Pulerwitz, Barker, Segundo and Nascimento. 2006.)
See the evaluation summary, available in English.
GEM Evaluation India: In India, the Horizons Programme, CORO for Literacy, MAMTA, and Instituto Promundo developed and piloted a behaviour-change intervention, Yaari-Dosti, based on Programme H implemented in Brazil. The team conducted research to examine the effectiveness of the interventions to improve young men’s attitudes toward gender roles and sexual relationships, and to reduce HIV risk behaviours and partner violence. Set in urban areas of Mumbai, the study tested the impact of different combinations of intervention activities on young men’s support for inequitable gender norms, HIV/STI risk behaviours, and partner violence.
Attitudes toward gender norms of the young men were assessed using a version of the Gender Equitable Men (GEM) Scale adapted to the Indian context with the addition of specific items, for example, “A man is happily married only if his wife brings a big dowry,” and, “A woman should not need to ask her husband for permission to visit her parents/family.” (Verma et al 2008)
See the evaluation summary, available in English.
International Men and Gender Equality Survey (IMAGES)
Instituto Promundo and the International Center on Research and Women developed the International Men and Gender Equality Survey (IMAGES) to strengthen understanding of men’s behaviours and attitudes, and changes in those attitudes and behaviours. The ultimate goal of the survey is to inform, drive and monitor policy development to promote gender equality by engaging men and women in such policies.
IMAGES is one of the most comprehensive surveys developed on this topic and has sought to incorporate the most recent instruments on gender, quality of life, childhood antecedents of violence, health indicators, gender-based violence, family gender dynamics, and fatherhood. In some countries, IMAGES is serving as a baseline instrument to assess men’s and women’s attitudes and behaviours on these key issues before the implementation of major policy and programme initiatives.
The specific objectives of the survey are to:
Additional Resources for Evaluation:
Measures for the Assessment of Dimensions of Violence against Women: A Compendium (Flood, 2008). Available in English.
Soul City 4 Evaluation Media Campaign Monitoring and Evaluation Materials. Available in English.
SASA! Tips Booklet - Strengthening Your Activism: Skills for Preventing Violence against Women and HIV Infection (Raising Voices, 2009). Available in English.
Good Schools Toolkit (Raising Voices, 2009). Available in English.
Incorporating Evaluation into Media Campaign Design (Institute for Health and Development Communication, South Africa). Available in English.
Lessons in Evaluating Communications Campaigns: Five Case Studies (Communications Consortium Media Center/ Harvard Family Research Project, 2003). Available in English.