QUICK ESCAPE FROM SITE

Conducting evaluation

Keep in mind that evaluations should be based on operational and theoretical frameworks and that they should be incorporated in a programme’s planning stages. Baseline and situation analyses are critical to monitoring and evaluation efforts, but rarely conducted. Please refer to the introductory section for additional information on developing an appropriate framework and collecting baseline data. (Bott, Guedes and Claramunt, 2004)  

 

Define a clear programmatic goal for the intervention - A goal reflects the basic, very broad, conceptual aim of the project and the desired long-term outcome.

Examples of possible goals include:

1) To improve the quality of care that survivors of gender-based violence receive in health care settings.

2) To strengthen the ability of the health care sector to prevent gender-based violence.

 

Keeping this overall goal in mind, identify clear objectives and expected results.

  • Remember to keep in mind the difference between proposed activities, outputs and outcomes, between what will be undertaken, what will be produced and what is expected will happen as a result. For example:
  • Activities might include conducting training for health service providers or developing standardized protocols for responding to cases of sexual violence.
  • Outputs might include the number or percentage of health service providers in a target area who have been trained or the number of health care facilities that have adopted standardized protocols for responding to cases of sexual violence.
  • Outcomes might include strengthened capacity on the part of health service providers to respond to violence against women in a meaningful, appropriate manner or an integrated response on the part of the health care facility following standardized protocols.

Develop indicators for measuring each of the objectives

  • Remember to distinguish between process and results indicators. Health programmes usually collect data on processes rather than on results or outcomes and may not focus on whether their activities were beneficial or effective. This does not mean however, that monitoring and evaluation frameworks should exclude process indicators.
  • Process Indicators are used to monitor the number and types of activities carried out, such as the number and types of services provided, number of people trained, number of materials produced and disseminated or number and percentage of clients screened.
  • Results Indicators are used to evaluate whether or not the activity achieved the intended objectives. Examples include indicators of providers’ or community level knowledge, attitudes and practices as measured by a survey, women’s perceptions about the quality and benefits of services provided by an organization or institution as measured by individual interviews, women’s experiences with health care, and  the appropriateness or readiness of health unit capacity and infrastructure. (Bott, Guedes and Claramunt, 2004)

 

Examples of strategies, objectives and indicators for monitoring and evaluation of health sector initiatives

Strategy/ intervention

Examples of possible objectives

Example indicators

1)   Dissemination of materials/ information

Raising health care providers’ awareness and understanding of gender-based violence, in particular:

a)    GBV as a critical human rights and public health issue b)    barriers women living with violence or survivors of violence face when accessing services c)    links between GBV and HIV and AIDS d)    laws addressing GBV and providers’ responsibilities

  • proportion of providers who identify GBV as a critical human rights and public health issue

  • proportion of providers who can identify and discuss (# of) barriers women face

  • proportion of providers who are able to identify (# of) links between GBV and HIV/AIDS

  • proportion of providers who correctly outline legal obligations with regard to gender-based violence

 

2)    Training of service providers

Strengthening health care providers’ ability to respond to cases of gender-based violence [in particular…]

a)    following appropriate routine screening protocols b)    responding to cases of rape and sexual violence c)    addressing GBV and HIV/AIDS links holistically d)    establishing and using community-based referral networks of care providers and social services e)    improving medico-legal documentation of cases f)     changing stigmatizing norms and attitudes g)    providing emergency and crisis care

 

Strengthening health care providers’ ability to prevent possible gender-based violence through:

a)    changing stigmatizing norms and attitudes b)    strengthening capacity to screen for possible violence, provide appropriate care and referrals as necessary

  • proportion of providers who understand and use appropriate screening protocols

  • proportion of providers who can provide appropriate care for survivors of rape and sexual violence

  • proportion of providers who address links between GBV and HIV in care

  • proportion of providers who are trained to identify, refer and care for survivors

  • proportion of providers who have made referrals for survivors

  • proportion of providers who feel comfortable asking about violence

  • proportion of providers who demonstrate appropriate practices and attitudes with respect to gender-based violence

  • proportion of women accessing services who indicate they received appropriate, comprehensive care

  • proportion of rape/ sexual violence survivors who indicate they received appropriate, comprehensive care

  • proportion of rape/ sexual violence survivors who indicate they received appropriate, comprehensive care

  • proportion of GBV survivors who indicate they received appropriate, comprehensive care

3)    Development of protocols and norms for managing GBV cases

  • Establishing system-wide protocols and norms

  • Improving implementation of system wide protocols, policies and  norms for managing GBV cases

  • Improving clinic infrastructure to provide safe, confidential spaces for consultations

  • Strengthening multisectoral collaboration with other community-based services as part of routine protocols

    • proportion of health units that have documented and adopted a protocol for the clinical management of GBV

    • proportion of health units that have done a readiness assessment for the delivery of GBV services

    • proportion of health units that have commodities for the clinical management of VAW

    • proportion of health units with at least one provider trained to care for and refer GBV cases

 
  • proportion of health units that have documented and adopted a protocol for the clinical management of GBV

  • proportion of health units that have done a readiness assessment for the delivery of GBV services

  • proportion of health units that have commodities for the clinical management of VAW

  • proportion of health units with at least one provider trained to care for and refer GBV cases

4)    Routine screening

  • Increasing levels of screening, detection and referrals

  • Making it easier for women who have experienced or live with violence to share their experiences

  • Strengthening the ability of health care providers to accurately diagnose and care for their patients

  • Improving and ensuring quality of care during screening

  • proportion of women who report physical and/or sexual violence

  • proportion of women who were asked about physical and/or sexual violence during a visit to a health unit

  • proportion of women screened and referred in accordance with clinic policies

  • percentage of women who report that the screening was done in private – not during the clinical exam – and in a sensitive and respectful manner

  • proportion of providers who are able to demonstrate the ability to screen and respond to disclosure adequately during a role play

  • percentage of women who would feel comfortable disclosing and discussing violence in their lives with the provider

 
  • proportion of women who report physical and/or sexual violence

  • proportion of women who were asked about physical and/or sexual violence during a visit to a health unit

  • proportion of women screened and referred in accordance with clinic policies

  • percentage of women who report that the screening was done in private – not during the clinical exam – and in a sensitive and respectful manner

  • proportion of providers who are able to demonstrate the ability to screen and respond to disclosure adequately during a role play

  • percentage of women who would feel comfortable disclosing and discussing violence in their lives with the provider

5)    Campaigns to empower women

  • Increasing women’s knowledge about possible sources of help for gender-based violence

  • Increase women’s sense of empowerment with respect to receiving appropriate care for gender-based violence

  • percentage of women who could identify organizations and resources for care and assistance for gender-based violence

  • percentage of women who articulate that gender-based violence is a health and human rights issue that health  care providers should be addressing

 
  • percentage of women who could identify organizations and resources for care and assistance for gender-based violence

  • percentage of women who articulate that gender-based violence is a health and human rights issue that health  care providers should be addressing

Source: PATH, 2010.

 

 

Case Study: International Planned Parenthood Western Hemisphere Region Evaluation to Improve the Health Sector Response to Gender-Based Violence

The evaluation included four main components.

1. A baseline evaluation study including:

A knowledge, attitudes and practices survey of providers using face to face interviews. IPPF/WHR designed a survey questionnaire to gather information on health care providers’ knowledge, attitudes, and practices related to gender-based violence. The questionnaire contains approximately 80 questions. Although the questionnaire includes a few open-ended questions, most of the questions are closed-ended so that the results can be tabulated and analyzed more easily. The questionnaire covers a range of topics, including: whether, how often and when providers have discussed violence with clients; what providers think are the barriers to screening; what providers do when they discover that a client has experienced violence; attitudes toward women who experience violence; knowledge about the consequences of gender-based violence; and what types of training providers have received in the past. This questionnaire can also be adapted to evaluate a single training. One possibility is to use all or part of the questionnaire before the workshop begins and use only part of the questionnaire after the workshop is over. If the questionnaire is used immediately “before and after” a single training, the organization may be able to measure changes in knowledge, but changes in attitudes and practices usually take time.

A clinic observation/ interview guide The Clinic Observation/Interview Guide gathers information on the human, physical, and written resources available in a clinic. The first half of the guide consists of an interview with a small group of staff members (for example, the clinic director, a doctor, and a counselor). This section includes questions about the clinic’s human resources; written protocols related to gender-based violence screening, care, and referral systems; and other resources, such as whether or not the clinic offers emergency contraception. Whenever possible, the guide instructs the interviewer to ask to see a copy or example of the item in order to confirm that the material exists and is available at the clinic. The second part of this guide involves an observation of the physical infrastructure and operations of the clinic, including privacy in consultation areas (for example, whether clients can be seen or heard from outside), as well as the availability of informational materials on issues related to gender-based violence.

2. Service statistics on detection rates and services provided using standardized screening questions and indicators.

Sample tables for gathering screening data. To ensure that all three participating associations could collect comparable screening data, IPPF/WHR developed a series of model tables, which each association completed every six months. These tables may or may not be useful for other health programmes, as this depends on whether or not the health programme decides to implement routine screening, what kind of policy it adopts, what kind of questions it asks, and what kind of information system it has. Nevertheless, these tables illustrate the types of data that can be collected and analyzed on a routine basis.

3. A midterm, primarily qualitative, evaluation including:

Focus group discussions and in-depth interviews with providers, survivors and external stakeholders/key informants: A summary protocol for collecting qualitative data describes these methods, including in-depth interviews and group discussions; and also provides an idea of what types of providers, clients and other stakeholders were asked to participate.

Client satisfaction surveys: The Client Exit Survey Questionnaire is a standard survey instrument for gathering information about clients’ opinions of the services they have received. This survey is primarily designed for health services that have implemented a routine screening policy. It is important to note that exit surveys tend to have a significant limitation: many clients do not want to share negative views of the services, especially when the interview is conducted at the health center. IPPF/WHR was not able to interview clients offsite, but it did arrange for all the interviewers to be from outside the organization, so that they could reassure the women who participated that they were not going to breach their confidentiality. This questionnaire contains mostly closed-ended questions about the services. It asks women whether they were asked about gender-based violence and about how they felt answering those questions; however, the questionnaire does not ask women to disclose whether or not they have experienced violence themselves.

Case studies of pilot strategies to address various aspects of gender-based violence.

4. A final evaluation serving as a follow up to the baseline including:
  • KAP survey of providers using face to face interviews
  • A clinic observation/ interview guide
  • Random records reviews and development of a protocol:

Throughout the course of the IPPF/WHR regional initiative, the participating associations gathered routine service statistics about clients, including the numbers and percentages of clients who said yes to screening questions.  However, the quality of these service statistics depends on the reliability of the information systems and the willingness of health care providers to comply with clinic policies—both of which may vary from clinic to clinic. IPPF/WHR therefore designed a protocol to measure screening levels and documentation using a random record review approach. This manual contains a brief description of the protocol as well as a tabulation sheet.

Download the main publications related to this initiative:

Basta! The Health Sector Addresses Gender-Based Violence. Available in English and Spanish.

Improving the Health Sector Response to Gender-Based Violence. Available in English and Spanish.

Source: Bott, Guedes and Claramunt 2004.