A Novel Approach for Including Effect Modifiers in Policy Evaluation – a Case of Gender Equality in Higher Education

Higher education suffers from gender imbalance and some are turning towards interventions such as policy changes as a means to create gender equality. Following such changes, it's important to evaluate and understand the outcomes. However, such institutions do not exist in a vacuum and are subject to the general trends and changes in society. We call events and changes outside the policy changes that influence the studied outcome effect modifiers. Alas, any evaluation should seek to identify effect modifiers and determine their impact. Effect modifiers may amplify the effect of the intervention causing false claims that the intervention solved the targeted problem. Effect modifiers may also dampen the effects of the intervention and lead to false conclusions regarding lack of effect from the intervention.

This presentation reports findings from a North American university gender equality in higher education. Many activities were initiated at different levels of the university over an extended period of time. During the intervention period, societal changes also took place which may have also influenced developments in equality at the university. This raises questions of the effect of the intervention and the possible influence of effect modifiers.

Evaluations are often approached by developing a logic model of the intervention and then comparing the logic model to the events that have unfolded. This theory-based evaluation allows evaluators to assess if the mechanisms of the logic model have been followed and hence to determine if the outcome can be attributed to the intervention. Some evaluation methods emphasize the context in which the intervention takes place and ask what works for whom and where. Here the method integrates context into the logic model by focusing on three elements: context, mechanism and outcome. Use of logic models provides the advantage of focusing on mechanisms but introduces a problem of handling contextual influences. As context is included into the logic model the possible contextual factors are developed apriori thereby limiting the scope. We present a novel approach - not relying on a logic model - to identify effect modifiers in policy interventions and assess their effect on the intervention outcome.

The model relies on group interviews where respondents are asked to recall events during the intervention period and analyze if these are part of the intervention or not i.e. contextual factors. The events are then grouped into themes and their impact on the intervention outcome. Testing the method at a North American university we were able to identify themes relating to both intervention and context that have influenced the outcome of the intervention. In particular, the method allows us to capture themes not envisioned during the design of the evaluation and hence allows a more nuanced perspective on the effects of the intervention. In turn, this reduces the risk of evaluators drawing faulty regarding the efficacy of the intervention. Including contextual analysis in evaluation is normally resource-consuming but this approach is both time and cost effective and we propose that this or similar are included in intervention evaluation to avoid false conclusions.