What is more important when a researcher wants to convince policymakers: to communicate timely, or to communicate evidence that is certain? To answer this, we should take into account the perspective of policymakers. They often have to come to decisions under time pressure, even when the evidence they have at their disposal is very uncertain. Improving regulation requires the timely communication of relevant evidence, even though the evidence is not always certain.

Using an innovative computational opinion dynamics model, we found that evidence should be presented early to convince policymakers, even if it is still uncertain. We also conclude that evidence may fail to convince everyone, no matter how certain it is.

Studying the users of evidence such as policymakers or politicians is difficult in real-life settings. We thus resorted to opinion dynamics modelling. In the field of social epistemology scholars have developed the Hegselmann–Krause model to study how people in a group discuss and change their minds. It allows to simulate phenomena such as knowledge spread and consensus or opinion polarization within a group. In such a computational agent-based model, agents strive to improve their beliefs by considering the beliefs of other agents. Opinion paths show how individual agents change their belief and converge or polarize over time. Figure 1 shows how the introduction of a new piece of evidence visibly disrupts the opinion path for some agents yet fails to convince all agents.

Figure 1: Opinion paths for a community of 100 agents. The opinion of each agent is indicated as a value between 0 and 1 (on the vertical axis) and is traced as a function of time (horizontally). At time step = 4, a new piece of evidence is introduced.

In general, such an agent-based model allows for manipulating what happens within a group of agents (policymakers, in our case) through changing parameters such as group size and their open-mindedness. To tailor the model to public decision settings, we added three crucial parameters.

1. Urge towards certainty. This builds on the assumption that policymakers will change their opinion for the pragmatic reason of reaching a decision. If they lack decisive evidence, assumptions will start driving the decision process. We manipulated the strength of this urge towards certainty in the simulation.
2. Timing of evidence. We also manipulated how early or late the new evidence is introduced?
3. Uncertainty of the evidence. We tested two cases that vary in level of uncertainty.

We modelled two case studies: intelligence officers sharing (highly uncertain) evidence on potential terrorist activity, and food safety inspectors sharing (less uncertain) evidence on the contamination of chicken meat. For each case, we ran a simulation 500 times for every combination of the first two parameters: the urge toward certainty and the timing of evidence. Our simulation results indicate that evidence should be presented early rather than late.

Here, we take a closer look at the chicken meat case. The evidence is relatively certain: the probability that it correctly shows that the meat is contaminated is 93%. In the heatmap in Figure 2, the horizontal axis stands for the timing of the evidence, from early to late. The vertical axis represents the urge towards certainty: the lower the number, the higher the urge. The colour scale indicates the percentage of the agents who have become convinced at the end of the process that the meat is infected, which is probably correct given the evidence. In dark green rectangles, 100% of the agents were convinced. In lighter green or yellow rectangles, at least a substantial part of the agents was ultimately not convinced by the evidence. Put differently: we see that when evidence is shared later, fewer agents will be convinced by this evidence (hence the increase of yellow rectangles when time passes). The heatmap suggests that the evidence should come very early, even when it is quite certain. 

Figure 2: Heatmap showing simulation results for a case in which the uncertainty of the evidence is relatively low.

Our study demonstrates some distinct advantages of agent-based models for studying evidence-informed policymaking. They allow us to:

• study emergent phenomena at the group level that arise due to interactions between individuals.
• measure the effect of individual parameters that cannot be monitored empirically in isolation due to their complex interactions in reality. For instance, we can measure the specific impact of the size of a group of decision makers (or another variable of interest) while keeping all other attributes fixed.
• manipulate parameters systematically that we cannot manipulate in reality, due to practical or ethical constraints. For instance, we could study heterogenous or homogeneous groups in terms of their initial opinions, open-mindedness, or any other variable of interest.
• formulate new research hypotheses and to test research designs prior to taking them to the field.

In conclusion: if evidence is to be impactful in the policy process, it needs to be communicated at the right time. Based on our simulations, we find that this should be done as early as possible. Simulations such as these can offer many possibilities to support future research into evidence-informed policymaking.

You can read the original research published in Policy Sciences here (and here)

Authors: Wouter Lammers is a doctoral candidate at KU Leuven Public Governance Institute. Valérie Pattyn is associate professor at the Institute of Public Administration of Leiden University. Sacha Ferrari is a doctoral candidate at the Centre for Logic and Philosophy of Science at KU Leuven. Sylvia Wenmackers is a research professor at the Centre for Logic and Philosophy of Science at KU Leuven. Steven Van de Walle is professor of public management at the KU Leuven Public Governance Institute.

Disclaimer: The views and opinions expressed in the blog articles belong solely to the author of the content, and do not necessarily reflect the European Commission's perspectives on the issue.