Facilitating the ideas-informed society: a systematic review

Abstract

Background: The concept of the ideas-informed society represents a desired situation in which: (1) citizens see value in staying up to date, and (2) citizens regularly keep themselves up to date by actively engaging with new ideas, developments and claims to truth, doing so both openly and critically. As a result, individuals become ever more knowledgeable, are better able to make good decisions, as well as find themselves in better position to re-align their values in response to new progressive norms and beliefs. Given these potential benefits, of primary interest are those who do not value staying up to date, nor attempt to do so.

Methods: With this systematic review we have sought to identify ways to consider how such “ideas refusers” might be switched-on to engaging with new ideas. We have done so by exploring: (1) the factors which act as barriers to and enablers of the actualisation of the ideas-informed society; (2) interventions/programmes and community-led activities developed to actualise the ideas-informed society; and (3) other non-empirically tested/verified suggestions for how the ideas-informed society might be actualised. Our findings derive from 25 research outputs (from a total of 631 originally identified) as well as examine case studies of “bottom-up” analogous activities.

Results: Our review highlights the presence of seemingly impactful approaches to enabling citizens to engage with new ideas, including science cafés and museum exhibitions. Other more bottom-up approaches include community-based events and festivals; social networks (and discussion within these networks) are also key to whether and how individuals engage with ideas, and the breadth of ideas they engage with.

Conclusions: We conclude by suggesting development and rigorous testing is now needed of interventions that seek to: (1) pique citizens’ curiosity; (2) establish connections to social networks and; (3) arm citizens with essential ideas-related dispositions.

Citation

Brown, C., Luzmore, R. and Groß Ophoff, J. (2023), "Facilitating the ideas-informed society: a systematic review", Emerald Open Research, Vol. 1 No. 1. https://doi.org/10.1108/EOR-01-2023-0008

Introduction

The concept of the democratic, ideas-informed society has a long-standing history: identifiable at least as far back as 375BCE in Plato’s treatise, The Republic (Plato, 2007). It represents a situation in which citizens regularly keep themselves up to date by actively, openly and critically engaging with new ideas, developments and claims to truth (Brown et al., 2022; Franco et al., 2019). To elucidate on what is meant by these three terms, for us “ideas” are the presentation of new possibilities relating to how we live, work, organise society, interact with others, treat ourselves, or engage with the natural world. The term “developments” encapsulates events or happenings of note, changes to how people are thinking about, understanding and/or representing the world, or the emergence of new ways to behave or engage (such as technological and medical advances, or new forms of social media). Finally, the phrase “claims to truth”, provides possible ways to respond to objective, verifiable facts, based on notions of causality and/or prediction. In other words, claims to truth detail why things are, and how they could be.

The desired outcome of this active and critical engagement is a situation in which individuals become ever more knowledgeable, and so better able to make good decisions and lifestyle choices. For instance, being ideas-informed is linked to better health and other measures of wellness, since the key to personal fitness – as well as tackling many illnesses, such as diabetes – is information and education. Likewise, within the overall umbrella of the coronavirus disease (COVID-19) pandemic, the spread of the Omicron variant (which occurred after the mass availability of vaccinations and booster jabs) highlighted how those less well informed were more likely to end up in intensive care units as a result. Being ideas-informed also ensures (amongst many other things) that people: i) maintain a broad understanding of scientific and technological advances and, where relevant, gain knowledge as to how these can be utilised to improve our lives/society (e.g. Pinker, 2018); ii) demonstrate a general awareness of artistic, literary or cultural developments – with cultural capital linked to a range of positive outcomes, from success at school to future level of income (e.g. Brown, 2021; DiMaggio, 1982); and iii) are cognizant of political events (Lamb et al., 2020). This latter point is likely to lead to strengthened democratic processes, in which politicians and local community leaders are held to account (Franco et al., 2019; Hochschild, 2010).

Further, as a result of being ideas-informed, citizens will also be in a better position to re-align their values in response to new progressive beliefs and norms (Franco et al., 2019; Pinker, 2021; Pinker et al., 2019): in other words, to re-align their values in response to beliefs, perspectives or proposals for change that seek to improve the human condition (i.e. in relation to the nature of our relationships with one another) and challenge unhelpful social and economic structures, as well as encourage us to reconsider how we behave towards the natural world: e.g. see Brown et al. (2022). This realignment is evident in work which shows that the vast majority of the UK population (80% of those surveyed, based on a weighted poll of 2,244 UK citizens) now believe it important to be attentive to issues of racial inequality and social justice. Likewise, progressive norms present themselves in terms of increased concern with regards to the current environmental emergency and attitudes concerning how it might be tackled, including the now widespread acceptance of veganism.

As a result, there are benefits which can be realised from the ideas-informed society: not just for individuals and their families, but also to communities and society more widely, as well as the natural environment at large (Innerarity, 2013; Pinker, 2021; Pinker et al., 2019; UNESCO, 2005). Yet, despite the advantages available from doing so, there are cold spots in terms of the extent to which different groups and communities engage with new ideas, developments and claims to truth (e.g. see Lamb et al., 2020). For instance, recent structural equation modelling (Brown et al., 2022), based on a representative sample of 1,000 respondents in England, indicates that those in communities displaying low levels of education, where there is also a socially narrow level of cohesion, are less likely to regard keeping up to date as important (here narrow social cohesion corresponds to situations in which one’s interpersonal connections are broadly homogenous: they consist predominantly of people with similarly low levels of education, as well as primarily employed in routine/manual job roles). Nor will members of such communities be as likely to see value in statements concerning inclusion and tolerance, the ethical and sustainable practices of businesses, or the need to support the physical and mental health or ourselves or others (Brown et al., 2022). At the same time, it is these very communities that are likely to benefit most from being able to engage with new ideas, make more considered decisions and choices, and adopt more progressive beliefs and norms. For instance, numerous empirical studies and think pieces point to discrepancies between highly educated, socially open communities, and lower educated, socially narrow communities with regards to:

• Understanding the importance of education and knowing how best to support children’s learning at school (e.g. Brown, 2021; Clark and Poulton, 2011; Hart and Risley, 2003);

• Understanding the importance of a healthy diet and the need to exercise (e.g. Qurban et al., 2018; Stempel, 2020);

• Understanding the importance of community and collaboration (e.g. Jackson, 2019; Lamb et al., 2020; Putnam, 2000);

• Having knowledge of the advantages of building social and cultural capital and how to capitalise on these benefits (e.g. Brown, 2021; DiMaggio, 1982; Jackson, 2019; Martí et al., 2017; Parcel and Bixby, 2015; Putnam, 2000).

Lamb et al. (2020) also caution that low levels of ideas-informed engagement amongst those in low education/socially narrow communities may also have knock on effects in terms of how such communities are subsequently reflected in leadership and decision-making at the community, regional and national levels. This potentially limits the extent to which decision-making reflects the needs and interests of members of low education/socially narrow groups.

The findings of Brown et al.’s (2022) structural equation model also, however, indicate that meaningful short/medium term interventions might yet catalyse progress towards an ideas-informed-society within these communities. For instance, this seemingly occurs when ideas-related conversation and dialogue is fostered amongst group members. Brown et al.’s (2022) study further identifies a number of other possible factors that might affect whether individuals in such communities both consider keeping up to date as important and see value in statements that might typify modern, scientifically literate, progressive societies. These factors include: i) relatively lower levels of pro-social attitudes in low education, socially narrow communities, which typically results in indifference with regards to anything other than one’s immediate situation (e.g. Putnam, 2000); ii) a lack of social-capital network connections in low education, socially narrow communities, which might otherwise expose citizens to new ideas and break down fixed ideas ways of thinking (e.g. Brown, 2021; Jackson, 2019; Putnam, 2000); iii) a lack of cultural capital in low education, socially narrow communities, which might otherwise help individuals feel invested in society more generally (Broadwood et al., 2012; Brown, 2021; Frederickson, 2003); and iv) a lack of engagement with social media by individuals in low education, socially narrow communities, which might otherwise stimulate pro-social behaviors (Modrek and Chakalov, 2019). Nonetheless, while all of the above are indicative of what might catalyse the emergence of the ideas-informed society in these communities, extant knowledge in this area is piecemeal rather than comprehensive.

Research questions

With this context in mind, the aim of this study was to develop a more comprehensive understanding of the barriers and enablers to the actualisation of an ideas-informed society, as well as suggestions for programmes, interventions or initiatives that might enable the ideas-informed society to flourish. The purpose in doing so is to provide insight into what approaches might be most effective in achieving ideas-informed society-related behaviours and outcomes for communities displaying low levels of education and socially narrow levels of cohesion, as well as how these programmes might best be implemented. The research questions addressed by the study are as follows.

1. What factors act as barriers to and enablers of the actualisation of the ideas-informed society?

2. What top-down, organisation or researcher led interventions/programmes have been developed and implemented to actualise the ideas-informed society? Which members of society were these interventions targeted at? How successful have these been (and what is the strength of this evidence)?

3. What bottom-up/citizen/community led activities/actions/phenomenon have been recorded as leading to the types of outcomes associated with the ideas-informed society? What were the activities? Who was involved? What were the outcomes? And to what extent can we associate these outcomes with the specific activity in question?

4. What other non-empirically tested/verified suggestions are there for how the ideas-informed society might be actualised?

Methods

To investigate these four areas, the research team carried out a systematic review (Luzmore and Brown, 2022), “a review of research literature using systematic and explicit, accountable methods” (Gough et al., 2013, p. 2), using an approach informed by Gough et al. (2012) and previously employed by Brown et al, (2021). In particular, our review featured the following features: 1) explicit and transparent methods were employed; 2) the review followed a staged process of retrieving, screening and reviewing literature items; 3) the review is replicable and updateable; 4) potential users were involved to ensure reports are relevant, useful and accessible (with engagement with users occurring pre-, during and following the review process; Rees and Oliver, 2013). To enable the research team to develop a robust and appropriate approach to the review, the following substantive preparatory work was undertaken prior to and during the project:

• Engagement with (four) “practitioner experts” to explore how the notion of the ideas-informed society and its outcomes can be conceptualised, and to gain an understanding of extant programmes and interventions which have sought to actualise ideas-informed society-type behaviours and outcomes. This supported the research team in developing our search strategy (see below) as well as providing a context for ensuring our research outputs are practically useful.

• An internet search for case studies of “bottom-up” instances of ideas-informed society-type activity. As above, this enabled the research team to ascertain the breadth of activities and outcomes the ideas-informed society comprises, as well help us further develop and refine our search terms (see below).

• Engagement with two academic colleagues working in this area to peer review our protocol (below). This approach provided both critique of the search strategy, as well as enabling the research team to identify key seminal works in the field, either as part of the review or for the purposes of providing further context.

Our search strategy

Following the procedures outlined in the EPPI systematic review guidelines (Gough et al., 2012), and the processes used in Brown et al. (2021) the research team developed a comprehensive search strategy to: i) reduce the possibility of bias (including publication bias); ii) maximize the breadth and quality of the literature engaged with; and iii) ensure the production of a review that is rigorous, trustworthy and transparent. EPPI guidelines recommend four main search approaches: a) electronic-database searches; b) hand searches of journals; c) specialist website searches; and d) the use of personal contacts, authors and experts in the field (which, in our case, also included those consulted prior to and during the review) (Gough et al., 2012). Correspondingly, for our review the following sources (which were thought to provide the best possible coverage of the field, based on our knowledge and those we consulted) were employed:

1. Electronic-database searches: Dimensions; EBSCO; JSTOR; Scopus; Proquest: Social Science Premium Collection; Web of Science.

2. Hand searches of journals: New Humanist; Journal of Public Science.

3. Specialist website searches: UNESDOC digital library; The Pew Research Centre.

4. The use of personal contacts, experts and authors: we garnered recommendations of seminal literature from academics and partitioners operating in this area. We also cast our search net more widely, however, asking for ideas and suggestions for ideas-informed society-related literature or case studies, using social media (Twitter), forums, blogposts and noticeboards.

Based on our preparation work, our search terms comprised those set out in Table 1 below. Search strings were developed by combining these synonyms/terms using Boolean logic. Overall, a combination of these search terms using the four sources of literature outlined above, returned a total of 658 outputs for review. Of these, 583 outputs resulted from the sources a, b and c. The use of personal contacts, experts and authors (d), meanwhile, yielded 75 outputs.

Selection strategy

Screening involves review team members assessing which of the outputs returned by their literature searches meet both their inclusion criteria, and the overall aims of their study. In our case, the screening process ensured that only relevant material from the 631 outputs identified were utilised (Gough et al., 2012, 2013). To engage in screening we employed an two-step process comprising:

1. Double screening of the title and abstract of initially identified articles (that is, screening by two reviewers); and

2. Double full-text screening of articles selected for inclusion after title and abstract screening.

Both stages involved the application of criteria to assess whether documents should be included or excluded. For the first stage – title and abstract screening – the following criteria were used (using an AND/OR approach):

1. The output is centred on the notion of the ideas-informed society (or an analogous concept/concepts).

2. The output focusses on actions associated with citizens engaging with new ideas, developments and claims to truth (or analogous concepts such as “staying up to date”).

3. The output focusses on outcomes associated with citizens engaging with new ideas, developments and claims to truth (or analogous concepts such as “staying up to date”).

4. Other than any seminal literature identified by experts, the output should have a publication date after 2005. This follows the publication of the United Nations Educational, Scientific and Cultural Organization (UNESCO) report “Towards Knowledge Societies” which, in recent times, brought the concept of the ideas-informed society to the fore.

To apply these first stage criteria, search results and abstracts were divided between members of the research team, who independently assessed them against the inclusion criteria. Any disagreement regarding a specific output was adjudicated by the project lead in discussion with an appointed external expert. This first stage process eliminated 534 outputs, leaving 97 outputs to review in the second stage: full-text screening. Of these 97, 21 could not be retrieved. The second stage thus comprised a full text review of 76 outputs; this required that studies meet at least one of the following four criteria (AND/OR):

1. The output presents ways in which the ideas-informed society (or analogous concepts) can be conceptualised.

2. The output presents suggestions for how the ideas-informed society (or analogous concepts), or facets of the ideas-informed society (or analogous concepts) can be realised. For example, detailing an intervention designed to encourage citizens to engage with new ideas, developments and claims to truth. Likewise, describing the actualisation of the ideas-informed society at a macro-level. These accounts can be either theoretical or empirical.

3. The output reports on bottom-up/citizen/community led activities/actions/phenomenon that seemingly lead to the types of outcomes associated with the ideas-informed society.

4. The output provides means of conceptualising outcomes associated with citizens of engaging with new ideas, developments and claims to truth (or analogous concepts such as “staying up to date”).

To undertake this second round of screening, researchers independently examined full texts to assess their relevance and potential inclusion. This eliminated a further 51 outputs, as only 25 outputs met these second stage criteria. These studies were subjected to in-depth review. No further studies were excluded at the in-depth review stage, as each addressed the review’s research questions as anticipated. Of the final set of 25 studies, we note that 19 were peer reviewed articles, two were doctoral dissertations and four were reports. Geographically, 11 were focused on the USA, four on the UK, three were international, two on the Netherlands, one on Spain, one on Korea, one on Canada, one on the Czech Republic and one on Australia. The methodological design of these studies included 11 surveys, seven case studies, three theoretical or instrumental reports, two experimental designs to test hypotheses, one analysis of publicly available data and one review of literature. Four of the case studies featured interventions in line with the aims of the ideas-informed society: three of these were “top down” in nature and one was “bottom-up” in design.

Adopting a configurative approach and our criteria for assessing quality

Typically, systematic reviews are either “aggregative” or “configurative” in nature (Gough et al., 2013; Gough, 2021). When the methodology of systematic reviewing was in its infancy aggregative approaches were most commonly employed, with the aim of such studies being to provide an “adding up” of findings to answer the review question. This type of approach is invaluable when considering issues of effectiveness or impact (such as when exploring questions of “what works”). But it is less useful when the aim of the review is more exploratory. Correspondingly, and in keeping with Brown et al. (2021), for this review, we adopted a configurative approach, i.e. one in which the synthesis is primarily concerned with organising (configuring) findings from literatures to address questions of an investigative nature. It is noted by Gough that, when undertaking configurative reviews, both qualitative and quantitative research can be considered for inclusion. It is also possible to incorporate research of varying quality when there is little to be lost by doing so: for instance, when advancing ideas for future policy or interventions, which will then subsequently be rigorously tested (Gough, 2021). Nonetheless, with Research Question 2 (RQ2) we do explicitly seek to consider the “success” of (or outcomes resulting from) ideas-informed society related programmes. As such, we imposed a requirement to be as rigorous as possible when judging effectiveness, so that more concrete programme-related recommendations could be made with greater confidence. To assess the quality of the studies that might potentially be used to explore RQ2, we drew on the research quality assessment frameworks developed by both Gorard et al. (2019) and Gough (2021).

In terms of the former, Gorard et al. (2019) suggest that assessments of any research study’s quality should be judged in terms of the design, scale, completeness of data, and the quality of data, using the criteria set out in Table 2 below. As such, the first step in using the framework is to identify the information for each of these quality criteria from each study being examined. If the study does not include key information, or if it is written in such a way that the reader cannot ascertain this information, then the research must be rated as having low security. Correspondingly, doubt must be cast on its findings (Gorard et al., 2019). The framework developed by Gough (2021) is analogous but also requires researchers to assess the perspectives underpinning the research. Specifically, Gough invites systematic reviewers to examine whether such perspectives cohere with the likely needs of the potential users of the review. Further, whether alternative perspectives could lead to different truth-claims being made. Should this transpire Gough argues that researchers can and should employ these alternative perspectives as a way of critiquing the trustworthiness of any study.

Results

We flag here that few high-quality studies (when judged using the frameworks above) emerged during the review process for any of our research questions. As such, the synthesis stage of the review involves us directly addressing each of our four research questions by bringing together the findings of all relevant (and, in the case of research question two, as trustworthy as possible) studies identified. The review therefore primarily provides a synthesis of current perspectives with regards to facilitating the ideas-informed society. The issue of quality aside, however, the result still represents a logical and coherent amalgam of findings, which ensures an end product that is “greater than the sum of the individual studies” (Gough et al., 2012, p. 283). We now set out the results of the review for each research question (RQ). Further, in accordance with the best practice suggestions made by Gough et al. (2012), responses to each question were presented to the consultation group (see “Methods” above) to ascertain their face validity and the extent to which findings might be of practical use. The syntheses below thus also incorporate feedback received from that group.

RQ1.

What factors act as barriers to and enablers to the actualisation of the ideas-informed society?

Barriers and enablers to the actualisation of ideas-informed societies appeared well considered and documented. In particular, five prominent barriers and enablers emerged from the review. These are:

1. Access to ideas: which could be through face-to-face networks, online social media networks and via physical, broadcast and digital information and ideas sources (Franco et al., 2019; Schütz, 1946). Here public broadcasting is thought to help provide access, even reaching those who do not actively seek to stay up to date (Aalberg, 2019). The requirement for access also provides a connection between the notion of the ideas-informed society and the concept of the knowledge society: where the use of information communication technology is used to ensure knowledge and information is freely available (Franco et al., 2019; UNESCO, 2005). Clearly, however, internet access is vital for accessing ideas and even now the digital divide – the gap between different socioeconomic groups’ access to and use of information and communications technology (ICT)/the internet is still a problem. For instance, data for 2020 highlights that only 51% of households in England with an income of £6k to £10k had access to the internet at home, compared with 99% of households with an income over £40k. Also vital is digital literacy: the skills to use ICT effectively to access ideas; with estimates suggesting that a fifth (22%) of the UK population are digitally illiterate (Hutton, 2021). The review thus points to the existence of a world of digital haves and digital have-nots which, when it comes to the ideas-informed society, represents a world of knowers and know-nots; people who have access to the world of ideas and those who don’t.

2. Education: which provides not only knowledge-related dispositions (including a desire to inquire, learn and question), but also literacy (a basic requirement for engaging across a multitude of domains) (Franco et al., 2019; Howell and Brossard, 2021). As the aforementioned structural equation model analysis undertaken by Brown et al. (2022) indicates, educated societies are therefore more likely to be informed societies (Hochschild, 2010; Schütz, 1946). Conversely, where education is lacking, it is suggested that individuals are more likely governed by micro-level intrinsic interests. Furthermore, individuals are also more likely to hold the view that the world beyond these micro interests functions in ways that are both beyond their control and without need of their input (Schütz, 1946).

3. Availability: as well as having access, pertinent ideas and information need to be allowed to circulate widely. This again ties the notion of the ideas-informed society to the concept of the knowledge society. Sometimes the circulation of pertinent new ideas requires leadership; although sometimes it can happen by “osmosis”. In either case, however, the more overlap that exists between socially networked communities, the greater the chance that a new disruptive idea comes to light and is adopted to the benefit of individuals and society (Franco et al., 2019; Henrich, 2020; Schütz, 1946). This same premise also relates to social media, where algorithmic-induced biases can limit the level of divergent ideas that individuals have access to, so potentially limiting exposure to novelty (Franco et al., 2019; Lewandowsky, 2020).

4. Judgement: in order for the ideas-informed society to deliver the types of benefits envisaged in the introduction to this paper, citizens need to know how to judge ideas, developments and claims to truth and place them into their proper context. First and foremost, judgement, means being an effective and critical, but also collaborative, consumer of ideas, with an ability to separate facts from opinion (Franco et al., 2019): a state also referred to as being “informationally literate” (Goldstein, 2020). This requirement is nicely reinformed by the work of Achterberg et al. (2017) which indicates that less educated citizens have a higher propensity to “distrust” scientific institutions than those with higher levels of education (although this doesn’t apply to scientific method per se: and for more on this area, see Anderson et al., 2012). As a result, those with lower levels of education are more likely to believe in theories or suggestions that emerge from non-scientific organisations, but which are couched in the language of science. Thus judgement (and the aforementioned notion of literacy) is required to ensure that people engage critically with ideas and are not duped by plausible sounding “conspiracy theories” or by the promotion of “alternative facts/truths”.

   Other facets of judgement include citizens needing to recognise the issues associated with idea “balkanisation” while challenging themselves to be open minded and willing to learn, rather than overlook ideas that challenge their pre-existing beliefs/worldview (Kahneman, 2011). Likewise, people can also often fall prey to engaging with ideas, developments and claims to truth through an emotional lens (Baer, 2020). Recognition of this can foster more critical judgement, which supports objectively better decision-making. Further, how we engage with ideas is intrinsically linked to our networks, which influence how we judge (e.g. based on the opinions and other forms of socially derived and/or socially approved knowledge we have access to: Brown, 2021; Schütz, 1946). Finally (linking it to availability, above), judgement also relates to one’s ability to recognise the sorts of informational biases which occur when social media platforms use algorithms to target individuals with information that is linked to past behaviours – so limiting the bandwidth of what is received (Franco et al., 2019); this means judgement is also affected by what is referred to as “media literacy”.

5. Ability to engage in respectful debate: An ideas-informed society will only function effectively if citizens have a mindset of engaging productively with new perspectives – i.e. they do so with a view to improving both their outcomes and those for society generally. In other words, citizens must be open to the possibility of change and ready to accept the validity of other opinions (Franco et al., 2019; Schütz, 1946). Achieving such a mindset requires both a desire and the ability to engage in respectful and reflective discussion in which citizens feel confident in challenging themselves as well as others. Also helpful is an ability to put oneself in the place of others in order to meaningfully understand them, as well as how they differ from ourselves; likewise, recognising that those we debate with are also people with beliefs, values, ideas and life experiences of their own (Baer, 2020; European Commission, 2018; Goldstein, 2020; Sacks, 2020). Detrimental therefore are attitudes concerned predominantly with “winning the debate” at all costs. This is particularly so when it comes to online discussion, where exchange can never be successful if all we do is engage in social media “indignation, condemnation, character assassination and self-righteousness…” (Sacks, 2020, p. 189).

RQ2.

What top-down, or researcher led interventions/programmes have been developed and implemented to actualise the ideas-informed society? Which members of society were these interventions targeted at? How successful have these been?

When considering the quality of underpinning evidence, only two “trustworthy” interventions for fostering ideas-engagement emerged from the review. The first centred on stimulating curiosity, so that people would begin to seek out information on a range of subjects that interested them. The second explored pro-active café-style attempts to inform citizens about new ideas, developments and claims to truth. Both approaches are now presented in more detail, below.

Fostering curiosity: Citizens are likely to engage with new ideas when they are curious (Brown et al., 2022; Gino, 2018). For instance, as part of an extensive study into curiosity at the workplace, Gino (2018) found that actively encouraging employees to be curious helps ensure they are less likely to fall back on “tried and tested” methods and more likely to actively look out for new or alternative approaches when seeking to problem solve. Further, Gino (2018) found that it is possible to foster states of curiosity, so stimulating this type of ideas-informed innovative behaviour. For instance, in a small-scale randomised control trial of 200 participants (working across a range of different industries and companies), Gino sent one half (the intervention group) a twice weekly text message for four weeks which asked: “What is one topic or activity you are curious about today? What is one thing you usually take for granted that you want to ask about? Please make sure you ask a few ‘Why questions’ as you engage in your work throughout the day. Please set aside a few minutes to identify how you’ll approach your work today with these questions in mind.” (Gino, 2018, p. 50). The other half (the control group) received a message designed to trigger reflection, but not raise their curiosity: “What is one topic or activity you’ll engage in today? What is one thing you usually work on or do that you’ll also complete today? Please make sure you think about this as you engage in your work throughout the day. Please set aside a few minutes to identify how you’ll approach your work today with these questions in mind.” (ibid). After four weeks, the participants in the intervention group scored higher than those in the control group on questions assessing their innovative behaviours at work (such as whether they had made constructive suggestions for implementing solutions to pressing organizational problems). This leads Gino to conclude that not only can curiosity be piqued but it also provides impetus “to seek new information and experiences” (Gino, 2018, p. 48).

Science cafés: For around 100 years across the 17^th and 18^th centuries, coffee houses were situated at the centre of urban life. They became popular meeting places where people gathered, not just for the coffee, but to also discuss politics, as well as make sense of scientific advances, geographic discoveries and emerging literary and artistic developments (Ellis, 2004). A contemporary reincarnation of this phenomenon (albeit one on a much more modest scale) are science cafés. Science cafés emerged in the UK in 1998 and are organised by universities or corporate bodies (such as public health agencies) to facilitate the coming together of researchers and members of the public to engage in debate and discussion around science issues. While the format of the café varies per location and event, the small number of studies in this area suggest some commonality of experience: i) an informal and welcoming atmosphere; ii) open access to all; and iii) the opportunity for discussion following presentations on research (Dijkstra, 2017; Dijkstra and Critchley, 2016). As well as cafés, other informal locations, including bookshops, bars or music venues (with some featuring live music as part of the event) are also used in order to provide a more relaxed setting in contrast to a formal lecture. Low cost to run, some science cafés are staffed by volunteers, although other organisers are sometimes paid to carry out tasks such as publicising the event, securing locations, finding speakers, researching topics and so on. A study undertaken by Dijkstra (2017) into the motivations and opinions of those attending a series of science cafes centred on the theme of nanotechnology in the Netherlands (drawn from a sample of 238 visitors), found that the main reasons for attendance were “interest in the theme of the meeting” and “to broaden or deepen my knowledge” (both 73%). Around about one third (38%) of the café visitors were attending for the first time, with one third (37%) having been to between two and five café sessions and 26%, indicating they were frequent engagers – attending meetings more than five times. This suggests that the science café format can have appeal when positioned correctly.

While communicating findings is an important goal of science cafés, their overall aim is “contributing to the processes of social learning” (Dijkstra, 2017, p. 3); with organisers placing a high value on the discussion and engagement aspect of each event, serving to differentiate this approach from more passive forms of knowledge exchange. The rationale to move past one way communication has also seen science cafes used as a pilot model to facilitate and improve health outcomes through targeted approaches to community outreach in in Minnesota and Florida in the US (known as “garden cafés”). Funded by National Institutes of Clinical and Translational Science, garden cafés were developed and designed by community leaders and used local community health data to inform priorities for what each café would cover. Cafés were hosted in easily accessible outdoor spaces and, as well as including information and dialogue from health researchers, they also provided an offer of onsite health checks for attendees, as well as an opportunity for to address wider needs with representatives from housing and libraries in attendance (Balls-Berry et al., 2018).

Attendees at both café-types reported higher levels of “literacy” in the topic that they have engaged in (Balls-Berry et al., 2018; Dijkstra, 2017), with studies again highlighting the importance of participation. For example, visitors to the nanotechnology events considered dialogue with researchers as a way of them (as citizens) being able to raise concerns, seek clarification and discuss the risks and benefits of new technology with researchers (Dijkstra, 2017). Those attending the garden café were able to provide feedback to organisers on future topics or areas of interest relating to their own health concerns for future sessions (Balls-Berry et al., 2018).

Finally, in a similar vein to the café approach, Kollman et al. (2013) provide a detailed case study of their work, situated in the Museum of Science (Boston, MA), which aims not only to inform citizens about science but to promote democratic dialogue through a semi-permanent exhibition called “Provocative Questions”. As well as encourage visitors to learn about extant research underpinning a current societal health issue (such as “should sugary drinks be taxed”), the exhibition also encouraged visitors to construct informed arguments about that issue, while enabling them to recognise how their personal experience and societal values informed their perspectives. The structure of the exhibition is set out in Figure 1 below; with Kollman et al. (2013) suggesting that two thirds of visitors indicate that they would, having engaged with the exhibition, subsequently feel comfortable talking about their opinions on the topic in question (as well as able to support their arguments with evidence). Yet, while the case study provides a detailed explanation of their approach, a lack of information in Kollman et al. (2013) about visitor numbers, as well as those completing surveys, means the impact of the “Provocative Questions” exhibition is difficult to assess.

RQ3.

What bottom-up/citizen/community led activities/actions/phenomenon have been recorded as leading to the types of outcomes associated with the ideas-informed society? What were the activities? Who was involved? What were the outcomes? And to what extent can we associate these outcomes with the specific activity in question?

A number of initiatives were also identified during our internet searches for case studies of “bottom-up” instances of ideas-informed society-type activity. These include:

1. The Academy of Ideas: The Academy of Ideas was founded in 2000 with the defining principle, “free speech allowed”. It seeks to create “intelligent, open-minded public debates that explore current events and new ideas”. As well as a monthly newsletter, it has a podcast and supports local, national and international forums and salons to facilitate discussion. For example, the Birmingham Salon in April 2022 was titled “The Unbearable Lightness of Citizenship” and featured panel discussions, with suggested readings for attendees to enable them to participate effectively. The Academy’s flagship event, The Battle of Ideas began as an annual event in 2005 to “create a space for public debate on the issues of the day”. Billed as a place where “you can meet your “enemy”, listen to opinions you have never heard before, argue back and even occasionally change your mind” (ibid) the Battle of Ideas holds debates themed around topics such as “Battle over Science”, “Contemporary Controversies”, “Institutional Meltdown” and the “Culture Wars”, with around 300 speakers engaging in 70 debates across one weekend. The Academy of Ideas has also established a charity (also called Battle of Ideas) which seeks to advance education and good citizenship, through encouraging a culture of learning and discussion. In 2020/21 the charity engaged over 1,000 individuals from 25 schools across the UK and Germany via online debates and seminars.

2. Pint of Science: a non-profit grassroots organisation that provides a space for researchers and members of the public to come together in a relaxed environment to share thoughts, questions, and ideas. Underpinned by a similar premise to that of Science Cafés, Pint of Science is based on the idea that it is easier to take scientists out into the community than to bring the general public into labs (and, more specifically, easier to encourage members of the public to hear about science in bars and pubs). From just three events in 2013, Pint of Science now supports events across 500 international cities, to engage citizens with topics such as “Tech Me Out”, “Atoms to Galaxies” and “Creative Reactions” (this latter event bringing together both art and science). The events have no fixed format, but the aim is for experts to not only present their research and ideas but to also interact with a lay audience through activities such as pub quizzes, games and discussion. The enterprise has been recognised and awarded by University of Oxford for Public Engagement and was given a Points of Light Award from the Prime Minister’s Office for its approach to making complex topics accessible to all.

3. How The Light Gets In: is a biannual event organised by the Institute of Art and Ideas. Attracting crowds of over 30,000 over four days, it enables attendees to engage with topics across a range of themes (such as culture, society, ethics, technology and language). Further, for an additional fee attendees can even book to have breakfast with Nobel laureates in the “inner circle”.

4. Votes for Schools: is an organisation which provides weekly resources for schools to “prompt impartial discussion on a challenging current issue”. Its tagline is, “Be informed. Be curious. Be heard”. Fully resourced lessons and assemblies are provided to subscribing teachers and leaders (differentiated for pupils from Primary to Further Education) which allows for participants to vote on a discussion point. The organisation has 850+ school subscribers and a reach of over 51,000 students. During the 2020/21 academic year Votes for Schools topics included, “should we share our medical data?”, “are the days of cash numbered?” and “is the refuges crisis really over?”. Votes for Schools shares the outcomes of student votes with relevant organisations and the UK government. When responses are received from policy-makers, these are reported back to the schools to demonstrate the potential impact of their “voice”.

Unfortunately, little empirical evidence corresponding to effectiveness of these initiatives was presented and that which did exist was weak in terms of design and scale, meaning each of 1) to 4) above scored zero when examined in light of the research quality assessment framework set out in Table 2 above (Gorard et al., 2019). Furthermore, no explicit theory of change for these initiatives was presented, meaning that effective ways of replicating them can only be supposed, based on the descriptions given (Brown and Flood, 2018).

RQ4.

What other non-empirically tested/verified suggestions are there for how the ideas-informed society might be actualised?

Beyond the initiatives detailed above, the literature also spotlighted other areas for consideration when seeking to actualise the ideas-informed society. For instance, the value of getting certain groups involved in the types of cultural activities that might help them feel more invested in a community or society generally; or even to improving their own life outcomes. In other words, a potential chain of cause and effect was posited in some of the outputs reviewed, which posed the question: if, after engaging in cultural activity, people subsequently and passionately care more about their community, society or their life outcomes, are they also likely to become more interested in the new ideas, developments and claims to truth, since these will have more relevance to them? (Gustafsson and Lazzaro, 2021). It was also suggested that this type of approach may be especially pertinent in areas of deprivation where cultural engagement is lower (e.g. see Brown, 2021) and the benefits to communities of citizens engaging with new ideas, developments and claims to truth, as well as adopting certain values, beliefs and behaviours is relatively higher (Brown, 2021; Brown et al., 2022). It may also have the additional benefits of building cultural capital where this is known to have long lasting benefits in terms of future outcomes (e.g. Brown, 2021; DiMaggio, 1982). In a similar vein, arts activities can also enable people to reflect on the world, thus potentially enabling them to relate to it in new ways (Gosden, 2021). Art can also offer hope by showing that alternative futures or outcomes are possible (for instance, by signifying or portraying something different to and better than the current situation). Feeling hopeful can subsequently kickstart a desire to find out more in order to steer future action: in other words, hope can catalyse people’s engagement with ideas (Broadwood et al., 2012; Gosden, 2021).

Findings from the review also indicated that citizen’s social networks affect whether they are likely to engage with new ideas or not (something afformed by the SEM). For instance, Dostie-Goulet (2009) surveyed 499 young people in four schools across Canada annually, over the course of three years, to see whether their social network influenced their interest in political matters. Dostie-Goulet’s (2009) results suggest that, where the networks of young people took part in political discussion (and, in particular, if their parents took part in such discussion) respondents were significantly more interested in politics than those whose networks did not. Further, we can also expect such networks to share a range of ideas. For instance, Masip et al. (2018) found that even in socially homogenous networks, people will share ideas and information that do not necessarily align with their own political or ideological positions, suggesting that discursive social networks tend not to become informationally isolated; even more so when discussion occurs in online social networks, with Thomas and Vinnuales (2017) suggesting that alternative viewpoints can pique users’ curiosity and encourage interaction and engagement.

At the same time, Erisen and Erisen (2012) argue that the nature of the social network that people belong to is more than just a source of information; it impacts of the quality of engagement with ideas, such as those related to political thinking. Here Erisen and Erisen (2012: 843) define “quality” political thinking as “the extent to which an individual can express knowledge on political issues and concepts, observe the causal connections in political issues, and differentiate among the distinct dimensions of issues and the alternative ways of approaching a potential political problem”. Their findings indicate that those in cohesive social networks (i.e. who have strong close relationships with relatives and close friends), are more likely to exhibit less “complex thinking” and less able to produce policy-relevant thoughts. In other words, having a closed social network is negatively associated with quality of political thinking while, conversely, a wider social network where communication includes people from different social contexts, is likely to increase the chances of quality political thinking.

As noted earlier, as well as engage with ideas, developments and claims to truth, it is vital that citizens can do so critically. Franco et al. (2019) posit a number of steps citizens can take to support themselves in this endeavour. These include: (i) engaging fully with the idea, development or claim to truth; (ii) finding the key arguments presented in favour of its main area of focus; (iii) scrutinising the credibility of the idea source (e.g., how is the source qualified to present the idea, development or claim to truth), while also assessing whether it is presented from a stance of neutrality and objectivity; (iv) examining what people or organizations are presented as figures of authority; (vii) reflecting on the objective of presenting the idea, development or claim to truth (e.g., is the intention to persuade, inform, entertain…) and who might benefit as a result; (viii) checking whether critique of the idea is welcome; (ix) searching for counter-ideas or arguments, which also do not corroborate or reinforce one’s own beliefs on that topic; (x) exploring if that same idea, development or claim to truth has been disseminated (and so supported) by another source; and (xi) determining whether there are references to support any evidence presented, and whether these can be verified. These ideas were tested with 23 participants (albeit those from a highly educated background, e.g. teachers, academics, journalists and librarians) as part of a workshop to raise participants’ awareness about the impact of disinformation. Here some three quarters (76.2%) of attendees rated the workshop as “excellent” and the remaining quarter “very good” at providing strategies that might prove useful in critiquing ideas, developments and claims to truth.

Finally, it is also beneficial when citizens engage in productive action on the bases of the ideas they are informed by (in other words, to turn knowledge into action). Here findings from one study exploring citizenship education suggested that being affectively engaged with the idea(s) in question (i.e. being empathetic towards, or connecting emotionally with it or what it relates to); exploring the idea (or ideas) in-depth; and having an understanding of how to act in relation to the idea (or ideas) all help lead to concrete action materialising as a result (Atkins et al., 2016). These findings cohere with the work of Kegan and Lahey (2009), whose research demonstrates that people will only alter their behaviour when they both understand what they need to do (the technical aspect of change) and have a strong commitment to doing something different (the adaptive aspect of change).

Discussion

In many ways our review illustrates that both the process and content of education are vital to the actualisation of the ideas-informed society. This is evident, for example, in the section on barriers and enablers, with the review highlighting the need to ensure that citizens join society armed with knowledge-related dispositions, such as: i) the ability to critically judge ideas, developments and claims to truth and place them into their proper context, or ii) ensuring future citizens are able to engage in respectful debates concerning new ideas. Likewise, the review highlights the importance of literacy, so that citizens can actually engage with ideas across a myriad of domains. This coheres with the findings of earlier structural equation modelling (Brown et al., 2022), which posits a strong link between level of education and ideas-engagement. Beyond what education can provide, however, we see that the presence of the digital divide, or ensuring that ideas spread widely and reach all, is affecting the realisation of the ideas-informed society in ways that mirror patterns of social inequality (again reflecting Brown et al., 2022). Yet, our review also identifies seemingly impactful approaches to enabling citizens to engage with new ideas or for removing the barriers to doing so. Such approaches included science or garden cafés and museum exhibitions. Other, more bottom-up approaches included community-based events and festivals (although the impact of these bottom-up approaches could not be ascertained), with social networks (and discussion within these networks) also seemingly key to whether and how individuals engage and the breadth of ideas they engage with.

But while it is important that citizens are able to engage with new ideas, what is also vital is that they actively want to do so (and of course, subsequently act on these wants). The original structural equation modelling analysis undertaken by Brown et al. (2022), however, indicates that approximately 29% of the population do not feel this impetus: with 13% actively regarding staying up to date as unimportant, and 16% seemingly ambivalent. So, what insight did our review provide for this group? As noted earlier, the structural equation model found a strong connection between level of education and the importance people ascribe to engaging with new ideas; in particular, for those individuals living in narrowly cohesive communities where ideas-related discussion doesn’t materialise to the same degree. Since higher levels of education provide individuals with a propensity to inquire, learn and question (Franco et al., 2019) it might be assumed that the converse applies within such communities, with the absence of social networks to engage individuals in ideas-related discussion meaning there are scant opportunities to catalyse such a propensity. If this conjecture is true, then perhaps the most promising intervention uncovered by this review is that by Gino (2018), which utilised a regular text messaging approach to stimulate curiosity, finding that doing so led to individuals subsequently engaging with new ideas and knowledge (although it should be noted that Gino’s intervention was not directly aimed at the specific group we are interested in). Further, as indicated in the findings of Dostie-Goulet (2009) and Erisen and Erisen (2012), the ability to discuss these ideas in social networks (face to face or via social media) is likely to reinforce ongoing high-quality engagement once curiosity has been piqued, so helping ensure that citizens become more knowledgeable, better able to make decisions and find themselves in a better position to re-align their values. Finally, also vital is that individuals from a low education background are also supported to develop ideas-related competencies that they might have not had the opportunity to develop at school. For instance, through the use of the 11-step process outlined by Franco et al. (2019) above, which provides guidance for ascertaining the veracity of an idea, as well as support for engaging in respectful but challenging debate (Baer, 2020).

Conclusion

It is speculated by Berger and Kellner that the first ever idea occurred when one of our early homo sapiens ancestors stopped to consider something more than the everyday activity of banging with implements and ensuring fires stayed lit. We can make a good guess at what that idea was: something akin to “this tribe is in a state of crisis” (Berger and Kellner, 1982, p. 143). We can suppose that this idea led to the tribe taking action and subsequently flourishing. But imagine if that idea had not been had. Or not communicated. Or not engaged with. The notion of the ideas-informed society represents a modern-day interpretation of this successful outcome – a desired situation in which: 1) citizens see value in staying up to date, and; 2) citizens regularly keep themselves up to date by actively and critically engage with new ideas, developments and claims to truth. As a result, it is anticipated that: 3) citizens become more knowledgeable; 4) citizens find themselves in a better position to make good decisions, and; 5) citizens align their perspectives with appropriate societal values. Given the potential benefits of 3) to 5), of primary interest, therefore, are “ideas refusers”: those who do not value staying up to date, nor attempt to do so (Brown et al., 2022; Dijkstra, 2017). With this review we have sought to identify ways to consider how such refusers might be switched-on to engaging with new ideas. While the review found many ways to ensure that citizens can engage with new ideas, we have also, vitally, found approaches that may ensure that citizens also actively want to engage, by arousing curiosity and providing subsequent avenues to effectively engage.

Nonetheless, while such approaches to stimulating demand for ideas seemingly display promise, it is still, at this stage, conjecture to suggest that any intervention which provides a triumvirate of piquing curiosity, establishing connections to social networks and arming citizens with ideas-related dispositions will actually result in an uptake in those becoming more ideas-informed. As such, and given the potential for very real benefits to materialise for those in those in low education, narrowly cohesive communities from becoming ideas informed (e.g. in terms of health, wealth, environment, democratic participation, community relations, and so on), it is suggested that future research activity in this area now focuses on testing the effectiveness of potential interventions. For instance, via the use of Randomised Control Trials or Quasi Experimental Approaches: i.e. research approaches which can ascertain whether approaches to stimulating ideas-engagement actually result in significantly beneficial behavioural change. We also believe this work should now become a priority. This is because, with society in the midst of an environmental emergency, an obesity crisis, facing growing disparities between the haves and have nots, where there is danger that hard won moves towards equity and rights might be lost, and when individuals are beginning to turn their back on what science and traditional politics has to offer (along with a concomitant growth in reactionary populism), we suggest that this research, and its potential for facilitating more effective ideas engagement, is more vital than ever.

Data availability

Underlying data

All data underlying our results are available as part of the article and no additional source data are required.

Extended data

OSF: Facilitating the ideas-informed society: a systematic review.

https://doi.org/10.17605/OSF.IO/5396R (Luzmore and Brown, 2022).

This project contains the following extended data:

• PRISMA Flow Diagram Facilitating the ideas-informed society: a systematic review.docx (this provides the flow of information through the different phases of the systematic review. In particular, the diagram maps out the number of records identified, included and excluded, and the reasons for exclusions).

Reporting guidelines

OSF: PRISMA checklist for “Facilitating the ideas informed society: a systematic review”.

https://doi.org/10.17605/OSF.IO/5396R (Luzmore and Brown, 2022).

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Publisher’s note

This article was originally published on the Emerald Open Research platform hosted by F1000, under the Fairer Society gateway.

The original DOI of the article was 10.35241/emeraldopenres.14729.1

Author roles

Brown C: Conceptualization, Data Curation, Formal Analysis, Methodology, Resources, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing; Luzmore R: Data Curation, Formal Analysis, Investigation, Project Administration, Writing – Review & Editing; GroßOphoff J: Conceptualization, Formal Analysis, Writing – Review & Editing

Funding statement

The author(s) declared that no grants were involved in supporting this work.

Competing interests

No competing interests were disclosed.

Reviewer response for version 1

Christopher Fisher, Australian Research Centre in Sex, Health & Society, La Trobe University, Melbourne, Vic, Australia

Competing interests: No competing interests were disclosed.

This review was published on 6 February 2023.

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

recommendation approve

Overall, an excellent article. Very well written with a very solid and well-explained methodology. Findings and subsequent discussion were appropriate to the research questions. Below are some thoughts to further improve the manuscript and add more depth to the discussion.

1. Suggest revising sentence 3 of paragraph 2 as the issues around not being vaccinated resulting in greater hospitalisation rates was not always so much about being "less well informed" but choosing to believe certain 'claims to truth' which were not objective and verifiable (e.g., anti-vaccination claims that it would lead to autism or other such outcomes).

2. 4th paragraph: May want to define what is meant by 'cold spot' or consider altering – maybe variations?

3. Factor iv in last paragraph before research questions is interesting given it has often been cited that social media is a primary vehicle for inaccurate information. It simultaneously can provide a sense of being ideas-informed while encouraging masses of people to adopt beliefs not grounded in objective and verifiable truths. This is an interesting juxtaposition to the 'Access to ideas' finding and is nicely nuanced by the 'Judgement' and 'Ability to engage in respectful debate' finding. It be interesting to see more wrestling with this dilemma in the discussion.

4. I've done 'Science Cafes' 'Pint of Science' events before. Great way to share and discuss new knowledge. But also perhaps a bit of a self-fulling prophecy in that those coming are already part of an ideas-informed society and open to new ideas, as noted in your findings. Did these documented cases of such interventions describe the low education/socio-economic populations identified as needing to be targeted? Could similar formats be used to spread subjective, unverifiable information? This would also be interesting to see discussed and deconstructed a bit more in the discussion.

Is the argument information presented in such a way that it can be understood by a non-academic audience?

Yes

Could any solutions being offered be effectively implemented in practice?

Yes

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable

Is real-world evidence provided to support any conclusions made?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Does the piece present solutions to actual real world challenges?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise: Adult education, Adolescent education, Sexual health.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Reader comments on this article

Suhaila AlHashemi, Sultan Qaboos University, Oman

The manuscript is interesting, clear in terms of purpose and well structured, some references need to be updated. Interesting how further studies can be conducted and research on this article can be investigated further

Competing interests: No competing interests were disclosed.

Figures

Figure 1.

The structure of the Provocative Questions museum exhibition.

Table 1.

Search terms used for the systematic review.

1) Ideas	2) Informed	3) Society	4) Facilitating	5) Outcomes
Developments	Based	Networks	Access	Choice*
Evidence	Judgement	Groups	Capital	Improvement
Facts	Understanding	Communities	Promoting	Enhanced
Innovation	Co-creation	Individuals	Programmes	Flourishing
Knowledge	Discourse	Collaboration	Interventions	Perspective*
Information	Rationa*	Partnership	Barriers	Progress*
New possibilities	Engage*	Citizens	Enablers	Decision*
Re-imagining	Driven	Conversation	Harnessing	Up to date
Research		Dialogue	Literat*	Wise
Truth			Information overload	Knowledgeable
Values			Quality	Optimal
Solutions			Assessment	Ethic*
Theory			Availability	Moral*
Science			Education	*Equality
Findings			Leadership	“force of the better argument”
Opinion			Social networks	Post-truth
Ideology				Misinformation
“Marketplace of ideas”				Echo chamber
Free speech				Pro-social
News				Ignorance
Current affairs				Trust in science
Curiosity				Informed citizens

Table 2.

A framework for assessing research quality (Gorard et al., 2019).

Design	Scale	Completeness of data	Data quality	Rating
Strong design for research question	Large number of cases per comparison group	Minimal missing data, no evidence of impact on findings	Standardised, independent, pre-specified, accurate	4
Good design for research question	Medium number of cases per comparison group	Some missing data, possible impact on findings	Standardised, independent, not pre-specified, some errors	3
Weak design for research question	Small number of cases per comparison group	Moderate missing data, likely impact on findings	Not standardised, independent, or pre-specified, some errors	2
Very weak design for research question	Very small number of cases per comparison group	High level of missing data, clear impact on findings	Weak measures, high level of error, too many outcomes	1
No consideration of design	A trivial scale of study, or number is unclear	Huge amount of missing data, or not reported	Very weak measures, or accuracy not addressed	0