Methodology to categorize the research output of academic institutions, through the Sustainable Development Goals (SDGs)

Abstract

Background: This article presents a methodology to categorize scientific publications according to the targets of the 17 Sustainable Development Goals (SDGs) of the United Nations. For the above, a dataset with bibliographic and descriptive attributes of 2,379 articles from 2017 by co-authors affiliated to the Pontificia Universidad Católica de Chile, indexed in the Web of Science and Scopus databases, was used.

Methods: The methodology considered three relevant and consecutive milestones: establishment of the reading level that was applied for each publication record, which considers a proportional amount of information; assignment of one of the 18 categories identified for the analysis of the information, which include the 17 SDGs and the option “unclassified” and one of the 169 subcategories corresponding to the specific goals; and, finally, recording the status of the review process carried out, which allowed control of the progress and quality of the cross-review.

Results: The results show that 58.6% of the articles contribute to a primary target, of these 233 contribute to a secondary target; goals 3, 4, 9, and 11 are the most frequent in the process of assigning SDGs. There is an 81% increase in the use of alphanumeric targets when they are assigned as secondary targets. At the same time, cross-checking is shown to be beneficial when allowing the reclassification of 190 articles to some of the targets. Finally, it is established that levels 2 and 3 enabled better classification, given that the contents considered provide more information; however, it is significant that through level 1, 355 articles were categorized as “unclassified”.

Conclusions: It is concluded that the methodology allows for a conclusive, exhaustive, rigorous, extensive, and varied classification through the different milestones and actions carried out, providing strategic information for decision making and research management in the academy-society relationship.

Citation

Cortes-Rodriguez, P., Rondanelli-Delpiano, R., Santander-Meneses, P. and Vilches-Vargas, R. (2023), "Methodology to categorize the research output of academic institutions, through the Sustainable Development Goals (SDGs)", Emerald Open Research, Vol. 1 No. 4. https://doi.org/10.1108/EOR-04-2023-0012

Introduction

The Sustainable Development Goals (SDGs) offer guidelines for their implementation according to the United Nations (UN) indications on the grounds that, through these consensual measures, it will be possible to emphasize the end of poverty, the well-being of the population, the improvement of health conditions, and guarantee access to education and gender equality, among others.

This was one of the essential outcomes of theUnited Nations Conference on Sustainable Development (2012), which involved the agreement of the member states to launch a process to develop a set of Sustainable Development Goals, that would build on the Millennium Development Goals ( Martínez-Agut, 2012).

"The Sustainable Development Goals were adopted by all UNDP (United Nations Development Programme) Member States in 2015 as a universal call to end poverty, protect the planet, and ensure that all people enjoy peace and prosperity by 2030. The 17 SDGs are integrated, as they recognize that interventions in one area will affect outcomes in others and that development must balance environmental, economic and social sustainability." ( https://www.undp.org/sustainable-development-goals).

At the national level, a plan of action has been adopted: "Chile agenda 2030 Sustainable Development Goals" which includes monitoring the rate of progress. Figure 1 shows the information according to the latest report.

For this reason, there is an interest at the institutional level in identifying how research is aligned with these goals by analyzing the scientific publications generated by researchers.

With this background, a methodology was developed that allowed an analysis of the total record of the scholarly production of the Pontificia Universidad Católica de Chile in 2017, categorizing each work according to one or two SDGs and their respective goals. This methodology requires "human intelligence" to apply a documentary analysis procedure that in this case is carried out by a team of librarians from the Bibliotecas UC, which has an advantage over an automatized analysis by providing a higher level of understanding from the writing to infer the eventual correlation, not only with the SDGs but more specifically with the targets.

Peña & Pirela (2007) indicate that documentary analysis is a process devised by the individual as a resource to organize and represent the knowledge recorded in documents, whose rate of production exceeds the possibilities of reading and capturing them. The action of this process is based on the analysis and synthesis of the data captured in such documents through the application of linguistic guidelines or regulations. Through this the substantive content is extracted, which may correspond to a specific term, groups of terms taken in isolation, or gathered in discursive constructions.

This article shows the need for academic and research institutions to record and analyze how their research is reflecting or incorporating the SDGs, and an understanding that it is a priority to implement a sustainable development agenda at the country level.

Methods

An information survey was conducted to identify tools or methodologies for classifying scientific publications within the SDGs. A relevant experience was reported by RELX Groups, a UK company that provides information and analysis services, through its SDG Resource Centre platform, whose objective is to support the UN in the implementation of the SDGs and raise awareness among the target audience of this platform through a limited number of publications classified in the general objectives, not in specific goals. Similarly, AURORA University Network is an initiative that aims to provide a tool to strengthen the focus on social challenges among its member universities and their research through the SDGs ( Vanderfeesten & Otten, 2017; Vanderfeesten et al., 2019).

Experiences and applied cases have also been identified that used information records from databases of scientific publications, but these are focused on classifying the goals and not the targets. They base their classification on topics or categories assigned to the journals where the research is published, consequently, they are the first step to perform a more accurate classification ( Jayabalasingham et al., 2019; LaFleur, 2019; Wastl et al., 2020).

Therefore, our methodological proposal consisted of assigning one or two targets associated with the SDGs to each scientific publication. This assignment was carried out through a manual item-by-item classification, by a team specialized in documentary analysis. The item-by-item classification methodology has traditionally been used as part of the work of information professionals to contribute to the categorization of scholarly academic production by discipline ( Glänzel et al., 1999). It is under these principles that the methodology described below is provided.

Technology and human resources

The Web of Science (WoS) Core Collection and Scopus databases were considered due to their broader coverage of scientific publications, as well as the high level of curation of their publication data performed by the respective editorial teams. WoS belongs to the provider Clarivate Analytics, which has a portfolio of platforms that can retrieve and analyze high-impact scientific literature. Also, Scopus, from the provider Elsevier, comprises the most extensive collection of journals and conference proceedings, among other information resources.

The information exported from the databases was processed in a spreadsheet and was called Worksheet.

In turn, a team of five librarians with skills in database use, management, analysis, and classification of information was considered. Thus, their skills and expertise allowed for an accurate, logical, and structured analysis of the information.

Datasets and attributes

The dataset comprised a single list of publications exported from WoS and Scopus, to which a process that eliminated duplicate records was applied. This considered 2,379 scientific publications in English or Spanish of type "Article" published in the year 2017, with authorship associated with the Pontificia Universidad Católica de Chile. These are original investigations with a standard structure comprising: title/abstract/keywords, objective(s), results, discussions, and conclusions.

The dataset was compiled and imported into a spreadsheet. It included bibliographic attributes and classification attributes for each of the publications. Regarding the bibliographic features, these were meant to describe and locate each publication record. Thus, the bibliographic characteristics of the articles were used as follows:

Descriptive attributes

Location attributes

Concerning the classification attributes, these allow the methodology to be executed, assigning a corresponding value to each record analyzed by the team of specialists, according to the manual classification process. In this regard, two relevant criteria were established that led to the creation of the process.

First, a maximum classification limit was established for each item, equivalent to the assignment of two goals and their respective targets, with the following considerations:

- Possibility that some publications do not contribute to the SDGs, in which case they were marked as “unclassified”.

- Possibility of detecting more than two targets to a document, in this case these were prioritized, corresponding to different SDGs, benefiting the thematic diversity of the document.

Secondly, a technical reading of the document's body was performed, considering the following sections: title, abstract, keywords, results, discussions/conclusions.

Given the above, the following attributes were generated:

Classification attributes

It is important to note that the detection and registration of a primary goal and target implies that these are determinants for the classification of the article. In the case of secondary goal and target, implies that the article addresses or tangentially resolves some aspect of the definition of this second objective detected.

Sorting process

The classification process carried out by the team of specialists considered three consecutive milestones: establishment of the reading level applied to each publication record; assignment of one of the 18 categories identified in the analysis of the information, which include the 17 SDGs and the "unclassified" option and one of the 169 subcategories corresponding to the targets; and, finally, recording the status of the review process carried out.

The following reading levels were determined:

First reading level (Level 1): Used as long as the information provided by the title and keywords was sufficient to assign a goal and corresponding target. It considers about 15 words.

Second reading level (Level 2): Used whenever the information provided by the summary was necessary to assign a goal and the corresponding target. It considers between 250 to 500 words.

Third reading level (Level 3): Used whenever the discussions or conclusions allowed the better classification of the assigned goal and the corresponding target. Considers over 500 words

Fourth reading level (Level 4): Whenever it was essential to consider reading the main body of research, focused on the results section, allowing the clarification of conceptual doubts or doubts regarding the objective of the research. It considers over 1,000 words.

As indicated above, the categories considered for the classification of each item include goals and targets. To work with the official description for each category, the information on the 17 SDGs available on the platform provided by the UN was considered.

These descriptions made it possible to understand the thematic considerations for each goal, according to the respectively associated targets. In order to facilitate the manual classification, the thematic considerations of each goal were reduced to "key topics", thereby recognizing their most relevant content, allowing an association with the target content of each article.

The "key topic" brings together different words relevant to the description of a specific goal. The conceptual delimitation of the target prevents it from being confused with another target. It also allowed it to be recognizable in the complete list of targets, corresponding to 169 targets, so that it can be more efficiently assigned to an article.

Table 1 shows, as an example, the key topics for each goal of Goal 3 "Good Health & Well-Being". As can be seen, the words assigned as "key topics" are present, in their entirety and textually, in the description of each target. Thus, the description of Target 3.1 is as follows "By 2030, reduce the global maternal mortality ratio to less than 70 per 100,000 live births" and the key topic is limited to "maternal mortality"; therefore, any article that focuses on maternal mortality will be of interest for classification in Target 3.1.

Regarding the status of the review process, it consisted of finalized, pending, or evaluation. If one or two targets were assigned to an article, the process was marked as completed; if it was only assigned as "unclassified", it remained as in evaluation. Finally, if the process is assigned as pending, it indicates that the full text of the article could not be accessed to correctly identify any target.

Records marked as evaluation were assigned to another reviewer as a cross-check. This allowed a comparative analysis and validation by another member of the work team. All of the above made it possible to establish follow-up and quality control of the analyses carried out by the team of specialists.

Classification stages

According to Figure 2, in general terms, the workflow of the classification process begins with the activity "enter the bibliographic record of the document in the respective database". Then, it proceeds with the technical reading of the content of the articles according to the appropriate level to detect any associated goal and ends with the activity "select process status as finished".

The following are the steps in the workflow:

1. Enter the document record in the respective database.

2. Read title, author keywords, and journal name: if sufficient (Yes) continue with step 7, indicating; If insufficient (No), continue with the next step.

3. Read abstract: If sufficient (Yes) continue with step 12; If insufficient (No) continue with the next step.

4. Access full text of the article: If available in the same database (Yes) continue with the next step; If not (No) continue with step 9, registering "pending".

5. Read discussion and/or conclusion section: if sufficient (Yes) continue with step 7; if insufficient (No), continue with the next step.

6. Read results section: if sufficient (Yes) continue with step 7; if still unsure about goal (No), continue with step 9 by recording "evaluation".

Process for assigning a goal

• 7 Assign, as appropriate, in the “Goal” column:

  1. Assign a goal. Continue with step 8.

  2. Assign “unclassified”: the article is not related to any SDGs. Continue with step 9 and enter “evaluation”.

• 8) Assign a specific target according to a selectable list in the worksheet.

• 9) Indicate the status of the categorization process in the “Categorization Process” column:

  1. Finished: the classification process is closed with a goal and definitive target.

  2. Pending: access to the full text is required. Continue with step 10.

  3. Evaluation: requires cross-review, by another reviewer. Continue with step 11.

• 10) Request search support for the full text of the article with a librarian colleague.

  1. If it is possible to access the full text, return to step 5.

  2. If it is not possible to access the complete text, record it as an observation.

• 11) A second review process is carried out for those records under “evaluation”. This process consists of a cross-review to ensure that the same record is not revised by the same reviewer and to enrich the analysis of the content of an article. Continue with step 12.

• 12) The second reviewer returns to step 1 and decides the level of analysis required to determine the goal and continue to step 7 and record as appropriate, goal and target or “unclassified”. Finally, it will be recorded as “finished”.

Examples of the classification are shown below:

Example 1)

Article ID: ICYT.2017-341

Article title: Management of severe ulcerative colitis. An up-to-date. DOI: http://dx.doi.org/10.4067/S0034-98872017000100013.

Document analysis level (1st review): Level 2. Abstract reading.

Primary goal (1st review): 3 “Good Health & Well-Being”.

Primary target (1st review): 3.4 “By 2030, reduce by one third premature mortality from non-communicable diseases through prevention and treatment and promote mental health and well-being”.

Categorization process (1st review): Finished.

In the previous example, the abstract section of the article clarified the focus of the proposed research, so the technical reading level corresponded to level 2 - reading the abstract. Then, the document indicates that it deals with ulcerative colitis (non-communicable disease) and, likewise, that its timely and up-to-date diagnosis and management has led to a reduction in mortality, therefore, it corresponds to the description of Target 3.4. Thus, the classification process is completed.

Example 2)

Article ID: ICYT.2017-2043

Article title: Change in symptomatic burden and life satisfaction during short‐term psychotherapy: Focusing on the role of family income. DOI: http://dx.doi.org/10.1002/capr.12158.

Document analysis level (1st review): Level 2. Abstract reading.

Primary goal (1st review): Unclassified

Primary target (1st review): Unclassified

Categorization process (1st review): Evaluation.

Document analysis level (2nd review): Level 3. Discussion or conclusion.

Primary goal (2nd review): 3 “Good Health & Well-Being”.

Primary target (2nd review): 3.4 “By 2030, reduce by one third premature mortality from non-communicable diseases through prevention and treatment and promote mental health and well-being”.

Categorization process (2nd review): Finished.

In the example above, it is noted that the first reviewer of the article considered that it did not apply to any objective when reading at level 2. The reading of the abstract, therefore, was categorized as in "evaluation" to be reviewed by another colleague or in a cross review. For the discussion or conclusion, hence, reading the additional content to the first reviewer allowed him to detect its relevance to Goal 3 and specifically, to Target 3.4, as this section clarifies and enriches with more details the final objective of the research focused on improving and promoting treatments for mental health. It is in this last instance where the classification process ends.

Results

Quantity and relative proportion of items classified in one or two goals

As a result of the application of the proposed methodology, 2,379 articles were analyzed, of which 1,396 records, corresponding to 58.6% of the total number of documents, were assigned a primary goal. In turn, 233 articles had a second or secondary goal ( Cortes et al., 2021).

As shown in Table 2, 45.3% of the articles were classified with Goal 3 "Good Health & Well-Being", as a primary goal. This indicates high productivity in medical sciences and health, given that 633 articles were classified under this goal. Goal 11 "Sustainable Cities & Communities" with 8.5%, Goal 4 "Quality Education" with 8%, Goal 9 "Industry, Innovation, & Infrastructure" with 6.9% and Goal 15 "Life on Land" with 6.3% also show a high percentage. The contribution to the remaining 12 goals decreases considerably, at under 4%.

Table 2 shows the number of articles with a secondary goal. These correspond to 233 entries, equivalent to 16.7% of the total number of documents with a primary goal. As with the assignment of the primary goal, we see that most of the articles contribute to secondary Goal 3 with 30.5%. In addition, Goal 16 with 10.7% and Goal 9 with 10.3% stand out with a high percentage.

When comparing the situation of the SDGs most frequently used in the classification, concerning the so-called primary and secondary goals, it is evident that Goals 3, 4, 9, and 11 are used more frequently in both cases, therefore, their importance in the set of articles analyzed is consistent, concluding that they are suitable both as a primary and secondary goal. On the other hand, the use of Goal 16 as a secondary goal is noteworthy, since, although it is not considered in the group of the most frequent in the primary goal classification, it does present a high frequency as a secondary goal with 10.7%.

It is also interesting to note that, although Goal 15 is located in the set of primary goals with the highest frequency at 6.3%, equivalent to 88 articles when considered as a secondary goal, it only reaches 3.9%, which is below the average frequency of classification. This indicates that Goal 15 is preferably applied as a primary goal.

Results of the distribution of the articles considering the assigned targets

The results are reported below in relation to the targets used for classification.

When considering the total number of articles classified by each objective, Table 3 shows the targets, both primary and secondary, used and assigned to the documents by the team of specialists. In addition, a comparison is made relative to the percentage of utilization of the total number of objectives comprising each goal.

Concerning the articles that do contribute to the SDGs (1,396), it is observed that a total of 1,629 targets were attributed, of which 1,396 correspond to the assigned primary target and 233 to the assigned secondary target. In total, 31 targets exceeded the average frequency of use (12.8), with the following standing out: 3.4, 3.B, 9.5, 3.C, 3.D, 3.8, 11.3, 15.1, 3.3 and 16.7 with 351, 106, 83, 59, 41 36, 34, 33, 33, 33 and 29 items assigned, respectively.

In general, Table 3 shows the distribution and frequency of articles assigned to each target of the 17 SDGs. It is important to note that, for this sample, duplication of articles was considered to represent the total number of assignments for each target, therefore, both primary and secondary targets were added together.

Continuing with the analysis presented in Table 3, it is possible to identify that 100% of the targets that include goals 3, 4, 6, 7, 9, and 11 were used for item classification. Considering that goals 3, 4, 9, and 11 were mentioned as those most frequently used for the categorization of articles, this information makes it possible to highlight goals 6 and 7, which although they are below the average frequency of use for the classification performed, nevertheless, each of their targets were feasible to use in the categorization and therefore, were distributed in the few documents associated with these goals. This situation shows the pertinence and relevance of all the targets that conform to goals 3, 4, 6, 7, 9, and 11 in the scientific research that was analyzed, as well as highlighting the specific analysis capacity achieved by the review team.

On the other hand, there is a low percentage, less than 50%, of the diversity of targets used for the total number of goals that comprise goals 1, 10 and 17. This indicates that the targets not used in the classification have a smaller scope, in terms of their suitability to be addressed by scientific research with similar characteristics and context to the set analyzed.

Furthermore, it is evident that 14 of the 17 SDGs present the maximum number of classified items (highlighted in color) in one of the numerics targets, for example, Targets 1.4, 2.4, or 3.4; and the remaining three goals, in the alphanumerics targets, for example, Targets 4.C or 7.A. This highlights the importance of using the first set of numeric targets for the proposed classification. Moreover, when considering this set of targets, it is observed that the number of SDGs that present 100% in the use of their numerical targets increases to eight.

This section shows the results of the classification concerning the strength of the relationship between a primary and secondary target, according to the double target assignment presented by the subset of 233 papers identified at the beginning of the previous section. Relationship strength is understood as the higher number of times or occurrences between a primary target and a secondary target. To analyze those with the highest relationship strength, the ones that exceeded the average number of relationships observed in the set analyzed, equivalent to an average of 3 times, were considered.

Table 4 shows, on the one hand, the number of articles that exceeded the average strength of relationship, therefore, whose primary and secondary targets coincided in their classification at least three times; on the other hand, it indicates the targets that are most closely related. The greater strength of the relationship is observed between the targets of Goal 3, highlighting the relationship between targets 3.4 and 3.C, which present a frequency of 14 articles that coincide in the relationship of both targets. The next most important relationship is between 3.4 and 3.B with a frequency of 8 times. Next, and with a lower number of associated articles, the relationships 3.4 - 16.7 and 7.A - 9.5, although they have a frequency of 4, stand out because they are relationships between targets of different goals, which is relevant given that it responds to one of the principles of the methodology, consisting of the use of various SDGs when classifying an article.

Table 5 goes further into the subset of the 233 articles, through an analysis that shows the preference for the use of numeric or alphanumeric targets as primary or secondary. Thus, it is observed that in 82% of the cases a numeric target was assigned as primary, while the percentage decreases to 67% when they are used as a secondary target. Consequently, alphanumeric targets increase their use by 81% when they are assigned as secondary targets. This supports and promotes the use of alphanumeric targets in a secondary position.

Results of the cross-review of "unclassified" articles

The proposed methodology considers repeating the process only for those records that are "unclassified" in the worksheet. Thus, as shown in Table 6, a total of 1,173 articles were categorized as "unclassified" in the first review carried out by the team. This set was distributed proportionally through a cross-review, allowing the records of one reviewer to be analyzed by the other reviewers. As a result of this second selective review, 983 articles were definitely “unclassified”.

The cross-check made it possible to reclassify the original "unclassified" category of 190 articles, corresponding to 16% of the 1,173 records that after the first review did not apply to any goal and target. Thus, Table 7 shows the reassignment of this subset of articles to the SDGs.

Table 7 shows that in this second revision it was possible to reclassify the articles in 15 of the 17 goals and that the highest number of documents were classified in Goal 3, equivalent to 45% of the total number of reassigned articles. Once again, the prevalence of this objective in the classification process is confirmed, together with evidence of the complexity of its use in the categorization of the SDGs, since in the first review its relevance had not been detected in the 85 reclassified articles.

Similarly, Goal 9 stands out for its frequency of use in this cross-review process, with 21% of the articles reclassified. This result demonstrates, once again, the conceptual complexity of the SDGs and the care required in the process, in order not to rule out their application in the articles analyzed in the first review.

It is also interesting to note that this new revision contributed to an increase in the number of articles that contributed to SDGs that were not frequently found in the general analysis, such as Goal 5, 8 and, 17.

Results of the articles reviewed, according to reading level

Finally, information is provided regarding the reading level reached in the analysis process that considered the final classification given to each article. This process, which involves assigning a reading level ranging from level 1 to 4, is based on the amount and complexity of the article's content required to determine the correspondence of an article to a goal and target or none, "unclassified".

According to the results shown in Table 8, the review team reached the most complex level of reading, consisting of reviewing up to the results of the research, in 0.4% of the total number of articles. In turn, most of the dataset, equivalent to 64% of the documents, were reviewed and, as a consequence, classified at level 2, which considers reading up to the abstract. However, it is interesting to note that when there was a reading level 1 that corresponds to the analysis of the title and keywords of the article, in 86% of the cases the reviewers considered that the document did not contribute to any SDGs. Therefore, this subset, which represents 14% of the articles, is feasible to exclude with the minimum data consigned in the title and keywords.

It is also evident that in 56% of the cases of analysis (level 2- 4), more content of the article was required to decide on the classification in some SDGs (articles with SDGs). In this regard, it is relevant to consider that an average abstract may contain 500 words and the overall body of the article from 2,000 words, therefore, it is essential to analyze a higher number of words to correctly identify the goal, recognizing the context, research purpose, main results and impact on the environment, among others, concerning research.

Conclusions

The proposed methodology made it possible to develop a categorization of articles into targets (SDGs) that was conclusive, exhaustive, rigorous, broad, and varied, through different milestones and actions, such as cross-review of articles with "unclassified" status; use of key topics that correspond to highlighted words included in the description of each target; reading level reached in each article review; attributes considered for the review; article sections that were valid to make the classification by the team more efficient.

The cross-review of “unclassified” articles allowed us to reduce the number of records that were not assigned a goal by 16%, which establishes the importance of having different points of view and enhancing decisions through human intelligence. Undoubtedly, an important part of this general and particular classification process, in this subset, was the use of a hierarchical structure of "key topics" that identify and conceptualize the most relevant aspects of the description of each target. This structure made it possible to narrow down the understanding and reading of each target description, in order to consider its assignment to the corresponding article.

Another relevant aspect that is possible to conclude corresponds to the level of classification reached for the articles; thus, levels 2 and 3 were the most used in this process. This shows that it is essential to consider the maximum amount of information provided by the article when classifying targets. As established in the methodology, these levels include the reading of the title and keywords (level 1), abstract (level 2), conclusions, and discussion (level 3), therefore, it is not enough to only generate a keyword strategy to determine the target that corresponds to an article. The evidence indicates that it is necessary to enhance these actions, given the subtlety in detecting the intrinsic objective and the specific content of the article, which is clearly not found in a limited number of keywords.

On the other hand, it became evident that a subset of 389 articles reached level 1 of reading and documentary analysis, which can be considered for applying an initial filter, through keywords or contents, that allows automatic classification of these and thus reduces the classification time of the rest of the articles. Moreover, considering that, within that subset of articles, 335 were categorized as "unclassified", therefore no target was assigned to them. Consequently, only the title and keyword information were sufficient for such an assignment.

As already mentioned, the methodology used in the categorization of these documents according to an specific goals and targets was based on a process of "human intelligence", where due to the technical reading of the articles (title, keywords, abstracts, etc.) it is possible to determine their content and associate them accurately to one or two goals and their respective targets. Among the activities developed by the team of librarians in charge of this categorization, the identification of the associated "key topics" was very effective and favored the documentary analysis process.

The initial decision to limit the assignment to a maximum of two targets per article was not necessary in most cases (83.3%) and in the cases where two SDGs were assigned (16.7%), it was decided to consider the most relevant one.

Another aspect analyzed in the results was the strength of the relationship between two targets assigned to the same article. This information allows us to contribute to whether it is possible to establish trends of relationships between targets. After reviewing the results of the cases where there was the same occurrence of two targets and with a frequency of at least three articles, it was determined that the data are not significant to support the strength of the relationship between targets. In any case, it is established that Target 3.4 is linked to 3.B in 14 articles, which is equivalent to 6% of the subset of 233 documents with two assigned targets.

The review made it possible to confirm a relationship between the primary target (numeric) and the secondary target (alphanumeric) associated with a subset of 233 articles. According to the results obtained, it is possible to specify the use of alphanumeric targets as secondary.

In addition, it can be seen that all 17 SDGs were used to a greater or lesser extent, and concerning the targets, out of a total of 169 targets, 127 were used, representing 75% of the total. Furthermore, we can highlight that in seven SDGs 100% of their targets were used and only three SDGs had an allocation of 50% or less of the total targets.

The results obtained in this methodology, which was applied to scientific publications, demonstrate the need to consider the use of human intelligence as fundamental and a priority in any process of classification of articles in targets, whether these are manual, one by one, or automatic through classification models that consider algorithms or search strategies. The contribution of human intelligence in this type of classification continues to present advantages over others of an automated nature since they allow an enriched, complex, and exhaustive analysis, getting closer to the discovery of the target and the intrinsic contribution of each article to the SDGs, which are possible to detect through our ability to understand both the explicit and implicit content.

The classification of a university's scientific production to the SDGs provides an overview of how research from different knowledge disciplines contributes to the SDG targets. Thus, it improves strategic decision-making and research management in the academia-society relationship, providing relevant information and progress indicators regarding the contribution of scientific research to this important global initiative.

Data availability

Underlying data

Repositorio UC: Categorization of articles 2017 with authorship of Pontificia Universidad Católica de Chile, through the SDGs. https://doi.org/10.7764/datasetuc/61951 ( Cortes et al., 2021)

This project contains the following underlying data:

• categorization_ods_article_uc_2017.xlsx (categorization process of articles)

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Publisher’s note

This article was originally published on the Emerald Open Research platform hosted by F1000, under the ℈Responsible Management℉ gateway.

The original DOI of the article was 10.35241/emeraldopenres.14355.1

Author roles

Cortes-Rodriguez P: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Supervision, Validation, Visualization, Writing - Original Draft Preparation, Writing - Review & Editing; Rondanelli-Delpiano R: Conceptualization, Formal Analysis, Investigation, Validation, Visualization, Writing - Original Draft Preparation; Santander-Meneses P: Conceptualization, Investigation, Project Administration, Validation, Visualization, Writing - Original Draft Preparation, Writing - Review & Editing; Vilches-Vargas R: Conceptualization, Formal Analysis, Investigation, Methodology, Resources, Validation, Visualization, Writing - Original Draft Preparation, Writing - Review & Editing

Funding statement

The author(s) declared that no grants were involved in supporting this work.

Competing interests

No competing interests were disclosed.

Reviewer response for version 1

Nkeiruka Ndubuka-McCallum, Aberdeen Business School, Robert Gordon University, Aberdeen, United Kingdom

Competing interests: No competing interests were disclosed.

This review was published on 26 May 2022.

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

recommendation: approve-with-reservations

This article shows the need for academic and research institutions to record and analyze how their research is reflecting or incorporating the SDGs, and an understanding that it is a priority to implement a sustainable development agenda at the country level.

The background provided is not compelling enough on this front, especially with the lack of robust discussion around Figure 1. Of the percentages reflected, is there an estimate for contributions made by academic and research institutions, if indeed part of the percentages? It would have been useful to unpack figure 1 for context and explore why goal 3 and goal 10 have the highest and lowest reporting status in Chile, respectively. Is there any relationship here with your study finding, given what seems to be a significant contribution to goal 3 (see Table 2)?

2) Read title, author keywords, and journal name: if sufficient (Yes) continue with step 7, indicating; If insufficient (No), continue with the next step.

It would be much easier to cross-reference the workflow steps and Figure 2 if Table 1 did not take up a page between them. I realise Table 1 is referred to within the main body of text before Figure 2 is, but having the workflow steps next to Figure 2 would seem more important.

If the steps, both in the workflow (page 5) and in the flowchart (figure 2), were annotated with a system counting the number of Yes and No answers, this could allow more specific reference to the flowchart steps within the workflow; “Y1N2” could, for example, indicate the box the user would arrive at after answering Yes once followed by No twice in a row (“Level 3. Analysis of discussions and conclusions”). This would, in turn, allow S earch in another database, through title or DOI to be referred to in the process as “N1”, whereas, as things stand, it cannot be called Step 2, Step 3 or any other number as it does not come before or after Level 1. Analysis of the title and keywords but is instead an alternative. Where there were two or more pathways to the same box, the annotation would default to the shortest route [e.g. Assign a goal and target (SDGs) would be annotated as Y2 since two Yes responses take you there, even though it could also be arrived at after the equivalents of Y1N1Y1, Y1N2Y1 or Y1N3Y1].

Interesting stats in terms of Table 2. What might be the reason for the significant numbers for goal 3? How many articles and associated studies were funded compared to others linked to goals other than SDG 3, and did this have any bearing on the number of articles reported? Given that the articles were published in 2017, we cannot link this directly or indirectly to the pandemic. However, it may be useful to know if the numbers have progressed (or not) since 2017 and to what extent the pandemic might have a bearing on this via future research. From a research excellence viewpoint, drawing on the UK REF (https://www.ref.ac.uk/about-the-ref/), what is the research environment like within your institution - specifically the medical sciences and health - compared to business and management discipline or business and engineering schools, for example? Does it have any bearing on the productivity reported against goal 3? Then again, a high number of publications is not synonymous with “high quality”; the so-called world-leading and internationally excellent research in terms of originality, significance, and rigour (within the UK higher education context). How many staff are research active within the medical sciences and health discipline compared to other disciplines and/or schools, did this have a bearing on the figure reported? How many of the faculty members or researchers/academics are aware of the SDGs agenda and are actively engaged in making contributions to their realisation through research? The strata in terms of positions [i.e. early career, mid-career and long-standing research-and-teaching-focused academics] may also be useful details that could be leveraged from your study via future research; that is, knowing how engaged they are towards the SDGs. I appreciate that they probably did not have the SDGs in mind when drafting the papers reviewed. However, you could gauge the awareness (according to the three categories) of how their studies (in scope articles), are pushing the SDGs agenda further. Related findings should help in informed decision making towards localising the SDGs and the critical role that academics across the three levels can play in achieving these goals through research and other channels.

Consequently, for those unaware of the SDGs or who consider it an abstract concept/agenda, the usefulness of your study is the opportunity it offers to help them make the necessary connections, see/recognise the applicability of their research and related practices to the SDGs – how they are helping realise them and areas for improvement (e.g., goals 1 and 5) at disciplinary, school/department and institutional levels (Filho et al., 2017; Vilalta et al., 2018). ¹,² The same applies to other academics in other HEIs within and beyond the study boundary – the impact of their research towards building a sustainable future for all. Their areas of research interest or disciplines may not be considered mainstream, nonetheless helping directly or indirectly to progress the agenda in a positive direction.

When comparing the situation of the SDGs most frequently used in the classification, concerning the so-called primary and secondary goals, it is evident that Goals 3, 4, 9, and 11 are used more frequently in both cases, therefore, their importance in the set of articles analyzed is consistent, concluding that they are suitable both as a primary and secondary goal.

Any possible impact of the specialists' disciplinary orientation, aside from general knowledge provided on the relevant United Nations' web page about the SDGs, on the highlighted? Were these goals more easily interpreted and more apparent than the others due to their background regardless of the methodology followed, which by the way, is transferable to other spheres/contexts outside higher education institutions (HEIs) within and beyond Chile; thus, a useful contribution. After all, “the success or failure of the SDGs will depend on effective monitoring to a great extent. Hence, well-crafted indicators and high-quality data are relevant in offering governments, businesses, academia, and civil society the information they need to target resources, policies, and programmes” (Neubauer and Calame, 2017, p. 74, citing Villiers, 2015). ³ However, given that the methodology is human-centric, it is worth noting that it is a subjective exercise to a certain degree and thus value-laden and not entirely free from bias. For future research looking to adopt the proposed methodology, a multidisciplinary team of experts and non-experts in SDGs should be involved in the analysis process in order to ensure rigour and, most importantly, limit/curb disciplinary-related biases from potentially skewing the results. This is reinforced by Neubauer and Calame (2017), who argue that setting up indicators is almost as important as the indicators themselves. Thus, different results in terms of indicators and shared societal visions are likely to emerge from closed expert discussions vs open democratic debates with citizens.

Goal 16 as a secondary goal is noteworthy, since, although it is not considered in the group of the most frequent in the primary goal classification, it does present a high frequency as a secondary goal with 10.7%.

The question about the disciplinary orientation of the specialists is probably a useful consideration here, especially pertaining to the 2 ^nd reviewer. An attempt to discuss the possible reason(s) would have been useful. The reading of Goal 16 as framed on the UN web page ( https://sdgs.un.org/goals) is almost as inclusive as goal 17. I say so because I see dimensions of goals 3, 4, 5, 8, 9, 10, and 11 in this one goal – 16. The SDGs though individual goals, are interlinked.

The reference list is not comprehensive [see reference list]. Overall, a relevant contribution whose findings can be leveraged on various fronts. “The question is not so much if HEIs have expertise or not, but rather whether this expertise and its further development will be prioritised enough...The SDGs agenda is an opportunity to reinvent and build on the humanist and emancipatory tradition of universities, to emphasize the value and agency of human beings, to prefer critical thinking over acceptance of outdated dogmas, and to promote research and education as political issues (in the best sense of the word), thereby contributing to building a fair worldwide community of emancipated citizens” (Neubauer and Calame, 2017, p. 76). Thus, it should not be hijacked and exploited in the further pursuit of the managerialist agenda steeped in performativity culture that has permeated HEIs – the would-be demise of a once noble profession.

Is the argument information presented in such a way that it can be understood by a non-academic audience?

Not applicable

Is the rationale for developing the new method (or application) clearly explained?

Partly

Could any solutions being offered be effectively implemented in practice?

Yes

Is the description of the method technically sound?

Partly

Is real-world evidence provided to support any conclusions made?

No

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

Does the piece present solutions to actual real world challenges?

No

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

No

Are sufficient details provided to allow replication of the method development and its use by others?

Partly

Reviewer Expertise:

Responsible Management Education/Learning (RME/L, ERS), Research Evaluation (inc. REF - Research Excellence Framework), Sustainable Development Goals (SDGs), Health, Safety and Risk Management (HSRM).

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

References

1. Leal Filho W, Wu Y, Brandli L, Avila L, et al.: Identifying and overcoming obstacles to the implementation of sustainable development at universities. Journal of Integrative Environmental Sciences. 2017; 14 (1): 93-108

2. Vilalta J. M, Betts A, Gomez V: Higher Education's role in the 2030 agenda: The why and how of GUNi's commitment to the SDGs in Sustainable development goals: actors and implementation. A report from the international conference. Global University Network for Innovation. 2018; 6.

3. Neubauer C, Calame M: Global Pressing Problems and the Sustainable Development Goals in Grau et al (ed.) Higher Education in the World 6 - Towards a Socially Responsible University: Balancing the Global with the Local. Global University Network for Innovation. 2017.

4. Jongbloed B, Veidemane A, Bayezid A: Embedding sustainability in academia: Deans as change makers. Center for Higher Education Policy Studies. 2021.

Reviewer response for version 1

Thaiane Oliveira, Fluminense Federal University, Niterói, Brazil

Competing interests: No competing interests were disclosed.

This review was published on 01 November 2021.

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

recommendation: approve-with-reservations

The article “Methodology to categorize the research output of academic institutions, through the Sustainable Development Goals (SDGs)” aims to propose a methodology to categorize scientific publications according to the targets of the 17 Sustainable Development Goals (SDGs) of the United Nations. The authors develop a method based on levels of reading and classification. They analyzed 2,379 articles indexed in the Web of Science and Scopus published by authors affiliated to the Pontificia Universidad Católica de Chile. The methodology considered three relevant and consecutive milestones: 1) establishment of the reading level applied for each publication record; 2) assignment the corpus in 17 SDGs and the option "unclassified" and one of the 169 subcategories corresponding to the specific goals;

It is a methodological proposal which is in accordance with important movements in the development of metrics and responsible indicators, accordingly for presenting less dependence on international ranking evaluation indicators. It shows the potential of replication of the methodology proposed in research and higher education institutions in different regions of the world. In other words, it is a responsible methodological proposal that tends not to reinforce the asymmetries in the evaluation of production and circulation, as seen in indicator and metric platforms of large companies that provide this data. The strength of the proposal lies in a systematic methodological effort to diagnose and assess the scientific production of universities.

However, some points could be explored further. First, there is an uncritical adoption of the sustainable goals in the literature review. The state of the art presented on the SDGs does not present other views on the SDG calendar. For example, Carlos Gómes Gil (2018) ¹ argues that The SDGs have been criticized for many of the goals being pure rhetoric, that many of the 169 targets are idealistic and visionary, along with very serious problems in the feasibility of the approved indicators. From a critical discourse analysis on the SDGs reports from a Modernization Theories perspective, Weber & Weber (2020) ² argue that SDGs focus on economic growth as a precondition for expanding social inclusion, and a prospect of successfully containing or reversing environmental degradation. They also point out that individual accountability does not make structural changes which are needed to attend the SDGs. Critics, such as those presented by Gil (2018) ¹ and Weber & Weber (2020) ², are fundamental when proposing methodologies for the development of indicators for universities. Incorporating these criticisms into the debate is fundamental to thinking about responsible indicators. By meeting the SDGs as criteria for evaluating science, it also means attributing responsibility for social changes that are unattainable for the university, as it requires an effort and commitment from other social spheres to actually be able to end poverty, protect the planet and ensure peace and prosperity for all are fundamental when proposing methodologies for the development of indicators for universities. Incorporating these criticisms into the debate is fundamental to thinking about responsible indicators.

It is in this sense that the SDGs should be thought of from a rationality of integration, in which it is recognized that the scientific, political, economic, media and social spheres are interconnected. It is necessary to recognize that scientific institutions alone are not capable of making significant structural changes without being interconnected with other national and international social spheres committed to the objectives of sustainable development. This perspective implies, therefore, the addition of another analysis necessary for a holistic understanding of the performance of scientific institutions in society. For example, it is recommended to add in the analysis of how these productions frame their performance in relation to public policies and economic spheres to meet and propose structural changes in society. Only from this perspective will it be possible to propose responsible indicators, without falling into the trap that higher education and research institutions are responsible for major social transformations and to achieve structural changes in society in attendance to the SDGs. In this sense, it is suggested to adopt this criterion in the corpus or to point out this limitation in the final version of the article.

Regarding the methodology, and following ideal Content Analysis protocols, it must be clarified whether double-blind categorization was adopted (Poole and Holmes, 1995 ³; Neudorf, 2011 ⁴; among others). The authors report that records marked as evaluation were assigned to another reviewer as a cross-check. This allowed for comparative analysis and validation by another staff member. In addition to the adoption of these protocols, it is important to emphasize that the classification criteria of content analysis requires the adoption of protocols so that it is strictly followed in a way that minimizes the risks of classifier bias. Likewise, the adoption of reliability tests would be recommended, as Krippendorff (2009) ⁵ suggests.

Finally, the authors point out that the classification enhances strategic decision making and research management in the academy-society relationship by providing relevant information and progress indicators as to the contribution of scientific research. This is an important point when it comes to university governance policies. Governance over scientific institutions or itself is part of the complex of programs, calculations, techniques, indicators, and metrics by which authorities seek to incorporate and evaluate what the university is. If the university has been configured as a space of privilege (Balbachevsky, 2017) ⁶, after the 1990s, it has been experiencing significant changes about institutional governance. On the one hand, due to demands for social equity, university governance policies have gained strength for greater expansion of access to higher education, driving university governance to adopt mechanisms of social inclusion for previously socially excluded groups. On the other hand, global economic changes have led the university to adopt an active role in regional development, incorporating the social function that the institution is also responsible for in the competitiveness of national economies in global markets.

The university, as an institution whose main commitment and responsibility is linked to social - and economic - transformation, is realized through the performance of three substantive functions: teaching, research, and university extension. Thus, university governance has been delineated from these spheres, which should be interconnected. Its task is also explained by three elements: processes, results, and valuation. In turn, scientific institutions are also understood the generators of relevant knowledge through basic and applied research, aiming to improve the quality of life and ensure greater social welfare. In other words, the formulation of indicators needs to take into account these dimensions in the institutional governance of scientific research and higher education. Combining these dimensions is important to avoid the reinforcement of asymmetries into the institutions. The classification for strategic decision-making and research management in the academy-society relationship, as pointed out by the authors, can not be a mechanism for excluding scientific research, extension actions, and educational practices that do not fit the classification of external parameters which is not part of the social function of the university. In other words, by enacting external benchmarks as classifiers for the evaluation of the scientific production of the institution, there is a risk of tangentially excluding the decision making process - especially in relation to the distribution of institutional resources - other activities inherent to the broad spectrum of activities and functions of universities.

Thus, summarizing the points that demand the authors' attention:

• Also bring critical strands on the SDGs incorporated into universities evaluation system;

• Include multisectorial actions with other spheres of society in the analysis to not fall into “pure rhetoric” (Gil, 2018) and not be a trap for the research institutions themselves by promising social changes impossible to be achieved without joint efforts;

• In case you do not consider the inclusion of the criteria specified above relevant or unfeasible to include in the analysis, include it as a limitation of the study.

• Reflect and consider whether the criteria established by the SGDs should be integrated into strategic decision making, especially in relation to research funding distribution in the university, running the risk that basic research or even other types of applied research may end up succumbing to this type of policy.

Is the argument information presented in such a way that it can be understood by a non-academic audience?

Not applicable

Is the rationale for developing the new method (or application) clearly explained?

Yes

Could any solutions being offered be effectively implemented in practice?

Yes

Is the description of the method technically sound?

Yes

Is real-world evidence provided to support any conclusions made?

No

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

Does the piece present solutions to actual real world challenges?

No

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

No

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

Reviewer Expertise:

Science Communication; Alternative metrics; Open Science; Responsible Indicators; Social Impact of Science; Academic Governance

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

References

1. Gil CG: Sustainable Development Objectives (ODS): a critical review. Roles of ecosocial relations and global change. 2018. 107-118

2. Weber H, Weber M: When means of implementation meet Ecological Modernization Theory: A critical frame for thinking about the Sustainable Development Goals initiative. World Development. 2020; 136.

3. POOLE M, HOLMES M: Decision Development in Computer-Assisted Group Decision Making. Human Communication Research. 1995; 22 (1): 90-127

4. Neuendorf K: Content Analysis-A Methodological Primer for Gender Research. Sex Roles. 2011; 64 (3-4): 276-289

5. Krippendorf K: Testing the reliability of content analysis data. The content analysis reader. 2009. 350-357

6. Balbachevsky E: Governanca na pesquisa cientifica: reflexoes sobre a pratica da pesquisa contemporanea e a experiencia brasileira. Sociologias. 2017; 19 (46): 76-101

Figures

Figure 1.

Distribution of mandatory monitoring indicators of the 2030 Agenda (preliminary balance) according to reporting status as of May 2019.

Source: https://sustainabledevelopment.un.org/content/documents/23507Informe_Nacional_Voluntario_CHILE_Junio_2019_final_1.pdf.

Figure 2.

Flowchart of the manual classification process of publications in goal and specific targets.

Table 1.

Example of the detection of "key topics" for the targets of Goal 3 "Good Health & Well-Being".

Sustainable Development Goal	Target	Target description	Key topics
Goal 3	3.1	By 2030, reduce the global maternal mortality ratio to less than 70 per 100,000 live births.	Maternal mortality
Goal 3	3.2	By 2030, end preventable deaths of newborns and childrenunder 5 years of age, with all countries aiming to reduce neonatal mortality to at least as low as 12 per 1,000 live births and under-5 mortality to at least as low as 25 per 1,000 live births.	Deaths of newborns and under 5 years of age
Goal 3	3.3	By 2030, end the epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases and combat hepatitis, water-borne diseases and other communicable diseases.	Communicable diseases
Goal 3	3.4	By 2030, reduce by one third premature mortality from non- communicable diseases through prevention and treatment and promote mental health and well-being.	Non-communicablediseases and mentalhealth
Goal 3	3.5	Strengthen the prevention and treatment of substance abuse, including narcotic drug abuse and harmful use of alcohol.	Substance abuse
Goal 3	3.6	By 2020, halve the number of global deaths and injuries from roadtraffic accidents.	Road traffic accidents
Goal 3	3.7	By 2030, ensure universal access to sexual and reproductivehealth-care services, including for family planning, information and education, and the integration of reproductive health into national strategies and programmes.	Sexual andreproductive health-care services
Goal 3	3.8	Achieve universal health coverage, including financial risk protection, access to quality essential health-care services and access to safe, effective, quality and affordable essential medicines and vaccines for all.	Universal healthcoverage
Goal 3	3.9	By 2030, substantially reduce the number of deaths and illnessesfrom hazardous chemicals and air, water and soil pollution andcontamination.	Deaths and illnessesfrom chemicals andcontamination
Goal 3	3.A	Strengthen the implementation of the World Health Organization Framework Convention on Tobacco Control in all countries, as appropriate.	Tobacco Control
Goal 3	3.B	Support the research and development of vaccines andmedicines for the communicable and noncommunicable diseases that primarily affect developing countries, provide access to affordable essential medicines and vaccines, in accordance with the Doha Declaration on the TRIPS Agreement and Public Health, which affirms the right of developing countries to use to the full the provisions in the Agreement on Trade Related Aspects of Intellectual Property Rights regarding flexibilities to protect public health, and, in particular, provide access to medicines for all.	Vaccines and medicines
Goal 3	3.C	Substantially increase health financing and the recruitment, development, training and retention of the health workforce in developing countries, especially in least developed countries and small island developing States.	Health financing andhealth workforce
Goal 3	3.D	Strengthen the capacity of all countries, in particular developing countries, for early warning, risk reduction and management of national and global health risks.	Early warning,risk reduction andmanagement

Table 2.

Distribution of articles classified in primary and secondary goal.

Sustainable Development Goals (SDGs)	N° articles with primary goal	% articles with primary goal	N° articles with secondary goal	% Articles with secondary goal
Goal 1: No Poverty	6	0.4%	3	1.3%
Goal 2: Zero Hunger	53	3.8%	8	3.4%
Goal 3: Good Health & Well-Being	633	45.3%	71	30.5%
Goal 4: Quality Education	111	8.0%	20	8.6%
Goal 5: Gender Equality	6	0.4%	2	0.9%
Goal 6: Clean Water & Sanitation	37	2.7%	7	3.0%
Goal 7: Affordable & Clean Energy	37	2.7%	5	2.1%
Goal 8: Decent Work & Economic Growth	26	1.9%	3	1.3%
Goal 9: Industry, Innovation, & Infraestructure	96	6.9%	24	10.3%
Goal 10: Reduced Inequalities	26	1.9%	9	3.9%
Goal 11: Sustainable Cities & Communities	119	8.5%	15	6.4%
Goal 12: Responsible Consumption & Production	26	1.9%	9	3.9%
Goal 13: Climate Action	33	2.4%	14	6.0%
Goal 14: Life Below Water	44	3.2%	6	2.6%
Goal 15: Life on Land	88	6.3%	9	3.9%
Goal 16: Peace, Justice & Strong Institutions	44	3.2%	25	10.7%
Goal 17: Partnerships	11	1%	3	1%
Total articles with SDGs	1,396	100%	233	100%

Table 3.

Distribution and frequency of articles assigned to each target of the 17 Sustainable Development Goals.

Target	.1	.2	.3	.4	.5	.6	.7	.8	.9	.10	.11	.12	.13	.14	.15	.16	.17	.18	.19	.A	.B	.C	.D	% use of total targets
Goal 1	1	-	-	5	3															-	-			43%
Goal 2	4	12	5	27	11															1	-	1		88%
Goal 3	10	23	33	351	13	3	12	36	5											8	110	59	41	100%
Goal 4	19	11	11	15	8	9	15													9	5	29		100%
Goal 5	1	1	-	-	2	1														1	1	1		78%
Goal 6	7	1	18	6	2	6														3	1			100%
Goal 7	5	13	3																	16	5			100%
Goal 8	3	4	4	1	6	-	1	6	1	1										2	-			83%
Goal 9	14	1	2	9	83															1	6	4		100%
Goal 10	-	18	9	2	-	-	5													1	-	-		50%
Goal 11	7	24	34	25	10	4	7													7	13	3		100%
Goal 12	-	11	3	4	2	-	1	-												11	2	1		73%
Goal 13	19	11	13																	-	4			80%
Goal 14	3	23	3	13	1	-	1													5	1	-		80%
Goal 15	33	10	9	12	21	-	-	6	4											1	-	1		75%
Goal 16	4	1	5	-	-	6	29	3	-	14										4	3			75%
Goal 17	-	-	-	-	-	-	1	2	1	-	-	-	1	-	-	-	1	7	1					37%
				= Color range, according to article count
		-		= Hyphen cell, indicates that the target were not used
				= Gray cell, indicates there is no Target for the Goal

Table 4.

Frequency of target ratio, according to number of items.

		Secondary target
		2.2	3.4	3.B	3.C	3.D	4.4	9.5	16.7
Primary target	3.1		4
	3.3			7		3
	3.4	3		8	14				4
	3.C						3
	4.4				3
	7.A							4

Table 5.

Distribution of the subset of articles (233), according to primary or secondary target and numerical or alphanumerical.

	Numeric target	Alphanumeric target	Total articles
Primary target	191	42	233
Secondary target	157	76	233

Table 6.

Number of “unclassified” articles in the 1st and 2nd review.

	Total articles
Unclassified after the 1st review	1,173
Unclassified after the 2nd review	983
Reclassified	190

Table 7.

Distribution of item reassignment in Sustainable Development Goals, after the second revision.

Sustainable Development Goals (SDGs)	N° articles	% Total reclassified
Goal 1: No Poverty	0	0%
Goal 2: Zero Hunger	9	5%
Goal 3: Good Health & Well-Being	85	45%
Goal 4: Quality Education	10	5%
Goal 5: Gender Equality	1	1%
Goal 6: Clean Water & Sanitation	5	3%
Goal 7: Affordable & Clean Energy	6	3%
Goal 8: Decent Work & Economic Growth	3	2%
Goal 9: Industry, Innovation, & Infraestructure	39	21%
Goal 10: Reduced Inequalities	2	1%
Goal 11: Sustainable Cities & Communities	14	7%
Goal 12: Responsible Consumption & Production	3	2%
Goal 13: Climate Action	1	1%
Goal 14: Life Below Water	0	0%
Goal 15: Life on Land	6	3%
Goal 16: Peace, Justice & Strong Institutions	5	3%
Goal 17: Partnerships	1	1%
Total reclassified	190	100%

Table 8.

Level of reading and analysis applied to each article.

Analysis level	Articles with Sustainable Development Goals	Articles unclassified	Total articles
Level 1. Analysis of the title and keywords	54	335	389
Level 2. Analysis of the abstract	966	558	1,524
Level 3. Analysis of discussions and conclusions	369	88	457
Level 4. Analysis of the results	7	2	9