Is the Paris Agreement effective? A systematic map of the evidence

The Paris Agreement (PA) sets out to strengthen the global response to climate change, setting targets for mitigation, adaptation, and finance, and establishing mechanisms through which to achieve these targets. The effectiveness of the PA's mechanisms in achieving its targets, however, has been questioned. This review systematically maps the peer-reviewed literature on the PA, categorizing the available evidence on whether or not the 'Paris Regime' can be effective. We split our analysis into three methodologically distinct sections: first we categorize the literature according to the mechanisms being studied. We find a diverse body of literature, albeit with a clear focus on mitigation, and identify adaptation and capacity building to be clear gaps. Second, we carry out a content analysis, identifying common drivers of, barriers to, and recommendations for effectiveness. Here we find mixed evidence, with potential drivers often qualified by more concrete barriers. Thirdly, we use scientometrics to identify six research clusters. These cover loss and damage, finance, legal issues, international politics, experimental evidence, and studies on tracking progress on the PA's targets. We conclude with a narrative discussion of our findings, presenting three central themes. First, transparency is widely considered a precondition for the PA to be institutionally effective. However, a lack of clear reporting standards and comparable information renders the PA's transparency provisions ineffective. Second, environmental effectiveness relies on national ambition, of which there is currently too little. It remains unclear to which extent the Paris Regime structure itself can induce significant ratcheting-up of ambition. Finally, the PA facilitates the diffusion of norms, enables learning and the sharing of best practices. This production of shared norms provides the most promising avenue for overcoming the current lack of ambition. One of the primary successes of the PA is in providing a platform for the exchange of experiences and ideas.

Export citation and abstract BibTeX RIS

Original content from this work may be used under the terms of the Creative Commons Attribution 3.0 license. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

The Paris Agreement (PA) presents an important opportunity to coordinate and strengthen the global response to climate change, setting global goals on mitigation, adaptation, and finance. It establishes a wide array of mechanisms through which to achieve these goals, ranging from the 'pledge and review' of nationally determined contributions, to the engagement of non-state actors in global efforts to address climate change.

Despite the diplomatic success of 195 member-states (MS) agreeing on such a consequential and legally binding text, the efficacy of the PA remains under intense scrutiny. For example, with the initial rounds of pledges severely lacking in ambition (United Nations Environment Programme 2019)—and global emissions continuing to rise (Friedlingstein et al 2019)—many are skeptical about the viability of a 'pledge and review' mechanism to ensure the necessary emissions reductions to keep global temperatures well below the 2°C threshold (and the substantially more ambitious 1.5 °C threshold) (Cramton et al 2017). Similar questions of efficacy emerge for other mechanisms detailed in the PA (Spash 2016, Pauw et al 2018, Schoenefeld et al 2018, Oh 2019). Therefore, ongoing negotiations on the Paris Rulebook and the subsequent operationalization of the many mechanisms the PA proposes to implement will be paramount in deciding whether or not the PA's targets can be achieved (Bodansky 2016).

Informing these developments, and assisting decision makers in the successful implementation of the PA's mechanisms thus remains a key task for academic research. Although research exists both supporting and questioning the efficacy of the PA, no attempts have been made to systematically synthesize this research field, with existing reviews either lacking systematic methods (Petticrew and Mccartney 2011, Minx et al 2017), or remaining too narrow in their focus (for an overview of existing reviews see page 4 of the protocol in the stacks.iop.org/ERL/15/083006/mmedia supplementary materials). We provide new evidence on the effectiveness of the Paris Agreement by systematically mapping the literature. To our knowledge this is the first application of systematic evidence synthesis to this area of literature. Further, we offer conceptual advances, assessing the PA according to drivers of, barriers to, and recommendations for effectiveness. Following a strict and transparent protocol, we create a comprehensive database of peer-reviewed literature on the PA that is non-trivial in scope and depth. We divide our subsequent analysis of this literature into three sections:

First, we systematically categorize each paper by the aspect of the PA being studied, gaining an overview of the coverage of research on the mechanisms established by the PA. We further categorize the literature according to its general appraisal of Agreement, identifying which documents depict the PA as a primarily positive or negative development, or offer a mixed appraisal, presenting both positive and negative aspects without favouring one over the other.

Second, we use content analysis to identify the key drivers of, barriers to, and recommendations for improving effectiveness. We define effectiveness here as whether or not the studied mechanism contributes to achieving the targets set out in Article 2 of the PA, namely; limiting global average temperature increase to 1.5°C–2°C, increasing resilience and the ability to adapt to the adverse impacts of climate change, and ensuring that financial flows are consistent with pathways needed to achieve both targets on mitigation and adaptation, and achieving these targets in an equitable manner (United Nations 2015, p 3).

Third, using bibliometric analysis we identify key epistemic communities studying the PA and their interaction.

We conclude with a narrative discussion of our results depicting what we see as the main arguments being made within the literature as to why, or why not, the Paris Agreement will prove successful in tackling the challenges of climate change. We further reflect on the method, presenting both its benefits in terms of comprehensiveness and transparency, and some limitations concerning its application to qualitative ex-ante policy assessment.

In the following section we very briefly summarize the mechanisms of the PA. We then outline the methodology used, present our results, and discuss these.

1.1. The Paris Agreement and its mechanisms: a brief overview

The Paris Agreement establishes a wide array of mechanisms through which to achieve its goals. We identify these within the Paris Agreement text, as well as its accompanying decision and Paris Rulebook and summarize the results in table 1.

Table 1. The Paris Agreement's mechanisms.

The central element of the PA is the 'pledge and review' mechanism whereby member states (MS) periodically submit nationally determined contributions (NDCs) that detail their intended climate action for a given period. These 'pledges' and their subsequent implementation are subject to review mechanisms designed to put pressure on states to both achieve their pledged contributions and to foster future pledges that are more ambitious (Falkner 2016, Keohane and Oppenheimer 2016). The NDCs should communicate efforts on mitigation, adaptation, finance, technology, capacity building, and transparency (Article 3 & 4, United Nations 2015, pp 3–6).

To aid MS in implementing their NDCs, the PA further enshrines the role of forests and the REDD + mechanism in achieving its targets on mitigation (Article 5, United Nations 2015, p 6), and establishes two mechanisms for the linkage of national climate policies. The first consists of a market-mechanism encouraging the international transfer of mitigation outcomes. Secondly a framework for non-market approaches to sustainable development is established with the aim of promoting ambition, enhancing non-state actor participation in the implementation of the NDCs, as well as enabling opportunities for coordination (Article 6, United Nations 2015, pp 7, 8). Although adaptation is also communicated through the NDCs, the PA puts further emphasis on the importance of adaptation in Article 7, establishing the adaptation communications as a means to track progress on national adaptation actions, and reiterating the importance of presenting national adaptation plans in order to guide this action (Article 7, United Nations 2015, pp 9–11). The PA addresses the issue of loss and damage, instituting the Warsaw International Mechanism for Loss and Damage associated with Climate Change Impacts as the primary means to facilitate understanding, action, support and cooperation on loss and damage (Article 8, United Nations 2015, pp 12, 13). The PA further establishes the UNFCCC's Financial Mechanism, constituted of the Green Climate Fund and the Global Environmental Facility as the primary financial mechanisms serving the PA 1 (Article 9, United Nations 2015, pp 13, 14, Zhang 2019). Acknowledging the importance of technologies and technology transfer for achieving the PA's goals, the UNFCCC's Technology Mechanism is further established to serve the PA, instituting a technology framework to support in the implementation of the PA's provisions (Article 10, United Nations 2015, pp 14, 15). The PA also establishes the Committee on Capacity-Building as the primary mechanism to review progress in the field, building on previous mechanisms under the Kyoto Protocol and Durban Forum (Article 11, United Nations 2015, pp 15, 16).

In order to review the NDCs and further information submitted by each MS, the PA establishes a transparency framework, setting rules for how nations are to report progress on the implementation of their NDCs, and outlining a process for assessing these reports (Obergassel et al 2019; Article 13, United Nations 2015, pp 16–18). Complementing the national review process set out through the transparency framework, the PA institutes a periodic assessment of global progress towards achieving the PA's goals, referred to as the 'Global Stocktake'. In contrast, the focus of the transparency framework lies in the technical reporting of national climate actions, not an assessment of the adequacy of these pledges (Milkoreit and Haapala 2019; Article 14, United Nations 2015, p 18). Finally the PA establishes a committee to facilitate the implementation of, and promote compliance with, the provisions of the PA, such as the structure and content of the NDCs. The committee is due to operate in a facilitative and non-punitive manner, resulting in advice and assistance for non-complying parties (Zihua et al 2019; Article 15, United Nations 2015, p 19).

Finally, the PA establishes two subsidiary bodies to assist in the governing of the PA, by providing information, and assisting in the assessment and review of implementation (Article 18, United Nations 2015, pp 21, 22).

Table 1 provides a full list of the mechanisms detailed in the PA and Decision 1/CP.21, and groups these for simplicity. Together these make up a complex 'regime' of interacting mechanisms. Although each mechanism's effectiveness can be assessed in its own right, in this review we focus exclusively on the common attributes of these mechanisms identified as key drivers, barriers and recommendations for the PA's overall effectiveness. We thus do not comment on the specific architecture of each mechanism. However, we encourage further use of the database of the relevant literature on each mechanism we have compiled, inviting further reviews on each mechanism separately.

This section presents the methods for identifying, selecting, and subsequently analysing the literature studying the Paris Agreement. We divide this up into:

• Literature search
• Screening for relevance
• Extracting relevant information

We conclude the section with a brief discussion of some difficulties we encountered in systematically synthesising the documents we identified as relevant.

2.1. Literature search

Our starting point for this study is to identify the relevant literature studying the PA and to justify our selection. Section 2 provided an overview of the PA's mechanisms, as detailed in the text of the PA, its accompanying decisions, and the Rulebook negotiated at the COP24 in Katowice last December. We use this list of mechanisms to iteratively develop a search query for the Web of Science and Scopus platforms, identifying any document within the encompassing databases that referenced the Paris Agreement (or an associated synonym) or one of the mechanisms identified (or an abbreviation of this mechanism) (see the review protocol published in the supplementary materials for the boolean search string used). We use a list of benchmark articles compiled through expert consultation in order to check the comprehensiveness of our search strategy.

We limit the date of publication to 2016 and onwards. Given the PA was concluded in December 2015 this ensures that the documents identified are relevant to the PA rather than previous climate agreements. We further exclude REDD + . This mechanism was operational long before the PA was being negotiated. Thus we found that most studies on REDD + focused on projects that precluded the PA, and were not relevant for our analysis of the PA's effectiveness. Finally, we are aware that restricting ourselves to the Web of Science and Scopus platforms limits the comprehensiveness of our search by excluding grey-literature. Our findings on existing research-gaps must therefore be qualified by the fact that we restrict ourselves to peer-reviewed 2 research for this study. Nonetheless, we maintain that uncovering a gap in the peer-reviewed literature remains an important and valid finding.

2.2. Screening for relevance

In order to ensure the relevance of the literature identified by our search we screen all documents identified by our search string at the title and abstract level using a strict set of inclusion and exclusion criteria. We include all documents explicitly studying the PA or one of its mechanisms (as identified in section 2), as well as analyses of analogous mechanisms with explicit reference to the PA, and analyses of the UNFCCC negotiations explicitly relating to a PA mechanism. We exclude studies focused on national/regional case-studies without explicit relation to the broader function of the mechanism they study. These cases often remain confined to their context, offering no comparable information on how the PA works more generally (i.e. case studies of individual GCF projects). We also exclude all studies on climate impacts, as these may make reference to the PA but do not directly relate to its function. Finally, we exclude ex-ante models of emissions or GDP scenarios, unless they explicitly relate to a function of a PA mechanism (i.e. the conducting of the Global Stocktake). This also excludes models of the likely warming effect of the submitted NDCs. Although such models technically study the effectiveness of the PA, they are deemed to be sufficiently synthesized in the annual Emissions Gap Reports (United Nations Environment Programme 2019), or similar assessments. For example, the most recent Emissions Gap Report depicts that based on current NDCs, we are on track to reach 56 Gt CO2e of Emissions by 2030, amounting to more than double the 25 Gt CO2e threshold cited necessary as limit global heating to 1.5 °C (United Nations Environment Programme 2019). To avoid replicating such existing synthesis we omit these studies from our research. We also exclude conference reports and book reviews. We test these criteria for consistency and clarity by screening random samples with multiple reviewers and subsequently discussing any resulting ambiguities.

We finally screen each remaining paper at a full-text level, resulting in a final database of 292 relevant documents published between January 2016 and June 2019 3 (see figure 1).

2.3. Information extraction and analysis

Our analysis of the documents comprises three distinct stages, with corresponding information extraction and analysis in each. First, we extract and compile the following information from each document:

• Meta-data (author, title, journal, year)
• Paris mechanism analyzed (see grouped mechanisms in table 1)
• General appraisal of the PA (Positive/Negative/Mixed/NA)

This basic information informs our systematic map of the Paris Agreement literature—a descriptive overview of the types of studies in this field, their main areas of investigation, common journals, and so forth. We source the meta-data directly from the Web of Science and Scopus platforms. We grouped the mechanisms used to inform our search query (table 1), and subsequently classify each document according to the mechanism group it studies. We added a 'general' category for relevant documents not explicitly studying only one of the mechanism groups, but rather covering more than one mechanism or the PA in general. 4 We finally classify the literature according to its overall appraisal of the Paris Agreement, distinguishing between documents that find the PA to be generally a positive development, negative development, or neither positive nor negative (mixed). We include an N/A category for documents that make no appraisal of the PA.

The second part of our analysis builds upon an assessment of drivers, barriers, and recommendations. In other words, we search for the main arguments made within the literature as to why, or why not, the Paris Agreement will prove effective in reaching its goals. Table 2 broadly defines what we mean by drivers, barriers and recommendations. Based on a first reading of all the documents, we iteratively develop a codebook for identifying detailed categories for drivers, barriers and recommendations: first we extract text excerpts from abstracts and conclusions that could be considered drivers/barriers/recommendations, then we develop common categories across these excerpts, and we refine our codebook in several rounds of coding sub-samples of 5–10 papers by all authors. Finally, the codebook (see appendix 1) was applied to all documents. We further differentiate between hypothetical/actual, as well as direct/indirect/distinct, 5 drivers and barriers. We only code the abstract and conclusions, reasoning that common arguments concerning the PA's effectiveness were most likely to be present in these sections.

Table 2. Description of drivers, barriers and recommendations.

The final part of our analysis applies scientometric methods to uncover the main epistemic communities conducting research on the Paris Agreement. Each document contains a list of references; we use this information to generate a bibliographic coupling network, identifying common patterns of referencing across the document set. In a bibliographic coupling network, two documents are coupled if they share at least two common references. We use the igraph Python package to display the network (using ForceAtlas2 layout) and perform the cluster analysis (Csardi and Nepusz 2006, Jacomy et al 2014), identifying groups of documents that tend to cite similar literatures. Combining these clusters with the categories we coded, we describe the mechanisms being studied by each cluster, as well as the primary drivers of and barriers to effectiveness identified by each epistemic group. Since not all documents share common references, our network is smaller than the total size of the document set (292 articles); we therefore make a careful distinction between this analysis and the broader literature in our results.

2.4. Caveats and intercoder reliability

Ensuring reliability in content analyses can be done through having at least two coders separately code the same units (in our case documents) (Krippendorff 2004). Having double coded a sample of 30 documents (10%) we found that our inter-coder reliability results were mixed. Our reliability was adequate for the more descriptive categories coding the mechanism being studied and the document's general appraisal of the PA. We were however not able to produce consistently reliable results for our coding of drivers, barriers and recommendations (See table 3 for an overview. Appendix 2 details reliability statistics for each driver and barrier). Despite months of testing our codebook we found that there were simply too many factors involved in this process that were subject to coder interpretation. We therefore divide our analysis into distinct sections, ensuring we do not conflate our less reliable results with those for which we have high reliability. Furthermore, we transparently offer our coding and codebook for reader scrutiny, thus making the interpretations we base our results on openly available. We strongly encourage inspection of these and welcome any comments by readers (see supplementary materials). We offer further reflections on these challenges in our conclusions.

Table 3. Inter-coder reliability scores.

Finally, it is important to stress that although our research approach provides for comprehensiveness and transparency, it remains a synthesis of existing knowledge. As such the added value of our findings is that they offer a truly comprehensive overview of existing peer-reviewed research on the PA, bringing together findings from a variety of fields, rather than in identifying completely new mechanisms or evaluating the validity of claims made with respect to the existence of specific mechanisms.

Our analysis comprises three distinct parts: First, we descriptively analyse the literature, presenting an overview of the mechanisms being studied, systematically mapping the literature, identifying key areas of focus as well as research gaps. Combining this with our coding of the documents' general appraisal of the PA, we begin to uncover potential patterns in the evidence for the PA's effectiveness.

Second, we provide a more in-depth analysis of the evidence on whether or not the PA is likely to be effective. Here we describe the results of our coding for drivers, barriers and recommendations. We identify key patterns, grouping the evidence to provide a simpler overview. However, given the low reliability of these findings we report them with lower confidence, separating them from the aforementioned systematic map.

Finally we present the results of our scientometric analysis, describing the epistemic clusters we identify, their links to one another, and the primary arguments they offer for whether or not the PA is/can be effective.

We present these results below. Our final discussion departs from the descriptive, quantitative and systematic analysis of the previous three sections. Instead, it develops our own qualitative analysis of PA effectiveness in a narrative building on our reading and analysis of the available academic research on the PA.

3.1. Systematic map: a descriptive overview of the literature

Our literature search resulted in 292 relevant papers studying the Paris Agreement. Of these almost half could be classified as covering the PA generally, with a further 18% (53 papers) focusing explicitly on the NDC's. The other mechanisms established by the PA receive decidedly less attention. Surprisingly, given the prominence of adaptation within the targets set by the PA itself, only four papers explicitly focus on the adaptation communications and national adaptation plans as separate mechanisms. We found no evidence of the literature we categorized as 'NDC' or 'General' tackling the issue of adaptation more substantively, with only ten papers examining adaptation in reference to one of the other identified mechanisms. Although no papers explicitly study the mechanisms established on capacity building we did find some evidence of capacity playing a role as a barrier to and recommendation for improving effectiveness.

Using the meta-data recorded we are able to identify the top ten Journals publishing research on the PA (see appendix 3). We find that these top ten Journals tend to cover a broad spread of mechanisms, with the most prominent journal, Climate Policy, covering all mechanisms except for non-state actors.

Concerning each document's general appraisal of the PA, we find that most of the literature evaluates the PA as mixed. Nonetheless, the literature on non-state actors stands out for its large proportion of positive appraisals, whereas the literature on the NDCs includes very few positive appraisals of the PA (see figure 3). We further find that the operational provisions of the PA tend to receive less positive appraisal than those provision not yet implemented. Alongside the NDCs, the technology and financial mechanisms of the UNFCCC are already operational, tracking and reporting MS progress. Although loss and damage provides little operative provisions within the PA itself, the Warsaw International Mechanism on Loss and Damage has been operational since 2013. Conversely, the PA's review mechanisms are only now being operationalized and many provisions of the Paris Rulebook are still being negotiated. Moreover, negotiations on the flexibility mechanisms have not yet been concluded. Whilst these are tentative findings, they do not speak kindly for the prospects of the PA to be effective as assessed by the academic literature, indicating that where there is operative experience with the PA's provisions, this experience is rarely positive.

3.2. Content analysis: drivers, barriers and recommendations

With most of the literature providing a mixed assessment of the PA, we aim to gain a better understanding of the specific factors driving or hindering the PA's effectiveness. We find a wide variety of common drivers, barriers and recommendations, depicted in table 4 below. The codebook in appendix 1 offers a detailed description of each driver, barrier and recommendation. Table 4 further depicts the number of times each driver/barrier/recommendation was coded providing some indication as to the importance of these. However, such an interpretation needs to be approached with caution. Certain drivers/barriers/recommendations may lend themselves to being included within a document. For example, the lack of ambition of current NDC's is a highly reported and salient topic, and is often used to provide context and motivation for a study analysing the PA. We thus preach caution when interpreting the relative weight of each driver/barrier/recommendation. Nonetheless we identify a number of trends upon which we elaborate in the following.

Table 4. Common drivers, barriers and recommendations (categories are not horizontally linked). *A = Actual, H = Hypothetical.

First, many of the common categories we identified recur as drivers, barriers, and recommendations. For example, transparency is the most commonly cited driver. It relates to the established review mechanisms and the need for transparent monitoring, reporting and verification (MRV) procedures (see appendix 1). However, transparency and MRV also appear as commonly cited barriers to effectiveness. Here the primary focus is on a lack of comparable information and clear reporting standards hindering an effective (and transparent) review process. Measurement comes up again as a common recommendation, with many documents detailing specific methods and indicators for measuring progress on climate policies.

A further example of recurring categories is differentiation which, as a driver, refers to the careful differentiation of responsibilities within the PA, moving beyond 'common but differentiated responsibilities' to include 'respective capacities' and 'national circumstances' (United Nations Framework Convention on Climate Change 2015a). However, this remains contentious with continuing conflicts over this differentiation of responsibilities remaining a frequently cited barrier, and two papers suggesting ways to overcome these conflicts. 'Experimentation/learning' is a further cited driver. It refers to the PA as a policy experiment, with MS and non-state actors reporting on, and subsequently learning from, each other's policy experiences. Conversely, a lack of opportunities under the PA regime for actors to gain feedback and learn from each other is referred to by the barrier 'feedback/learning'. The recommendation 'communication/learning' encompasses suggestions on how to better structure communication processes under the PA in order to enable better learning between actors.

We identify similar trends for the procedures established by the PA, which are referred to as both drivers and barriers. The same is true for international cooperation, indicating that authors disagree on the extent to which current international cooperation structures are actually driving, or rather are an obstacle to, effectiveness. Finally, some authors argue that the legal nature of the PA is a positive attribute driving effectiveness, however, many more documents also depict the PA's lack of legal stringency to be a primary barrier.

Second, excerpts pertaining to drivers/barriers are not always equally certain of their verdicts. Some drivers/barriers were communicated as actual or current drivers/barriers, whereas others were communicated as hypothetical, leaving open whether they would come to pass. We coded for this difference (see brackets in the 'Counts' columns of table 4), and found that drivers are more often depicted as hypothetical, with 46% of Drivers communicated as hypothetical, compared to only 24% of Barriers. This reflects that the factors driving the effectiveness of the PA have not yet been fully implemented and implemented measures have yet to have consequences that can be evaluated. In contrast, those factors hindering the PA's effectiveness are mostly presented as actual barriers to effectiveness. Hence it seems that the evidence on the hurdles the PA faces in order to be effective is stronger than the evidence for the PA being able to overcome these hurdles.

The literature nonetheless offers some insight into how to overcome these hurdles. Most prominently, we have collected 40 specific recommendations for how to collect and measure climate policy, overcoming barriers to an effective transparency and review mechanism (see appendix 4 for a comprehensive list). Here we identify a few recurring themes:

First, a number of papers recommend using a variety of different indicators, allowing science and other stakeholders to discuss their pros and cons, and thus enabling nations to choose indicators and methods best adapted to their specific context and capabilities (Magnan and Ribera 2016, Aldy et al 2017, Jacoby et al 2017, Höhne et al 2018, Winkler et al 2018). Second, we identify multiple suggestions to link the monitoring of climate action with monitoring of sustainable development (Sarr 2018, Chan et al 2019, Waisman et al 2019). The need to track NDC progress beyond mere emissions accounting was a further recurring topic (Iyer et al 2017, Nature Climate Change 2017, Jeffery et al 2018). We identify a wide variety of specific recommendations on how to structure the MRV process, ranging from requiring nations to include explanations of how progress on implementing adaptation plans is assessed (Morgan et al 2019) to including long-term mitigation strategies within the transparency framework of the PA (Mayer 2019). Finally a large group of documents recommends specific methods for tracking progress, primarily focusing on measuring mitigation efforts (Herrala and Goel 2016, Peters et al 2017, Kameyama and Kawamoto 2018, Craft and Fisher 2018a, Müller and Michaelowa 2019, Scotford and Minas 2019, Waisman et al 2019).

Beyond proposals for how to measure, or track progress on, climate action, recommendations present options for increasing and maintaining ambition, including revising the allocation of finance, pricing carbon, enhancing national capacities, forming minilateral climate clubs, and linking the climate and trade regimes. Further recommendations provide insights into how the PA 'Regime' could be developed in the future to enhance existing drivers for effectiveness such as introducing better communication and learning strategies, finding innovative forms of legal compliance, linking the three established review mechanisms, and furthering existing human rights provisions within the PA.

Finally, we identify 58 papers that recommend avenues for further research. Here we find an immensely diverse set of research questions on all aspects of the PA (for a comprehensive list of these divided by PA mechanism see appendix 5).

3.3. Bibliometrics

Our third and final analytical section uses scientometrics to identify different epistemic communities studying the PA. We use reference data from the documents in our database to generate a bibliographic coupling network, whereby two documents are coupled if they share at least two common references. This network is then clustered using a community detection algorithm, identifying groups of documents that tend to cite similar literatures. We find 6 distinct research clusters, labelling these by manually going through the documents in each cluster and identifying common research topics and methodological approaches. Figure 4 depicts these clusters and their relationship to one another, with each node denoting a document within our database and the linkages between the nodes indicating that two documents share at least two common references. Thus the distance between clusters can be used as a proxy for the extent to which these clusters are linked. Finally, the size of each node denotes the number of times that document has been cited overall. We label the most cited documents.

We further combine these clusters with our coded categories, depicting the most prominent mechanisms studied by each cluster, their general appraisal of the PA, as well as the most cited drivers, barriers and recommendations within each cluster (see table 5 and figure 5).

Table 5. The most studied mechanisms, and most cited drivers, barriers and recommendations within each cluster.

Interpreting the identified clusters comes with a caveat; they are not necessarily representative of the wider literatures on their topics. Therefor the results discussed below cannot be generalised beyond the papers depicted in our bibliographic network. The results from this analysis nonetheless offer some interesting, if tentative, insights:

Cluster 6 focuses on the experimental assessment of the PA and its mechanisms. We find that these experiments rarely result in a positive appraisal of the PA, rather calling for minilateralism as a means to overcome the current lack in ambition. The documents in Cluster 2 explore means for tracking progress on meeting the PA's targets. This cluster provides a pessimistic outlook on the PA, with significantly more negative than positive appraisals. The literature in the cluster commonly cites technology and transparency as key drivers of effectiveness, but also references the lack of ambition, and problems with MRV as primary barriers. Interestingly these two clusters share few common references, despite both focusing on how the pledge and review process functions. Clusters 4 and 5 form two highly related clusters on climate finance and loss and damage respectively. They are strongly linked by common reference to loss and damage, with a large part of the literature in Cluster 4 on climate finance focused on finance for loss and damage. Finally the two centrally depicted Clusters (1 & 3) focus on the Paris Agreement more generally, with Cluster 1 encompassing insights from general international politics, and Cluster 3 retaining a legal focus. Both offer mixed appraisals of the PA with the legal literature featuring a particularly high number of positive assessments. Whereas the international politics literature highlights the importance of national and non-state action, the legal literature presents the transparency provisions and their legal nature as primary drivers. Both, however, also highlight the current lack of ambition as a significant barrier.

Considering the results and analysis above we conclude by discussing primary insights from our research. First we identify a number of research gaps, as well as areas for consolidation. Second, we offer some narrative insights we gained from having read and categorized all these documents. These insights steer away from a systematic analysis of the literature presented above, rather offering our own interpretation of the most important arguments made for why, or why not, the PA is effective. Finally we reflect on our novel application of systematic evidence synthesis methods to collect and analyse the literature on the PA, offering insights into the added value of applying these methods, and some potential limitations.

4.1. Research gaps

The literature we identify on the PA largely focuses on the PA in general, and on the NDCs (See figure 2). Given the PA remains in its infancy, with negotiations on the operationalization of many of its provisions still ongoing, the relative absence of literature on the other mechanisms established by the PA is unsurprising. In this sense it is positive that so much literature already exists studying the NDCs, and, as the other mechanisms become operational, the volume of research on these can be expected to increase. However, the lack of research explicitly studying the adaptation provisions within the PA is a gap. This is mitigated somewhat by a number of papers considering adaptation as part of the PA generally (Mathur and Mohan 2016, Lyster 2017, Sharma 2017, Hall and Persson 2018, Dovie 2019), the PA's finance provisions (Sovacool et al 2017), the NDC's (Atteridge et al 2019), and the review mechanisms (Tompkins et al 2018, Craft and Fisher 2018b). However, given the prominence of adaptation as a standalone goal under Article 2 of the PA, the lack of peer-reviewed literature on this topic is striking.

Even more striking is the complete lack of documents explicitly studying the mechanisms on capacity building. We find some evidence of capacity building playing a role as a barrier and recommendation, most of which focuses on the need for greater capacity for transparency and review (Brechin 2016, Millar et al 2016, Umemiya et al 2017, Winkler et al 2017, Tian and Xiang 2018, Tompkins et al 2018), financial accounting (Roberts et al 2017, Sovacool et al 2017, Weikmans and Roberts 2019), and technology (Puig et al 2018, Romijn et al 2018, Harwatt 2019, Hofman and van der Gaast 2019). However, this only serves as further support for the need for more research on the ways in which the Paris Committee on Capacity Building can overcome these barriers and incorporate existing recommendations. In light of the current emissions gap, new research on strengthening capacities to increase ambition seems to be a strong desideratum.

Beyond filling these two clear gaps, our bibliometric analysis offers some further areas with potential for consolidation. First, there is a clear lack of connection between the literature providing experimental evidence for the pledge and review process's effectiveness (Cluster 1), and the literature focused on a more practical analysis tracking progress on achieving the PA's targets (Cluster 2). Both clusters focus on the pledge and review process, and whilst they employ different analytical lenses, they could offer important insights to one another. The fact that they do not cite similar literatures implies that this is not yet happening. The same is true of the literature on climate finance (Cluster 4), and the literature studying the reporting and monitoring of NDC's in line with the PA's targets (Cluster 2). With many of the NDCs contingent on financing (Zhang and Pan 2016; Kissinger et al 2019), it is somewhat surprising that the literature tracking progress on the NDCs does not link to the literature on climate finance more closely.

4.2. Is the Paris Agreement effective?

Considering our results above we identify three main arguments made for whether or not the PA is effective. Borrowing from Dimitrov et al (2019) we distinguish between institutional and environmental effectiveness, with institutional effectiveness denoting that the mechanisms established by the PA are robust and function effectively, and environmental effectiveness denoting whether or not the PA's targets are ultimately met.

Concerning institutional effectiveness; we find that transparency is widely considered an imperative institutional precondition for the PA to be effective. The 'pledge and review' process, by which national climate action is to be coordinated and its ambition periodically increased, relies primarily on a transparent review of national pledges in order to both effectively track progress towards the PA goals, and apply scrutiny on member-state's climate policies. Thus, an institutionally effective PA is one that ensures the periodic submission of increasingly ambitious and comparable pledges. Implementation of these is transparently monitored and reported on, with the stocktake providing periodic accounts of collective action. However, whilst transparency is evidently a primary driver of the PA's institutional effectiveness, it coincides with extensive reference to MRV as a barrier to such effectiveness. Here the literature references a lack of comparable information and clear reporting standards as hindering the transparent review of member-state's climate actions. While a large number of documents recommend ways to overcome this barrier, detailing methods to measure progress on the PA's goals, the promise of transparency, and by extension 'pledge and review', clearly comes with a caveat; existing means of review are not yet effective, but could become so if subsequent negotiations deliver sufficient outcomes and barriers are overcome.

In terms of environmental effectiveness the PA relies entirely on national and non-state actions in order to meet its targets. Even under an institutionally effective agreement, submitted and implemented pledges may simply not be ambitious enough to reach the PA's targets, and civil society and non-state action may be unable to make up the missing gap. 6 Indeed, current levels of ambition fall far short of what is needed to achieve the Paris Agreement's goals with the literature making extensive reference to the lack of ambition, not only in existing NDCs, but also citing a general lack of funding and the withdrawal of the United States as primary barriers to effectiveness.

Ensuring the PA's institutional effectiveness (for example by overcoming barriers to transparency) alone may not be enough to achieve its targets. Here we identify an intermediary channel whereby the PA influences national, and non-state, action, or environmental effectiveness. The PA is consistently presented as a significant normative shift with (all) nations agreeing on the pressing nature of the climate problem, and recognising the need for collective action that goes beyond just the nation state. Moreover, it institutionalises new elements such as Loss and Damage and Human Rights, expanding the ways in which the climate problem is approached, and opening new doors for climate action such as human rights litigation or the need for orderly migration procedures that go beyond the refugee convention. The PA thus sends a signal to private and public actors alike, helping to diffuse new ideas, setting a common direction, and helping maintain momentum supporting climate action. Aiding this, the PA establishes a number of processes allowing for experimentation and learning, helping spread best-practices and finding innovative solutions to the climate problem. Thus, while this process of feedback and learning remains imperfect and underdeveloped, one of the primary successes of the PA is in providing a platform for the exchange of experiences and ideas. As such it increases the salience of climate change around the world, aids in tipping global attitudes towards climate action, and enables the diffusion of solutions, facilitating the rapid transformations needed to achieve the PA's targets. Thus, over time the PA may develop to provide a future platform for creating the still lacking but necessary ambition.

In summary: in order to be institutionally effective the PA must overcome barriers to transparency. However, overcoming these barriers does not inevitably lead to more ambitious national and non-state climate actions. Here the PA's properties of norm and value diffusion, and experimentation and learning, play an important role. To enhance environmental effectiveness of the Regime, more substantial reforms might be required (e.g. implementing institutions that reduce free-riding by substantially altering the incentives of states based on concepts like reciprocity (Kornek and Edenhofer 2020)).

4.3. Methodological reflections and limitations

To our knowledge, this study is the first application of systematic evidence synthesis to a body of literature assessing, ex-ante, an international political regime. As such we had little previous research experience upon which we could build. Therefore, we offer some reflections on the method as applied to ex-ante policy assessments, highlighting both its benefits and draw-backs.

Using systematic methods to collect relevant literature adds tremendous value to the process of carrying out a review, overcoming the selection-bias of traditional reviews. Through agreeing on a clear set of screening criteria and screening such a large corpus of abstracts for relevance, the authors also gain a much better insight into the general research landscape, helping orient them and further define their research focus.

Once all relevant literature has been collected, a manual coding of these according to broad descriptive categories is very useful in providing an overview of this research landscape, and identifying key gaps. However, we found manual coding to be time-consuming, and to require a lot of careful consideration from all involved authors. In this sense, if the focus of the research does not go beyond providing a broad overview of the research landscape, using computer assisted methods such as topic modelling provides a less work intensive alternative (Lamb et al 2019).

For this project we wanted to go beyond a broad description of the research landscape, and synthesise the evidence on whether or not the PA is/can be effective. Our conceptualisation of effectiveness offers a novel way to synthesise qualitative policy assessments. The use of common categories to synthesise the literature's findings offers a transparent and objective method for review. However, we also identify significant limitations.

We found such a systematic synthesis of qualitative ex-ante policy assessments to be methodologically difficult. Most of the mechanisms established by the PA are not yet operational, and so can only be assessed ex-ante. As such, much of the literature we review does not explicitly frame its findings in terms of effectiveness. Apart from the analyses of the ambition of existing NDCs, little aggregable data on the PA's effectiveness exists. This made it challenging to systematically synthesise this research. Identifying common drivers, barriers and recommendations was therefore subject to quite some interpretation, a task that is further complicated by the complexity of the PA itself and the diverse epistemic communities studying it. Whilst our findings remain insightful, the lack of inter-coder reliability in this part of the analysis is a significant limitation. Furthermore, the effort needed to iteratively develop a codebook, and subsequently code each document, renders such a task limited in its scalability. At least, it requires planning with significant resources for the coding exercise from the start (proposal) phase of the project.

Thus far qualitative syntheses have been primarily carried out through a more narrative form of review (e.g. Dimitrov et al 2019). Although these reviews provide immensely valuable insights into the existing evidence, 7 the rapid increase in both the volume and diversity of climate related literature has questioned the ability for such reviews to remain comprehensive and transparent (Petticrew and Mccartney 2011, Minx et al 2017). Our experience has shown that in order to answer the call for more systematic evidence synthesis on policy processes we need better systematic methods for categorizing and collating qualitative policy assessments that are scalable to be able to overcome the challenge of 'big-literature'. Advances in big-data methods offer some important opportunities here (Minx et al 2017, Lamb et al 2018, 2019).

5. Conclusions and open questions

To conclude, we find a large and diverse body of literature studying the PA. Adaptation and capacity building stand out as two clear research gaps in the literature, and a number of areas exist that might benefit from more consolidation. As of yet, there is no consensus on whether the PA will be effective. Most of the literature presents mixed results, citing a wide variety of drivers and barriers supporting, and hindering, the PA effectiveness. We find that, in general, the barriers cited are communicated more strongly, with drivers often cited as hypothetical. By and large this indicates that, in its current state, the PA is unlikely to enable the necessary conditions to achieve its targets. However, the PA remains in its infancy, with many provisions not yet implemented, and plenty of scope for adjusting provisions once first experiences can be reflected on. Hence barriers such as the lack of comparable information or clear reporting standards may yet be overcome enhancing the PA's institutional effectiveness. Nonetheless, we find that the most significant obstacle to ensuring the PA achieves its targets remains the current lack of ambition. Only if national and non-state ambitions are significantly increased and sincerely implemented, can the PA be environmentally, as well as institutionally, effective. Here, the PA's diffusion of norms and values, and its properties as a platform for periodic exchange and learning are key. Further research should explore these properties further, assessing ways to enhance their impact on ambition, and coming up with suggestions for how to further develop the PA's mechanisms to facilitate this.

Beyond the Paris Agreement, further research is needed studying national/regional processes for deciding on, and subsequently monitoring and reforming, climate policies. Although not included in this study, we found a number of such case studies while searching for relevant literature on the PA (e.g. Amjath-Babu et al 2019, Baek et al 2019, Boehnke et al 2019, Gallo and Albrecht 2019, Mohan and Wehnert 2019, Selvakkumaran and Silveira 2019, Simsek et al 2019). 8 A further synthesis of these documents would provide important insights. Beyond assessing how ambition can be raised nationally, given the uncertainties surrounding the PA's effectiveness, it is pertinent to examine the adequacy of the existing international cooperation processes. However, we find no evidence of such a discussion taking place, with very few papers questioning the adequacy of the UNFCCC and COP processes for enabling global climate action. We urge further explorative research here, and especially encourage collaboration with researchers assessing other areas of international relations and law.

The PA remains the primary means by which climate policy is coordinated internationally. Considering our findings, the prospects for the PA to deliver on achieving its targets seem slim. However, the PA enshrines the role of domestic, regional, and local climate action, leaving it up to governments, businesses and citizens to implement the policies and behavioural changes necessary to address climate change. Unlike the Kyoto Protocol preceding it, it does not define who should do what, but rather offers a platform through which all these actors may communicate, collaborate and learn from each other. Perhaps it is therefore imprudent to judge the PA predominantly on effectiveness criteria; not least because the counterfactual may have been a legally binding solution with drastically reduced participation. Perhaps it is most important that the PA (and the UNFCCC more generally) offers a forum for multilateral and multilevel exchange, where all countries have a voice, and tackling climate change remains the primary focus.

We would like to thank Jan Minx for his support and determination in promoting systematic evidence synthesis, Max Callaghan for his tireless work in keeping the APSIS platform working, the Hertie PhD's for their valuable advice, everyone at the MCC for their constant support, as well as our friends and families.

Any data that support the findings of this study is included within the supplementary information.