High Income But Poor COVID-19 Outcomes: The  Contrasting Results of The Pre-COVID 19 Joint  External Evaluation Assessment Tool in Measuring Countries’ Global Health Security Preparedness and Response Capacities

Alam N; Lee P; Alam M; Chu C

High Income But Poor COVID-19 Outcomes: The Contrasting Results of The Pre-COVID 19 Joint External Evaluation Assessment Tool in Measuring Countries’ Global Health Security Preparedness and Response Capacities

by Alam N^1,2*, Lee P^1,3, Alam M⁴, Chu C¹

¹Centre for Environment and Population Health, Griffith University, 170 Kessels Road, Nathan, Queensland 4111, Australia

²Queensland Health, 15 Butterfield Street, Herston, Queensland 4006, Australia

³Department of Medical Research, China Medical University Hospital, Taichung City, Taiwan.

⁴Queensland University of Technology, Garden Point Campus, 2 George St. Brisbane, QLD 2000

*Corresponding author: Alam N, Centre for Environment and Population Health, Griffith University, 170 Kessels Road, Nathan, Queensland 4111, Australia.

Received Date: 11 June, 2024

Accepted Date: 18 June, 2024

Published Date: 21 June, 2024

Citation: Alam N, Lee P, Alam M, Chu C (2024) High Income But Poor COVID-19 Outcomes: The Contrasting Results of The PreCOVID 19 Joint External Evaluation Assessment Tool in Measuring Countries’ Global Health Security Preparedness and Response Capacities. Rep GlobHealth Res 7: 202. https://doi.org/10.29011/2690-9480.100202.

Abstract

Background: The unprecedented global impact of the COVID-19 pandemic heightened the concern over countries’ preparedness and response capacities for dealing with pandemics. One of the global assessment tools to measure these capabilities is the Joint External Evaluation (JEE), with 52 indicators in 19 technical areas, contained in WHO’s International Health Regulations (IHR). Under the JEE, a higher score indicates a higher pandemic response capacity. Thus, we examined countries’ pre-COVID-19 preparedness and response capacities in accordance with their pre-pandemic JEE score, and compared it with their income status and COVID-19 outcomes. Methods: We extracted pre-COVID-19 country-specific JEE reports for 95 countries and COVID-19 cases and deaths data from the World Health Organization’s databases as of 31 May 2020 and 9 June 2023 respectively. We grouped the JEE countries according to the World Bank country classifications by income level. Multivariate regression analysis was performed to examine the relationship between JEE scores and countries’ COVID-19 cases and deaths, after accounting for countries’ income status according to the World Bank income classification of countries. Results: Among the 95 countries, 67 (71%) had limited or no capacity to respond to a health emergency such as a pandemic (JEE scores 2 and 1 respectively). The capacities varied substantially across countries with only 5% of the high-income countries having limited or no capacities (JEE score <3) compared to all (100%) of the low-income countries having the same (p<0.001). Importantly, there was a strong positive correlation between JEE scores and COVID-19 cases and deaths (rho coefficients, =0.72 and =0.66 respectively, both p<0.001). This finding challenges the conventional wisdom that higher income and higher JEE scores lead to better outcomes. In fact, high- and middle-income countries with higher JEE scores experienced higher cumulative COVID-19 cases and deaths compared to low-income countries (p<0.001), highlighting the need for a more nuanced approach to preparedness. Conclusions: The JEE is intended to reflect countries’ capacity to prevent, detect and respond to health threats. The higher the JEE score a country has, the better its pandemic outcomes should be, in terms of cases and mortality. We examined the pre-COVID JEE of 95 countries, and found that higher-income countries tend to have higher JEE scores. However, when we compared countries’ pre-COVID-19 JEE scores and income levels with their performance during the COVID-19 pandemic, the JEE scores seemed to contradict countries’ actual performance. Many high-income countries with high JEE scores performed poorly, worse than lower-income countries. This highlights the need for health security preparedness assessment tools, such as the JEE, to be continuously reviewed in light of the COVID-19 pandemic, and on an ongoing basis following every major health security event. They need to be purpose-built and outcome-focused so that they can meaningfully assist countries to better prepare for future pandemics.

Keywords: Global Health Security; COVID-19; Pandemic; Capacity; Joint External Evaluation; International Health Regulations.

Introduction

The 2003 outbreak of novel coronavirus disease, severe acute respiratory syndrome (SARS), which originated in Southeast Asia, rapidly spread over continents, causing over 8,000 human infections and 774 deaths [1]. The first global public health emergency of the 21st century [2] quickly swept through more than 20 countries, demonstrating the human vulnerability to emerging pathogens of epidemic and pandemic potential. The relatively easy mode of airborne transmission facilitated by intensive population movements in the era of globalization contributed to the swift spread of the virus across three continents within a short span of time. The speed of transmission and the severity of the emerging disease proved that countries, irrespective of the strength of their health systems, were ill-prepared to respond to a novel pathogen of pandemic potential.

The 2003 SARS epidemic prompted numerous global efforts to improve pandemic preparedness recognizing the need for strengthening countries’ preparedness and response capacities against future epidemics and pandemics [3]. One noteworthy initiative was the 2005 revision of the existing International Health Regulations (IHR 1969) to include more stringent provisions for “state party obligations to develop certain minimum core public health capacities [2].” Furthermore, there has been a proliferation of mechanisms and tools for assessing country preparedness and response capacities and recommendations for improvement [4-6]. Two such global assessment instruments have been the Global Health Security Index [7] and the Joint External Evaluation (JEE) of the International Health Regulations (2005) [8].

The JEE is a collaborative process between WHO, the host nation, and external partners to assess the country’s readiness and capacity to prevent, detect, and rapidly respond to public health threats under the International Health Regulations (2005). Voluntary for countries, the JEE exercise is intended to help countries assess their capacities, identify and fill any gaps and prioritize opportunities for strengthening preparedness and response [9]. The JEE tools consist of 19 technical areas with 52 indicators within the IHR’s four core capacity areas: (1) prevention (2) detection (3) response and (4) other Hazards (IHR-related hazards and points of entry). The JEE results provide critical information about the country’s capacities so they know where their capacities lie and what actions to undertake to fill any gaps in their current preparedness and response capacities [10].

Almost two decades after the 2003 SARS epidemic and after almost 100 countries completed their first voluntary JEE, the severe acute respiratory syndrome coronavirus 2 (COVID-19) pandemic provided an ideal opportunity to reflect on the country’s pre-pandemic JEE-assessed capacities to prevent, detect and respond to a pandemic. We examined the extent to which JEE country-specific scores reflected the countries’ observed status of preparedness and response capacities as evidenced in their COVID-19 cases and deaths. Specifically, we sought to answer two questions: firstly, whether JEE scores were biased towards the income status of the counties, and secondly, how the pre-pandemic JEE scores reflected upon the countries’ COVID-19 performance as measured by COVID-19 cases and deaths.

Methods

On 31 May 2020, we extracted country-specific Joint External Evaluations (JEE) reports available from the WHO’s publicly available JEE database [8]. We manually extracted JEE data for each of the countries and stored them in Microsoft Excel spreadsheets for analysis. We reviewed and analyzed the JEE results measured against 52 indicators for each of the countries. Due to completeness limitations (four indicators were not assessed by JEE for most countries), 48 out of 52 indicators were included in this study. The JEE assessed country capacities with a score of 1 to 5, where 1=no capacity, 2=limited capacity, 3=developed capacity, 4=demonstrated capacity, and 5=sustainable capacity. We used a composite score of 3 by averaging all the JEE scores for each of the 48 indicators for each country. We considered a JEE score of 3 as a benchmark of minimum capacity, below which countries have “limited” or “no capacity” (JEE scores of 2 and 1 respectively) to respond to a pandemic. We grouped JEE countries according to the World Bank country classifications by income level: 20202021 (high-income, upper-middle income, lower-middle income, low-income) [11]. We used the cumulative number of COVID-19 cases and deaths (per 100,000 populations) to determine country capacity as denoted by the JEE scores.

After grouping JEE countries by the World Bank country classifications by income level, we compared the countries’ JEE scores with their corresponding cumulative COVID-19 cases and death data sourced from the WHO COVID-19 as of 9 June 2023 [12]. EpiInfo (version 7.2.5) StatCalc function and SPSS statistics software package (v.29) were used to conduct descriptive, bivariate, and multivariate statistical analyses. As JEE mean score and the three outcome variables (COVID-19 cases per 100,000, COVID-19 deaths per 100,000, and COVID-19 case fatality rate) were not normally distributed, non-parametric bivariate analyses (Kruskal-Wallis multi-sample tests and Spearman correlations) were conducted to examine the relationships between country income category according to the World Bank income classification, JEE mean score and each of the COVID-19 outcome variables. In addition, non-linear (exponential and quadratic) models were performed to fit the data concerning the relationships between JEE mean score and each of the outcome variables. Hierarchical multivariate regression models were developed to fit log-transformed data of the COVID-19 outcomes, considering the combined effect of income level and JEE score on each of the outcomes.

Results

As of 31 May 2020, pre-COVID-19 country-specific JEE mission reports were available on the WHO website for 96 WHO member states and territories, with the earliest JEE mission completed on 22-25 February 2016 (Tanzania) and the latest on 8-12 July 2019 (Guinea-Bissau). Of them, Zanzibar was excluded as there was no data available at the World Bank income classification level. Of the 95 countries retained for analysis, 43 (45%) were from WHO Africa Region, 17 (18%) were from the Eastern Mediterranean Region, 14 (15%) were from the Europe Region, 11 (12%) were from the Western Pacific Region, eight (8%) were from the SouthEast Asia Region, and two countries (2%) were from the America Region. Overall, 67 out of 95 countries (71%) had limited or no capacity to respond to a pandemic as measured by the JEE (JEE scores 2 and 1 respectively). In the African region, only one country (2%) was assessed to have a ‘developed’, capacity (JEE score of 3), compared to the same for all (100%) of the countries in the Americas and 71% of countries in Europe regions. A similar pattern of JEE scoring scheme was evident in JEE countries, with 95% of high-income countries assessed to have higher capacity (JEE scores of 3 or above) compared to 29% of upper-middleincome, 12% of lower-middle income countries and none (0%) of the low-income countries (Table 1).

WHO Regions	World Bank country classifications by income level								Total countries	% of countries with higher capacity (JEE score of 3 or more) by WHO Region
WHO Regions	Low income		Lower-middle income		Upper-middle income		High income
	JEE score <3	JEE score ≥3	JEE score <3	JEE score ≥3	JEE score <3	JEE score ≥3	JEE score <3	JEE score ≥3
Africa	23	0	14	0	4	1	1	0	43	2%
Americas	0	0	0	0	0	0	0	2	2	100%
Eastern Mediterranean	2	0	4	1	4	0	0	6	17	41%
Europe	0	0	1	1	3	3	0	6	14	71%
South-East Asia	0	0	5	1	1	1	0	0	8	25%
Western Pacific	0	0	5	1	0	0	0	5	11	55%
Total	25	0	29	4	12	5	1	19	95
% of countries with a benchmark capacity (JEE score 3 or more) by WB income category		0%		12%		29%		95%
JEE score <3 = limited to no capacity. JEE score ≥3 = developed to sustainable capacity.

Table 1: Countries with a JEE before the COVID-19 pandemic by WHO Region and the World Bank income classifications.

The JEE-assessed capacities varied substantially across countries classified by the World Bank income categories with only 5% of the high-income countries assessed by JEE demonstrating limited or no capacities (JEE score <3) compared to the same by all (100%) of the low-income countries (p<0.001) (Table 2).

Income classifications	Number of countries with JEE mean scores for capacity			% of countries with limited or no capacity (JEE mean score <3)	Chi-sq for linear trend (p-value)
Income classifications	<3	≥3	Total		Chi-sq for linear trend (p-value)
High-income	1	19	20	5%	55.82 (<0.001)
Upper middle income	12	5	17	71%
Lower middle income	29	4	33	88%
Low-income	25	0	25	100%
Total	67	28	95	71%

Table 2: Countries with JEE assessment mean scores of capacities with their income classification level

As of 9 June 2023, a higher cumulative number of COVID-19 cases and deaths were observed among the high-middle-income countries compared to the low-income countries (Figure 1).

Figure 1: JEE assessed countries (n=95) with their income status according to the World Bank income classifications and the countries’ COVID-19 outcomes measured by COVID-19 cases and deaths as of 8 June 2023. HI=High income countries, UMI=Upper middle income, LMI=Lower middle income, LI=Low income

Figures 2(a) to 2(c) present the distributions of JEE mean score and the three COVID-19 outcomes by income category. It is evident that JEE mean score increases with income levels. The differences in JEE mean score among different country income groups were statistically significant (p<0.05, Kruskal-Wallis and post-hoc test results not shown). In addition, higher-income countries seem to have significantly higher levels of cumulative COVID-19 case numbers (p<0.001 for the Kruskal-Wallis test and all between-group comparisons. Test results not shown). The upper-middle-income countries have the highest mean cumulative COVID-19 deaths, followed by high-income countries, while the two low-income country groups have a relatively low average number of COVID-19 deaths (p<0.001 for all between-group comparisons).

Figure 2: Distribution of – a. JEE country mean score by income category; b. COVID-19 cases per 100,000 by income category; c. COVID-19 deaths per 100,000 by income category.

The results of Spearman correlation analyses (Supplementary Table S1) indicated strong positive correlations between JEE mean score and COVID-19 cases and COVID-19 deaths (rho coefficients, ρ=0.72 and 0.66 respectively, both p<0.001). Supplementary Figures S1-S2 illustrate exponential relationships between JEE mean score and COVID-19 outcomes. It is noted that the plots with JEE score >3.0 scatter away from the exponential line in the models for COVID-19 cases and deaths. As both COVID-19 outcome variables are positively skewed, they were log-transformed to improve data distributions for further multivariate modelling processes. Figures 3(a) and 3(b) present the quadratic models created to fit the log-transformed COVID-19 outcomes (y-axis) in relation to the changes in JEE mean score by country income level. The results confirm the positive relationships between JEE scores and COVID-19 cases and deaths as presented below. The figures also demonstrate income gradients along with the changes in JEE score and COVID-19 outcomes.

	JEE mean score	COVID-19 cases	COVID-19 deaths	COVID-19 CFR
JEE Country mean score	-	0.723^***	0.660^***	-0.384^***
COVID-19 cases	0.723^***	-	0.880^***	-0.586^***
COVID-19 deaths	0.660^***	0.880^***	-	-0.19
COVID-19 case fatality rate (CFR)	-0.384^***	-0.586^***	-0.19	-
Note: *** p<0.001

Supplementary Table S1: Spearman correlations between JEE mean score and three COVID-19 outcomes.

Supplementary Figure S1: Preliminary model fitting to describe the relationship between JEE mean score and COVID-19 cases.

Supplementary Figure S2: Preliminary model fitting to describe the relationship between JEE mean score and COVID-19 deaths.

Figure 3: Quadratic model describing the relationship between JEE score and log-transformed - a. COVID-19 cases; b. COVID-19 deaths.

Taking into account the strong association of income category with JEE score and all COVID-19 outcomes, a series of non-linear models were performed to assess the adjusted associations between JEE score and COVID-19 outcomes. The results of the final quadratic model (A) in the first part (top) of Table 3 demonstrate that JEE score (including the quadratic term) makes a significant contribution (p<0.001) in predicting the changes in COVID-19 cases after controlling for the influence of income level. The positive coefficient (2.28, 95% CI: 0.65, 3.90) for JEE score reflects its overall positive relation with COVID-19 cases; while the negative coefficient (-0.34, 95% CI: -0.61, -0.07) for the quadratic term (JEE squared) suggests that the slope of the curve is flattened toward the high values e.g. JEE scores of >3), meaning a weaker relation toward higher JEE scores (also refer to Figure 3(a). The results in the bottom part of Table 3 (Model B) regarding COVID-19 deaths are in accord with those in Model (A). The model indicating the relationship between JEE score and COVID-19 deaths becomes negative for high values (e.g., JEE >4, also see Figure 3(b) as shown in the negative coefficient (-0.53, 95% CI: -0.80, -0.26) for the quadratic term. Both models perform well, explaining 75% (R2=0.75) and 64% (R2=0.64) of the total variances in COVID-19 cases and COVID-19 deaths respectively.

Model (A) Effect of JEE score on COVID-19 cumulative cases
Independent variable	B (95%CI)	Std. Error	β	t
Income category	1·44 (1·12, 1·76)***	0·16	0·77	8·85
JEE Country mean score	2·28 (0·65, 3·90)**	0·82	1·02	2·78
Squared JEE mean score	-0·34 (-0·61, -0·07)*	0·14	-0·92	-2·51
(Constant)	1·11 (-1·13, 3·36)	1·13	-	0·99
Model (B) Effect of JEE score on COVID-19 cumulative deaths
Independent variable	B (95%CI)	Std. Error	β	t
Income category	1·16 (0·84, 1·49)***	0·16	0·74	7·08
JEE Country mean score	3·21 (1·57, 4·86)***	0·83	1·71	3·89
Squared JEE mean score	-·53 (-0·80, -0·26)***	0·14	-1·70	-3·86
(Constant)	-3·92 (-6·18, -1·65)***	1·14	-	-3·43
Note: Overall R²=0·75 and 0·64 for predicting COVID-19 cases and deaths respectively; Significance levels: * p<0·05, p<0·01, * p<0·001

Table 3: Results of the final quadratic models in explaining the effect of JEE mean score on the variation in two COVID-19 outcomes (log-transformed).

Discussion

In the first two decades of the 21st century, the world experienced several epidemics and pandemics, which not only caused formidable losses of human lives, well-being, and economics worldwide but also provided many valuable lessons [13]. One such important lesson was that no country was fully prepared for an emerging disease such as the COVID-19 pandemic. Taking lessons from the past SARS and Ebola outbreaks, the JEE tool was revised in 2018, [14] with the aim of evaluating the countries’ capacities to prevent, detect, and rapidly respond to significant public health events. Thus, the evaluation criteria were expected to be more robust, reflecting on countries’ capacities independent of their socio-economic status. However, an explosive growth of COVID-19 cases and deaths in some high-income countries [15] that scored high (JEE scores of 4 and 5 – “demonstrated” and “sustainable” capacity respectively) in pre-COVID-19 JEE was remarkable. Our findings indicated that JEE capacities are consistent with the countries’ income status, with higher-income countries showing higher capacity and readiness to respond to a pandemic than lower income countries (Figure 2(A). Our study has also revealed a positive curvy relationship between JEE-assessed capacities and COVID-19 cumulative cases and deaths. Ironically, the high-income countries represent only 15 percent of the world population but constitute about 80 percent of the COVID-19 death toll [16]. Thus, JEE assessment criteria and the corresponding scoring system used before COVID-19 were proven inadequate during the COVID-19 pandemic.

The failure to effectively respond to the COVID-19 pandemic by many of the high-income countries that once scored high by JEE (i.e., countries with JEE >3 were predominantly middle-high to high income countries), raised questions about the perceived global preparedness capacities and the accuracy or precision of the above global measures. These anomalies prompted the IHR Review Committee under the auspice of the WHO Health Emergencies Programme to further review and revise the JEE tools through wider consultation with global experts, resulting in the Third Edition of the Joint External Evaluation Tool [10]. More recently, WHO has released a new monitoring and evaluation framework within the COVID-19 Strategic Preparedness and Response Plan 2022 (SPRP 2022), which outlines key strategic adjustments of the existing tools and approaches, review, and operational tracking of preparedness and response frameworks to ensure accountability and transparency [17].

Our study has identified several critical areas for WHO to consider for improvement in the future JEE country capacity assessment. Firstly, country capacity assessment is an immensely vast and highly sensitive judgmental exercise requiring an encyclopedic lens to examine the perspectives or the context of each of the capacity domains under assessment. The numeric scoring system for the indicator-based JEE assessment is easy to award, read, and take away as a position of capacity for a country. However, the numeric scoring should be pre-qualified based on a thorough qualitative contextual assessment of the conditions surrounding the technical areas. Thus, there is a need for an agreed mechanism of how these contextual assessment mechanisms for the evaluators to account for when scoring against an indicator so that any interjurisdictional comparisons of capacities would be sensible. Having emphasised a contextual analysis would allow the evaluators to award scores more judiciously, which in turn would enable countries to adopt a holistic response strategy to prepare for and combat future health emergencies. Moreover, this would minimize the risk of potentially inaccurately or imprecisely making an assessment of countries’ capacities, which might give them a false sense of complacency regarding their capacities to prepare for future health emergencies. To this end, the JEE tool should consider the inclusion of such contextual assessment conditions for all relevant indicators based on current best practice evidence.

Secondly, WHO’s critical role in leading the global architecture for health emergency preparedness, response, and resilience to fight against a new pandemic has been acknowledged [18]. Numerous programs and initiatives, both internal and external to WHO, have emerged locally and globally following the COVID-19 pandemic. It is important for WHO to coordinate those efforts and identify and minimize any gaps in the global health architecture. WHO may consider bringing all fragmented plans and actions at the global and country levels under a unified platform to ensure continuous monitoring of country capacities to take place in a participatory, outcome-focused, and coordinated manner.

Thirdly, many high-income countries that scored high in IHR technical capacity areas through the JEE before COVID-19 fared poorly during the initial stage of the pandemic. JEE tools are primarily focused on measurable capacities based on predefined criteria. While the JEE exercise may find a country scoring “developed” or “sustainable” capacity based on the JEE assessment criteria, the respective country may not be able to demonstrate the same level of capacities and achieve expected outcomes in a real-life emergency due to many factors including delayed response [19,20], investment priority in critical sectors [21], demographic factors such as relative younger or older populations [22] Nevertheless, JEE tools must be reliable and action-oriented. Stowell and Garfield recommended a more implementationfocused approach to the JEE measurement criteria, which we concur with [23]. Additionally, we recommend frequent countrylevel self-assessments, especially following every major health emergency. This should be complemented by more frequent independent expert reviews of the JEE approaches and tools for further improvement.

To our knowledge, this is the first study to have systematically examined the pre-COVID-19 country capacities as assessed by the WHO Joint External Evaluation results and compared their demonstrated capacities during COVID-19 by countries’ income status. There are, however, several limitations to this study. We used COVID-19 cumulative incidence and death rates (per 100,000 population) as of 8 June 2023, a time when many countries had already had a high uptake of the COVID-19 vaccine, resulting in lower cases and deaths compared to those countries that had lower vaccine uptake which may have confounded the relationship between higher JEE scores and COVID-19 cases and deaths. Secondly, the observed COVID-19 incidence and deaths across countries may not reflect their true rates due to varying detection rates, testing policies and reporting methods. Thirdly, population age structure may have confounded the relationship between the JEE scores and the COVID-19 outcomes. For example, many high-income countries with higher JEE scores have relatively larger older populations that experienced the worst COVID-19 outcome. Future evaluation studies addressing these gaps might contribute to further improvement of the JEE assessment system to be robust and reliable and the results more useful for the countries.

Conclusion

Despite the above limitations, the results of our study provide important implications for major changes to the country-capacity evaluation systems and processes for a more accurate reflection of countries’ strengths and gaps to assist them in undertaking appropriate and timely measures to better prepare for future health security events. Health security preparedness assessment tools such as the JEE need continuous review in light of the COVID-19 pandemic and, on an ongoing basis, following every major health security event. They need to be purpose-built and outcome-focused so that they can meaningfully assist countries in better preparing for future pandemics.