World TB Day 2017 (2): Something very serious doesn't add up in the numbers...

As a tiny stakeholder in the battle against tuberculosis we’ve been monitoring the pandemic through the lens of the annual WHO Global TB Reports year on year. We have to confess that we often found them difficult to penetrate beyond their executive summaries. This year, however, we’ve identified what we think are some major and very serious anomalies. We’ve written twice to Dr Katherine Floyd (the Report’s lead author) to ask if either she or her colleagues could clarify them for us without reply. Given the disappointing lack of response, we’re concluding that the WHO is reluctant either to explain or explore them so (in both exasperation and despair) we now publish the principle issues we raised below only because we think that they imply something that is very serious indeed.

The issues both revolve around the numbers as reported in the recent Report (a report which effectively wound up the era running up to the end of 2015). Whilst they may not immediately seem that important because they're historical, they are actually still of live concern because of how they inevitably impact on how these same numbers are going to be reported in the coming five, ten, fifteen and twenty years (which will, of course, be vital in terms of how we fare globally with this disease).

A summary of our observations

1. By the Report’s own definitions, the way that the 2015 Target of reducing the global death rate was reported (by comparing a 2015 estimated death rate to that of 1990) was not valid (to the extent that reasonable conclusions relating to the Targets that were set for 2015 could not drawn).

2. Notwithstanding this, in the latest Report two of the main graphs are incoherent with each other and data within them have been misrepresented. Furthermore the published non-HIV death rate of 15/100,000 is demonstrably wrong (it is actually 19/100,000) and this in turn means that the reduction in non-HIV death rate can be nowhere near 50% as is claimed.

3. There are no baselines yet set for 2015 and, given the ongoing improvement in surveillance, it looks like any new baselines will have to be subject to continuing retrospective re-adjustment throughout the post-2015 period. This hasn’t been done up to now in respect of the 2015 Targets (which is why they are invalid) but having to do so now is going to make things extremely confusing throughout the post-2015 period specifically in terms of how the data can be monitored by those who have most at stake in them.

4. There are also inevitable implications from this for both the World Health Assembly (WHA) and the UN, and also, of course, for the new Director General of the WHO when he/she is in post. We remain unconvinced that these major game-players are truly alert to these complexities as they need to be.

An explanation of these serious issues follows.

The Graphs

We took a quick overview of the summaries in this new 2016 Report, and particularly we took a look at the main graphs. Initially we noted that they largely followed the same vein as the years before and also accorded with the summary of the new Report which said that “the number of TB deaths and the TB incidence rate continue to fall globally”. This was further qualified by the Report’s statement that the “estimated trends in TB incidence and mortality remain similar to those published in previous years, with incidence falling by 2% per year over 2000–2015 and mortality falling by 3.3% per year over the same period.”

Of course, because it wasn’t concealed, we’d also spotted the fact that the reported estimated death toll had shot up this year by a third of a million and that the estimated number of new cases of disease had shot up by 800,000 as well. Clearly things weren’t as simple as they sounded and we realised though that these diverse statements deserved further examination.

In fact the Report appeared to be both explicit and coy at the same time on these extra numbers: “the [2016] report shows that the TB burden is actually higher than previously estimated, reflecting new surveillance and survey data from India” it explained. But in all respects the vitally important overall trends were still being reported to be in decline – we could even confirm this from the graph of trends of rates of disease 1990-2015, one of which showed that the non-HIV death rate had dropped by about 48% (as had been predicted in previous Reports), importantly adding up to the 2015 Target nearly being met. What’s more we noted that there was no visible up-kick in the graph reflecting the newly estimated 300,000 extra deaths – which we recognised meant that the line in the graph must have been generally shifted upwards in its entirety since the previous Report.

So far, so good. We also noted, however, that several respected commentators read the Report in a less positive way – specifically identifying that the number of people dying of TB was “growing”. But what, we wondered, about the Report’s executive summary that states so so explicitly that mortality was still falling during 2015 at 3.3% a year? So we took a look at the second main graph that focused on the one depicting mortality numbers (not this time the rates per 100,000 population). Again we could see no visible up-kick in the graph for the most recent year on record (for 2015): in other words, according to the Report both the death rate AND the numbers were still falling so the declining lines in both graphs must have been adjusted upwards to allow for this along with the extra numbers.

Of course we were also well aware of the usual small print at the bottom of this graphic that this “excludes TB deaths among HIV-positive people" (so the graph doesn’t in any case include every estimated TB mortality) but, notwithstanding this, it did seem reasonable to try to figure out exactly where those extra 300,000 deaths were in the graph. What follows is the result of our investigation.

Absolute numbers and the 1990 baseline

Before explaining further, we should add that we also noted something else: whilst the first graph goes back to 1990 (the baseline for the 2015 Targets) this second graph of absolute numbers only covers the period 2000-2015. As such we realised that the Report effectively leaves us in the dark about the actual numerical death toll that must have originally been estimated for 1990. What we COULD be sure about, however, (in spite of the downward trend in mortality rates in the first graph) was that annual numerical deaths had been rising from 1990 right through to around 2005 when (at least according to the most recent estimate in the graph above) they are now believed to have peaked when running at somewhere around 2.2 million a year.

We turned to the WHO Reports from 1997-2002 for further help (the ones which reported on data from the year 2000 and before) but unfortunately we couldn't find anything in them to enlighten us about that original 1990 death toll either. This was simply because the estimated annual death tolls from TB weren’t published in the annual Reports until 2005 because (we assume) of the paucity of available data. But at least in the following year’s Report (i.e. the one for 2006) we finally found the original baseline death rate for 1990 identified, which was 29/100,000.

The estimated death tolls for 2015 and what appears to be a serious mistake in the graphs

But we were still left wondering about HOW MANY people were estimated to have been dying from TB back in 1990. We could see that this wasn't identified because prior to 2005 they weren’t being estimated, but we realised that it would still be possible to figure this out by using the death rate that was pegged for 1990 that year (i.e. 29/100,000) and then computing it against the global population for the same year. So we ended up drawing up a rough table to help illustrate this for ourselves and (as can be seen below) we realised that this not only made for more confusion, it created some very significant disquiet about the current estimates and how they are being represented, not least because there now appeared to be a major error in the first graph above.

First we established that the global population in 1990 was estimated to be 5.28 billion which made the numerical non-HIV TB death toll 1.53 million at the quoted death rate of 29/100,000. Today the non-HIV TB numerical death toll is estimated to be 1.4 million, so the absolute number of non-HIV deaths can be deduced to have reduced by only 9% (1.53 million down to 1.4 million) between 1990 and today. But that was far from the end of it as we'll explain immediately.

This was because in the course of figuring this out something significant jumped out that hadn’t been visible at all, at least as far as we could see, within the 2016 Report itself.

The death rate for 2015 is reported/estimated to have been 15/100,000 (making for that supposed 48% reduction). On a whim we decided to compute this against the population estimate of 7.3 billion for 2015 in exactly the same way that we’d just done for 1990, and we found that this makes for a calculated death toll of 1.1 million. But the Report had categorically pegged the non-HIV death toll from TB at 1.4 million! We double checked, and realised that these two estimates, both drawn directly from the Report’s own data, are simply incoherent with each other. In fact they are out by 300,000 and we realised that we’d stumbled across those extra deaths, and that they weren’t actually being shown in one of the graphs.

We think that this comprises a major and misleading error..

Try as we could, we could see no other explanation of the discrepancy, but in trying to explain it to ourselves, we also realised that there were several other observations which can be reasonably made from our little table – all of which are, we think, are of sufficient public concern for them to need to be published.

1. The published reduction in non-HIV TB death rates (29/100,000 -> 15/100,000) may add up to a decrease of 48% but during this same period there appears to be a 12.5% increase in the OVERALL annual death toll (from 1.6 million [see note 3 below] to 1.8 million). We realise, however, that while this might at first appear to be gobbledegook, it should be borne in mind that there have been substantial population growths in many high burden countries which at least explains some of this (and of course also a lot of HIV/AIDS).

2. Then we figured out with some simple maths that a non-HIV death toll of 1.4 million in a population of 7.3 million actually makes for a death rate of 19/100,000, which is not really that close to 15/100,000 at all. So why, we now ask, was this re-estimation not shown in the graph that relates directly to one of the Targets set for 2015? According to our table this means that it looks like the death rate can only be claimed to have reduced by 38% and not by 48% (the latter percentage looking much closer to that 50% MDG Target that is still been reported in this latest Report to have been nearly achieved). At the very least these are significant anomalies that need further explanation.

3. We also know that 1990 saw the HIV pandemic still very much in its infancy. According to UNAIDS data there were estimated to be 300,000 HIV-associated deaths worldwide in that year. We quickly figured out that if 25% of them actually died of TB (which might be a reasonable percentage based on later data) then 75,000 deaths needed to be added to our 1990 non-HIV death toll of 1.53 million which we’d just calculated – which made 1.6 million as our own total ‘all TB’ death toll for 1990. Since the current death toll is pegged at 1.8 million, this in turn suggests that the TOTAL absolute number of annual TB deaths may have increased by 12.5% during the period 1990-2015 that was governed by the Millennium targets. This surely shows how poorly the HIV-TB syndemic was managed in the early years - but since the post-2015 targets for mortality now demand reductions of ABSOLUTE NUMBERS of all TB deaths it surely also tells us (if any of us were still under any illusion) how immensely challenging meeting these targets is going to be.

4. And once we'd included these HIV/TB deaths in the totals (i.e. 1.8 million deaths as per the most recent Report) the current overall TB death rate looks to be nearly 25/100,000 making for a rather paltry reduction of just 14% in the overall death rate for TB for the total period of 1990-2015. It occurs to us that this sort of insignificant reduction really deserved to be reported alongside that of non-HIV deaths because of the impact it must inevitably have on the post-2015 period and its targets.

5. We also noted (again using UNAIDS data) that there were 1.5 million HIV deaths in 2000. Using our previous estimated percentage of 25% of these actually dying from tuberculosis, this made for a possible 2.1 million total all-TB’ deaths during 2000 so this is what we added into the table above in order to complete it. It may not be accurate but it certainly didn’t make it look incoherent.

6. Finally we made a quick check of the estimated reductions in each of the ‘Big Three’ infectious killers in this period 2000-2015 (HIV, malaria and TB). The numbers of new HIV infections dropped by 32 percent, and the number of HIV deaths declined by 31 percent. Malaria infections dropped by 18 percent, and deaths by 48 percent. New cases of TB, meanwhile, appear to have only globally dropped by about 1 percent, and deaths — when not including the numbers of tuberculosis-related deaths among people with HIV — only fell by 22 percent, while ‘all-TB’ deaths only fell 14%. In other words TB has very much emerged as the biggest of the big three at the end of the 2015 era, and we feel that this surely should have been reported as it is a matter of such grave concern to us all.

Reductions between 2000 and 2015 Deaths New cases

Malaria 48% 18%

HIV 31% 32%

TB 14% 1%

If these rough calculations are remotely correct, it occurs to us that all of these findings shouldn’t just be in the public domain, they should also be particularly in both the hands and minds of those in the World Assembly and the UN (and also the new Director General) who all have the influence to make the very necessary changes to the battle against this pandemic in the next couple of years.

The difference between 2000-2015 time period and the one before it and the (in)validity of direct comparisons

We also noticed something else, however. We saw that this same baseline 1990 death-rate of 29/100,000 is also indicated in the current Report's graph (above) for 2015 – something which contrats with all of the subsequent retro-calculations that have been historically re-estimating the numbers.

Since this original 1990 death rate of 29/100,000 is exactly the same as the rate that appears in the current graph (the current one computed for 2015), and given that this rate has been in all the Reports since 2005, we realised that not everything can have been subjected to retro-calculation in these Reports as one might have expected. In fact the evidence suggested that the 1990 baseline (and quite possibly the entire data for the whole period 1990-1999) has actually NEVER been retro-calculated while almost everything since the year 2000 has been. We then homed in on a small-print disclaimer in the Report that (as we read it) implies that it should be assumed that no retro-calculation has ever been attempted or applied for any data prior to 2000:

“The updates can affect the entire time-series back to 2000. Therefore, estimates presented in this chapter for 2000−2014 supersede those of previous reports, and direct comparisons (e.g.between the 2014 estimates in this report and the 2014 estimates in the previous report) are not appropriate.”

This may all sound like a load of unimportant semantics to most readers of these reports but this disclaimer, while it may explain one anomaly, surely creates another which we believe is a very big deal indeed. This is because if “direct comparisons” aren’t appropriate between Reports (which is what the above disclaimer avers), then the same thing applies to any claims being made relating to the Goals to 2015 because they DIRECTLY compare current data (ones which have been adjusted based on better surveillance and now supersede all previous) with data from the 1990s which have never been retro-calculated nor superseded.

As such, in the Report’s own words (and by its own definitions) any comparisons made between data from 1990 and 2015 and any claims made about them cannot be deemed to be “appropriate”. In turn this suggests that all the reporting that’s been made in relation to meeting these targets has been questionable if not meaningless. This was a shocking conclusion for us to have drawn.

But there were now some other things that weren’t adding up as well. We still couldn't completely resolve, for instance, how (according to this Report) things seemed to be both deteriorating and improving at the same time.

More on the 1990 baseline

First of all, 1990 was both previous to and outside of the current time line series of 2015 back to 2000 (the time line which the Report states is subject to re-adjustments). As such it might seem to be reasonable to argue that it’s exempt from adjustment, but the truth is that the 2015 Targets (and thus the death rate for 1990) were actually set in 2005. As such it must have been set with the available data and intelligence from that year (i.e. from data available within the 2000-2015 period). This means (at least it does to us!) that despite the fact that 1990 is outside the 2000-2015 period, the estimated 1990 death rate is implicitly still a component part of the current time line series and so could (and should) have been re-adjusted and superseded in the same way as all the other post-2000 data have been.

The bottom line is that a set of goal posts at one end appear to have been moved regularly, while the one at the other hasn’t been moved at all.

Of course we also appreciate the enormous challenges that are faced by those compiling the Reports. We can’t help but conclude, however, that the disclaimer that states that each Report supersedes all previous ones has effectively made any assessment of any real progress in defeating this disease by the rest of the world difficult if not impossible. By not moving any goal posts from the period 1990-1999 it makes any assessment of the Millennium targets (including when done by the WHO itself in its Reports) a very tentative endeavour indeed - if not worthless. We can't help but wonder, moreover, whether anyone of those 192 signatories in the World Health Assembly was aware of this two years ago when they all signed up to the Post-2015 targets. In other words, do the play makers who’ve signed up to this deadly 'game' really have any ral idea about its true realities in terms of what does or doesn’t get adjusted as we go along?