A simple formula for enumerating comparisons in trials and network meta-analysis

The formula

The formula below solves this where n is the number of arms in a single study or network and N is the number of pairwise comparisons:

Where n > 0;

n is a natural number;

Then every intervention is compared to every other intervention except itself so: n*(n-1);

Because N is a bidirectional comparison (X vs. Y = Y vs. X) so: (n*(n-1))/2;

This is an established formula from combinatorics for calculating number of pairs for a number of items in a set.

The networks of 2 to 10 interventions will create networks in shapes of line, triangle, rectangle, pentagon, hexagon, heptagon, octagon, nonagon and decagon, respectively. A visual proof of a network of five interventions and (5*(5-1))/2=10 pairwise comparisons is presented in Figure 1.

Figure 1. Network of five interventions and (5*(5-1))/2=10 pairwise comparisons.

Adding any new intervention to the trial or network will create n-1 new pairwise comparisons. For example, where there are 6 arms in a trial—or 6 nodes in network meta-analysis—there will be (6*(6-1))/2=15 comparisons; adding a new intervention (6+1=7) will create 7-1=6 new pairwise direct comparisons in an individual trial and 6 direct or indirect comparisons in a network meta-analysis. Although this formula has been used for other purposes such as Metcalfe’s law in telecommunication, its use in the current context is novel.

Testing the formula: working back from existing network meta-analyses

We used the open data⁵ from our previously published network meta-analysis⁶ to re-create and enumerate the comparisons within the network. Using the direct comparisons reported in the trials within the network, we applied the formula and then compared the number of potential or expected comparisons (formula-derived) and the actual or observed number reported within the network analysis.

Number of direct and indirect comparisons

We built a small study-based register based—thus avoiding the pitfall of multiple counting—containing all 133 included studies in our previous network meta-analysis^6,7. These trials reported comparisons from 8 interventions. Using our formula, 8 interventions should create 28 unique comparisons: (8*(8-1))/2=28 (Figure 2).

Figure 2. All the possible unique bidirectional comparisons of 8 ADHD medications.

Only 16 out of 28 comparisons have been directly compared in trials (green lines).

Reported comparisons within the trials

We extracted the separate intervention arms from the open data to re-create the direct comparisons from within trials. The trials had either two or three arms so each study could create either two or three comparisons. As a result the 133 studies had 163 comparisons, the majority of which were duplicated. After removing these duplicates, this created 16 unique direct comparisons with between 1 and 47 studies per comparison for 8 interventions (Table 1). These 16 observed comparisons are 57% of the 28 expected by use of the formula above.

Direct comparisons eligible for network meta-analysis

Among five networks reported in the final paper, the number of comparisons in these five network meta-analyses, however, varies from 6 (for 3 networks) to 11 (for 1 network) and 13 (for 1 network) (Figure 3). As visualized in Figure 3, only 21.42% to 46.42% of comparisons were eligible for pairwise meta-analysis (Table 2).

Figure 3. Direct and indirect comparisons in the network meta-analysis of 8 interventions for primary outcome.

(Dark lines are eligible comparisons for pairwise meta-analysis, added dotted blue lines show indirect comparisons). This image has been modified from Cortese et al. 2018⁶ under Creative Commons Attribution License (CC BY).

Comparisons in network meta-analysis plots

From Figure 3 we can calculate that about 42% of comparisons expected through use of the formula have not been tested directly in trials. This is a direct evidence-gap. The number of missing comparisons varies between nine out of 15 in three networks with six interventions, 17 out of 28 in one network with eight interventions, and 15 out of 28 in another network with eight interventions (Figure 3). However, all 28 comparisons expected by use of the formula were utilized and reported within the network meta-analysis. It is possible that some of the comparisons predicted by the formula would have been deemed ineligible—either by adherence to a network review protocol or through post hoc exclusions—but this was not the case in this particular review (Figure 4). This diagram shows that only some of the comparisons from trials in study-based register could be included in pairwise meta-analysis. In addition, the number of comparisons in network meta-analysis (calculated by formula) is larger and inclusive of all the comparisons in the network of interventions and includes all the possible unique comparisons even if the comparisons are not in trials or in pairwise meta-analysis.

Figure 4. Venn diagram showing the coverage of comparisons by the network meta-analysis (from formula), and pairwise meta-analysis (from network plots), and trials (from study-based register).