BootCompareCI

Description

BootCompareCI returns a bootstrap 95% confidence interval for a statistic comparing two independent samples. The function provides a non-parametric alternative to common statistical comparisons such as Welch’s two-sample t-test to compare means or an F test to compare variances. The user defines the nature of the comparison statistic that is used.

The function depends on BootCompare to return bootstrapped comparison statistics and applies either the percentile or BCa method to calculate the confidence interval.

Syntax

BootCompareCI(sample1, sample2, fn, [bca=FALSE], [B=2000], [bw=0]) where

• sample1 and sample2 are the samples the user is comparing. They can be univariate (row or column vectors) or multivariate (matrix, with rows as observations). If the inputs are multivariate they are resampled by rows.
• fn is a user-defined callback function that determines the nature of the comparison between samples. It has two input parameters and must return a single, scalar value: the comparison statistic that is being bootstrapped. For example, LAMBDA(x, y, VAR.S(x)/VAR.S(y)) will perform a (non-parametric) bootstrap version of an F-test.
• bca is a boolean that determines whether the BCa method will be used to calculate the confidence interval
• B is the number of bootstraps to perform. The default is 2000, though a higher value is recommended when returning a BCa confidence interval since that method may reach further into the tails if the sampling distribution is skewed.
• bw is the bandwidth of a smoothed bootstrap, which is applied to each sample before combining into one statistic. The default is bw=0, meaning no smoothing. The meaning of a nonzero value is the same as with the KDE functions like Density: a value of bw=1 applies the Silverman rule-of-thumb bandwidth to the data before sampling. This argument is ignored if the input data sample1 and sample2 are multivariate.

Like all the bootstrap functions, BootCompareCI is volatile since it depends on BootSample, which uses Excel’s RANDARRAY random number generator.

Example

A demonstration of the use and utility of bootstrapping for sample comparison is shown below. We are comparing the measurement errors of male and female university students for the analysis of sulfate by gravimetric analysis. The standard t-test and F-test are shown for the comparison of accuracy (no significant difference) and precision (significantly different). Both tests are parametric, making assumptions about the distribution of the comparison statistic.

In the case of the t-test we have the Central Limit Theorem that states the differences of means will tend towards normality at larger sample sizes; indeed the bootstrapped version of the t-test yields virtually the same 95% confidence interval and likewise does not label the difference in bias significant.

But the F-test makes a rather stricter assumption that the individual measurements are normally distributed, a fact that boxplots plainly contradict. The bootstrapped version of the F-test makes no such assumption but due to the outliers fails to detect a significant difference in precision.

However, bootstrapping can be used to compare more robust scale statistics; in the example below, the ratio of MAD values is used for the comparison and the confidence interval (barely) shows a statistically significant difference in values.

The Excel file is shown below. All BootCompareCI functions are volatile, and the computationally-intensive BCa method is used (three times) with $B = 5000$ so it may take a beat before the confidence intervals are calculated.

Download: BootCompareCI.xlsx

Source Code

Helpers used: BCaPercentiles, BootObs

The function has two dependencies: BootCompare, which returns the bootstrapped comparison statistics, and BCaPercentiles, which calculates the BCa confidence interval. See BootCompare for a listing and description of that function.

BootCompareCI = LAMBDA(
    sample1, sample2,     // two independent samples to compare
    fn,                   // lambda function returns a comparison stat, e.g. LAMBDA(x,y,MEDIAN(x)-MEDIAN(y))
    [bca],     // use the BCa method instead of the plain percentile method (default FALSE)
    [B],       // number of bootstrap sample comparisons (default 2000)
    [bw],      // smoothing bandwidth factor for the underlying BootCompare draws (default 0 = unsmoothed)
    LET(
        B, IF(ISOMITTED(B), 2000, B),
        bca, IF(ISOMITTED(bca), FALSE, bca),
        bw, IF(ISOMITTED(bw), 0, bw),
        stats, BootCompare(sample1, sample2, fn, B, bw),
        percentileResult, VSTACK(PERCENTILE.INC(stats, 0.025), PERCENTILE.INC(stats, 0.975)),
        bcaResult, IFERROR(
            LET(
                sampA, BootObs(sample1),    // same shape fn sees throughout BootCompare/BootSample
                sampB, BootObs(sample2),
                nA, ROWS(sampA),
                nB, ROWS(sampB),
                thetaHat, fn(sampA, sampB),   // statistic on the full original samples
                idxA, SEQUENCE(nA),
                idxB, SEQUENCE(nB),
                // pooled two-sample leave-one-out jackknife: drop one observation (one row,
                // for matrix input) from one sample at a time, holding the other sample
                // fixed, for every observation in both samples
                thetaJackA, MAP(idxA, LAMBDA(i, fn(FILTER(sampA, idxA <> i), sampB))),
                thetaJackB, MAP(idxB, LAMBDA(i, fn(sampA, FILTER(sampB, idxB <> i)))),
                thetaJack, VSTACK(thetaJackA, thetaJackB),
                BCaPercentiles(stats, thetaHat, thetaJack)
            ),
            "BCa error: bootstrap distribution or jackknife estimates too degenerate to compute a BCa interval (try a larger B)"
        ),
        IF(bca, bcaResult, percentileResult)
    )
);