Watch this Ask the Expert session to see examples of best practices for implementing bootstrap programs efficiently in SAS.
Watch the Webinar
You will learn:
- What the bootstrap method is and when it should be used.
- How to bootstrap a univariate statistic in SAS.
- Efficiency and details of the bootstrap in SAS.
- How to bootstrap regression estimates in SAS.
- SAS procedures that support built-in bootstrap methods.
The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.
Q&A
Does the bootstrap procedure yield useful joint distributions for two or more statistics if one is interested in correlations between the statistics?
Yes. In the regression example, you can think of those coefficient estimates as being two different statistics – a statistic for estimating the intercept parameter and for estimating the slope parameter. When you use PROC CORR on the bootstrap distribution, you can examine the correlation or the covariance of the joint distribution.
Is the confidence interval always estimated via quantiles in bootstrapping?
Yes. The percentile confidence interval (CI) uses the quantiles directly. If you use the bias corrected and adjusted CI, you are using a transformation of the quantiles. The transformation adjusts for bias and the skewness of the distribution.
Why should macro loops be avoided when bootstrapping?
It has to do with efficiency. When you use a macro loop, you are basically just repeating SAS code. You're calling some DATA step and procedures 5000 times. Using a BY-group analysis is a more efficient way to achieve the same result.
How many observations were in the original dataset?
There were 50 observations in the original data set.
How do you decide on what is the appropriate NumSample # for your dataset?
Basically, the confidence intervals that you get out of the bootstrap method are first order accurate. In many cases, if you want to double the precision of your bootstrap estimates, you need to quadruple the number of bootstrap samples. I suggest you run a bootstrap with 1000 samples, then repeat the bootstrap for 4000 bootstrap estimates, and look at the differences in the bootstrap estimates. The reason I used 5000 is because that tends to be fast and accurate. You could use 10,000 or 100,000 samples, but often you're just wasting computer time without a significant gain in accuracy.
Can the bootstrapping method be used to establish the normality of the underlying data, as is required for many statistical modelling procedures?
No, I wouldn't say that it can test for normality. However, if you run a parametric test that assumes normality and you run a bootstrap analysis and get a similar result, then that tells you the parametric method was a fine method. If the method has normality as its assumption, that could give some credence to the fact that the population is normally distributed. But I wouldn't use it as a test for normality.
Can you give some examples of how you use this in the real world?
Suppose I predict, based on some statistical model, that it's going to take 10 minutes to complete some tasks. My boss asks, what's the worst case and the best-case scenario? They want to know a confidence interval. Is the task likely to take 9 1/2 to 10 1/2 minutes? Or is it 8 to 12 minutes? You can use the bootstrap to come up with a confidence interval estimate, which is often more useful than a point estimate.
Is the bootstrap in Viya?
SAS Viya comes with Base SAS and SAS/STAT procedures. SAS/IML is also part of Viya. So, all the programs I showed today will run in Viya. Maybe you are asking whether there is a way to use parallel processing in Viya to perform bootstrapping faster? Yes, you can use the IML action to perform parallel bootstrap sampling and analysis.
Is there any difference in accuracy between IML and using various procedures? What is your preference?
No, there's no difference in accuracy. You can choose the tools (DATA step, procs, PROC IML,…) that you are comfortable with. For multivariate statistics, IML is a more natural language because it enables you to treat matrices and vectors as objects in a high-level language. But it's entirely up to you.
Are there any applications/scenarios that bootstrapping should be avoided?
Yes. Bootstrapping extreme percentiles such as 99th or the maximum is not recommended. You need to be careful estimating some quantiles, such as a median, especially for small data. In some cases, you can use a variation of the bootstrap called the smooth bootstrap.
What is a recommended minimum number of original observations to consider using bootstrap methods?
I've seen the bootstrap method used in as few as 12 observations. Parametric models might not work for small data. If you can't get more data, the bootstrap enables you to get some sort of estimates from the data you have. I feel more comfortable with the results if I have 30 observations rather than 10 or 15.
What is the difference between URS and BALBOOT in Bootstrapping using SURVEYSELECT?
URS is “unrestricted random sampling,” otherwise known as sampling with replacement. If you use URS to generate B samples, the i_th observation appears approximately B times in the union of the samples. The BALBOOT method performs balanced sampling. In a balanced set of samples, the i_th observation appears EXACTLY B times in the union of the samples. See “The balanced bootstrap in SAS.”
Can the result be reproduced (i.e., getting the same resamples) if I use the same seed number?
Yes. The same seed generates the same bootstrap samples. Therefore, the bootstrap estimates are the same.
Can you discuss more about the theory behind this method, e.g., assuming that the original sample is representative of the population distribution? If that were the case, wouldn't the point estimate be good enough?
The theory is that the empirical distribution (EDF) is the best estimate we have of the unknown population distribution. Therefore, we can model the sampling variability in a statistic by “simulating” random samples from the ECDF. These simulated samples are exactly the bootstrap samples. No matter how “representative” a sample is, a statistic has inherent variability due to sampling: take a different sample and the statistics change. The bootstrap enables you to estimate that variability.
If I want to bootstrap residual, which residual should I use from multiple mixed reg model with repeated observation in bootstrap?
This is complicated. I direct you to Chernick and LaBudde (2011), “An Introduction to Bootstrap Methods with Applications to R,” section 3.7.
Can you use PROC UNIVARIATE percentiles for each regression coefficient and see if any regression term changes sign from p2.5 and p97.5?
I think you are asking whether you can use PROC UNIVARIATE to obtain the 0.025th and 0.975th quantiles and thereby estimate the 95% confidence interval. Yes, that is correct. If 0 is in the interval, then you cannot reject the hypothesis that the regression coefficient is 0.
What is the difference between "A" bootstrap method and "THE" bootstrap method?
Every time you use a computer to perform a bootstrap analysis, you are performing “A” bootstrap because the result depends on the random samples that you generate. There is no such thing as “THE” bootstrap analysis.
Could you please share references for time-to-event bootstrapping?
The classic paper is Efron (1981) “Censored Data and the Bootstrap” at https://www.jstor.org/stable/2287832 It might be in some books from the 1990s.
What is your opinion on using data transformation for heavily skewed / not normally distributed data/very left-censored data? Bad idea? Other methods?
There is nothing wrong with it. Logarithmic transformation has a long of being used for this purpose. In general, you can choose from the Box-Cox family of transformations.
What about bootstrapping samples with design-based sampling weights?
I don’t know. I don’t see why it would be any different than the usual bootstrap.
How do I know when it's necessary to use the more complex methods like BCa?
Two ways: (1) Compare the original statistic and the bootstrap mean. If they are far apart, that is evidence of bias. (2) Graph the bootstrap distribution. If it is severely skewed, you need BCa.
Does the bootstrap method adjust the possible autocorrelations within the repeated samples?
Yes, provided you do it properly. See the block bootstrap.
What is the difference between the bootstrap method and jackknife method? Which one is more efficient?
The jackknife is a leave-one-out method. It is the precursor to the bootstrap. The jackknife is deterministic, whereas the bootstrap incorporates randomness. There are N jackknife samples of a sample of size N. They represent N out of the N^N possible bootstrap samples, so the bootstrap is more general.
How does accuracy improve with increasing the number of samples?
For many simple estimates, the accuracy is proportional to 1/sqrt(B), where B is the number of bootstrap samples.
Does bootstrapping work well for regression parameters when I have an extremely unbalanced binary variable in my model? Should I alter my sampling technique in this type of instance?
Depends on the underlying routine that is analyzing the data. Some methods, such as logistic regression, can handle unbalanced response categories provided there are at least a few observations in each category. In that case, you should be okay. You could run into problems if the rare response is so rare that some bootstrap samples contain only one of the response values.
Could precision estimates be applied to sporting game outcomes?
Yes.
If the bootstrap results do not contain 0, then do we say the means are different or does that mean to move to the non-parametric tests?
The phrase “contains 0” is usually used when you are examining a null hypothesis such as “is the parameter 0.” If the confidence interval for the parameter contains 0, then we cannot reject the null hypothesis. In the case of a two-sample t test, the hypothesis is that the difference between the means of two groups is 0. For that use case, if 0 is not in the CI, we would reject the null hypothesis and conclude that the groups have different means.
If you have a plethora of missing values and use mean imputation, is bootstrapping still viable?
Yes, but imputation methods usually have their own confidence interval estimates, which is like resampling, so you might not need bootstrap estimates.
Is the re-sampling technique somehow like bootstrap?
Yes, the bootstrap is one example of a resampling method. Another is permutation tests.
Do you have any references for reporting bootstrap statistics, such as in study literature? Would you typically report statistics without using bootstrap alongside this?
I don’t have any references, but it is common to report a statistic by reporting the point estimate followed by a confidence interval. I would write something like this: “The statistic is t=1.23 with a Wald confidence interval of (-0.1, 2.3). A bootstrap confidence interval is (-0.12, 1.98).”
What is your opinion on using bootstrap with clinical data?
I don’t think the source of the data makes any difference.
Recommended Resources
The essential guide to bootstrapping in SAS
Compute a bootstrap confidence interval in SAS
The BOOTSTRAP statement for t tests in SAS
Bootstrap regression estimates: Case resampling
Wicklin, Rick (2013), "Resampling and Bootstrap Methods," Chapter 15 in Simulating Data with SAS, SA...
Please see additional resources in the attached slide deck.
Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.
Click image to register for webinar