1A/B Testing: Designs and Analysis
2Three Essential Components of Statistics
(Data Science):
Data+Computer+Analytics
1 Introduction 3
1 Introduction
1.1 What is A/B testing?
A/B test is the shorthand for a simple controlled experiment. As the
name implies, two versions (A and B) of a single variable are
compared, which are identical except for one variation that might
affect a user’s behavior. A/B tests are widely considered the simplest
form of controlled experiment. However, by adding more variants to
the test, this becomes more complex.
A/B testing is the process of comparing two variations of a page
element, usually by testing users’ response to variant A vs variant B,
and concluding which of the two variants is more effective.
1 Introduction 4
A/B tests are useful for understanding user engagement and
satisfaction of online features, such as a new feature or product. Large
social media sites like LinkedIn, Facebook, and Instagram use A/B
testing to make user experiences more successful and as a way to
streamline their services.
1 Introduction 5
Today, A/B tests are being used to run more complex experiments,
such as network effects when users are offline, how online services
affect user actions, and how users influence one another. Many jobs
use the data from A/B tests. This includes, data engineers, marketers,
designers, software engineers, and entrepreneurs. Many positions rely
on the data from A/B tests, as they allow companies to understand
growth, increase revenue, and optimize customer satisfaction.
1 Introduction 6
Version A might be the currently used version (control), while version
B is modified in some respect (treatment). For instance, on an
e-commerce website the purchase funnel is typically a good candidate
for A/B testing, as even marginal decreases in drop-off rates can
represent a significant gain in sales. Significant improvements can
sometimes be seen through testing elements like copy text, layouts,
images and colors, but not always. In these tests, users only see one of
two versions, as the goal is to discover which of the two versions is
preferable.
1 Introduction 7
Controlled experiments have a long and fascinating history. They are
sometimes called A/B tests, A/B/C tests (multiple variants), field
experiments, randomized controlled experiments, split tests, bucket
tests, and flights.
1 Introduction 8
1.2 Online experiments
Example 1. Online A/B testing. (Kohavi and Thomke, 2017,
Harvard Business Review) Microsoft, Amazon, Facebook and Google
conduct more than 10,000 online controlled experiments annually, with
many tests engaging millions of users.
Amazon’s experiment.
Treatment A: Credit card offers on front page.
Treatment B: Credit card offers on the shopping cart page.
This (change from A to B) boosted profits by tens of millions of US
Dollars annually.
1 Introduction 9
1.2.1 A/B Testing in eCommerce Industry
Through A/B testing, online stores can increase the average order
value, optimize their checkout funnel, reduce cart abandonment rate,
and so on. You may try testing: the way shipping cost is displayed and
where, if, and how free shipping feature is highlighted, text and color
tweaks on the payment page or checkout page, the visibility of reviews
or ratings, etc.
1 Introduction 10
In the eCommerce industry, Amazon is at the forefront in conversion
optimization partly due to the scale they operate at and partly due to
their immense dedication to providing the best customer experience.
Amongst the many revolutionary practices they brought to the
eCommerce industry, the most prolific one has been their ‘1-Click
Ordering’. Introduced in the late 1990s after much testing and
analysis, 1-Click Ordering lets users make purchases without having to
use the shopping cart at all. Once users enter their default billing card
details and shipping address, all they need to do is click on the button
and wait for the ordered products to get delivered. Users don’t have to
enter their billing and shipping details again while placing any orders.
With the 1-Click Ordering, it became impossible for users to ignore the
ease of purchase and go to another store. This change had such a
huge business impact that Amazon got it patented (now expired) in
1999. In fact, in 2000, even Apple bought a license for the same to be
used in their online store.
1 Introduction 11
People working to optimize Amazon’s website do not have sudden
‘Eureka’ moments for every change they make. It is through
continuous and structured A/B testing that Amazon is able to deliver
the kind of user experience that it does. Every change on the website
is first tested on their audience and then deployed. If you were to
notice Amazon’s purchase funnel, you would realize that even though
the funnel more or less replicates other websites’ purchase funnels,
each an every element in it is fully optimized, and matches the
audience’s expectations.
1 Introduction 12
Every page, starting from the homepage to the payment page, only
contains the essential details and leads to the exact next step required
to push the users further into the conversion funnel. Additionally, using
extensive user insights and website data, each step is simplified to their
maximum possible potential to match their users’ expectations.
1 Introduction 13
Take their omnipresent shopping cart, for example. There is a small
cart icon at the top right of Amazon’s homepage that stays visible no
matter which page of the website you are on.
1 Introduction 14
The icon is not just a shortcut to the cart or reminder for added
products. In its current version, it offers 5 options:
(i) Continue shopping (if there are no products added to the cart)
(ii) Learn about today’s deals (if there are no products added to the
cart)
(iii) Wish List (if there are no products added to the cart)
(iv) empty cart
(v) Proceed to checkout (when there are products in the cart). Sign in
to turn on 1-Click Checkout (when there are products in the cart).
1 Introduction 15
With one click on the tiny icon offering so many options, the user’s
cognitive load is reduced, and they have a great user experience. As
can be seen in the above screenshot, the same cart page also suggests
similar products so that customers can navigate back into the website
and continue shopping. All this is achieved with one weapon: A/B
Testing.
1 Introduction 16
1.2.2 A/B Testing in Travel Industry
Increase the number of successful bookings on your website or mobile
app, your revenue from ancillary purchases, and much more through
A/B testing. You may try testing your home page search modals,
search results page, ancillary product presentation, your checkout
progress bar, and so on.
1 Introduction 17
In the travel industry, Booking.com easily surpasses all other
eCommerce businesses when it comes to using A/B testing for their
optimization needs. They test like it’s nobody’s business. From the
day of its inception, Booking.com has treated A/B testing as the
treadmill that introduces a flywheel effect for revenue. The scale at
which Booking.com A/B tests is unmatched, especially when it comes
to testing their copy. While you are reading this, there are nearly 1000
A/B tests running on Booking.com’s website.
1 Introduction 18
Even though Booking.com has been A/B testing for more than a
decade now, they still think there is more that they can do to improve
user experience. And this is what makes Booking.com the ace in the
game. Since the company started, Booking.com incorporated A/B
testing into its everyday work process. They have increased their
testing velocity to its current rate by eliminating HiPPOs and giving
priority to data before anything else. And to increase the testing
velocity, even more, all of Booking.com’s employees were allowed to
run tests on ideas they thought could help grow the business.
1 Introduction 19
This example will demonstrate the lengths to which Booking.com can
go to optimize their users’ interaction with the website. Booking.com
decided to broaden its reach in 2017 by offering rental properties for
vacations alongside hotels. This led to Booking.com partnering with
Outbrain, a native advertising platform, to help grow their global
property owner registration.
1 Introduction 20
Within the first few days of the launch, the team at Booking.com
realized that even though a lot of property owners completed the first
sign-up step, they got stuck in the next steps. At this time, pages built
for the paid search of their native campaigns were used for the sign-up
process.
1 Introduction 21
Both the teams decided to work together and created three versions of
landing page copy for Booking.com. Additional details like social
proof, awards, and recognitions, user rewards, etc. were added to the
variations.
1 Introduction 22
The test ran for two weeks and produced a 25% uplift in owner
registration. The test results also showed a significant decrease in the
cost of each registration.
1 Introduction 23
1.2.3 A/B Testing in B2B/SaaS Industry
Generate high-quality leads for your sales team, increase the number of
free trial requests, attract your target buyers, and perform other such
actions by testing and polishing important elements of your demand
generation engine. To get to these goals, marketing teams put up the
most relevant content on their website, send out ads to prospect
buyers, conduct webinars, put up special sales, and much more. But
all their effort would go to waste if the landing page which clients are
directed to is not fully optimized to give the best user experience. The
aim of SaaS (Software as a service) A/B testing is to provide the best
user experience and to improve conversions. You can try testing your
lead form components, free trial sign-up flow, homepage messaging,
CTA text, social proof on the home page, and so on.
1 Introduction 24
POSist, a leading SaaS-based restaurant management platform with
more than 5,000 customers at over 100 locations across six countries,
wanted to increase their demo requests. Their website homepage and
Contact Us page are the most important pages in their funnel. The
team at POSist wanted to reduce drop-off on these pages. To achieve
this, the team created two variations of the homepage as well as two
variations of the Contact Us page to be tested. Let’s take a look at the
changes made to the homepage. This is what the control looked like:
1 Introduction 25
The team at POSist hypothesized that adding more relevant and
conversion-focused content to the website will improve user
experience, as well as generate higher conversions. So they created
two variations to be tested against the control.
Control was first tested against Variation 1, and the winner was
Variation 1. To further improve the page, variation one was then
tested against variation two, and the winner was variation 2. The new
variation increased page visits by about 5%.
1 Introduction 26
1.3 Clinical trials
Example 2. HIV transmission. Connor et al. (1994, The New
England Journal of Medicine) report a clinical trial to evaluate the
drug AZT in reducing the risk of maternal-infant HIV transmission.
50-50 randomization scheme is used:
AZT Group—239 pregnant women (20 HIV positive infants).
placebo group—238 pregnant women (60 HIV positive
infants).
1 Introduction 27
Given the seriousness of the outcome of this study, it is reasonable to
argue that 50-50 allocation was unethical. As accruing information
favoring (albeit, not conclusively) the AZT treatment became
available, allocation probabilities should have been shifted from
50-50 allocation proportional to weight of evidence for
AZT. Designs which attempt to do this are called Response-Adaptive
designs (Response-Adaptive Randomization).
1 Introduction 28
If the treatment assignments had been done with the DBCD (Hu
and Zhang, 2004, Annals of Statistics) with urn target:
AZT Group— 360 patients
placebo group—117 patients
then, only 60 (instead of 80) infants would be HIV positive.
1 Introduction 29
Example 3: Remdesivir-COVID-19 trial (China). Remdesivir
in adults with severe COVID-19 trial (Wang et al. 2020) is a
randomized, double-blind, placebo-controlled, multicentre trial that
aimed to compare Remvesivir with placebo. There were 236 patients
in the trial. There are about 20 baseline covariates for each patient,
including 10 continuous variables (e.g. age and White blood cell
count) and 10 discrete variables (e.g. gender and Hypertension). The
stratified (according to the level of respiratory support) permuted
block (30 patients per block) randomization procedure were
implemented. At the end of this trial, some important imbalances
existed at enrollment between the groups, including more patients with
hypertension, diabetes, or coronary artery disease in the Remdesivir
group than the placebo group.
1 Introduction 30
Example 4: Moderna COVID-19 vaccine trial (2020). The
trial began on July 27, 2020, and enrolled 30,420 adult volunteers at
clinical research sites across the United States. Volunteers were
randomly assigned 1:1 to receive either two 100 microgram (mcg)
doses of the investigational vaccine or two shots of saline placebo 28
days apart. The average age of volunteers is 51 years. Approximately
47% are female, 25% are 65 years or older and 17% are under the age
of 65 with medical conditions placing them at higher risk for severe
COVID-19. Approximately 79% of participants are white, 10% are
Black or African American, 5% are Asian, 0.8% are American Indian or
Alaska Native, 0.2% are Native Hawaiian or Other Pacific Islander, 2%
are multiracial, and 21% (of any race) are Hispanic or Latino.
1 Introduction 31
From the start of the trial through Nov. 25, 2020, investigators
recorded 196 cases of symptomatic COVID-19 occurring among
participants at least 14 days after they received their second shot. One
hundred and eighty-five cases (30 of which were classified as severe
COVID-19) occurred in the placebo group and 11 cases (0 of which
were classified as severe COVID-19) occurred in the group receiving
mRNA-1273. The incidence of symptomatic COVID-19 was 94.1%
lower in those participants who received mRNA-1273 as compared to
those receiving placebo.
1 Introduction 32
Investigators observed 236 cases of symptomatic COVID-19 among
participants at least 14 days after they received their first shot, with
225 cases in the placebo group and 11 cases in the group receiving
mRNA-1273. The vaccine efficacy was 95.2% for this secondary
analysis.
Long-term Treatment Effects?
1 Introduction 33
1.4 Economics and Social Science
Political A/B testing
A/B tests are used for more than corporations, but are also driving
political campaigns. In 2007, Barack Obama’s presidential campaign
used A/B testing as a way to garner online attraction and understand
what voters wanted to see from the presidential candidate. For
example, Obama’s team tested four distinct buttons on their website
that led users to sign up for newsletters. Additionally, the team used
six different accompanying images to draw in users. Through A/B
testing, staffers were able to determine how to effectively draw in
voters and garner additional interest.
1 Introduction 34
Example 5. The Project GATE (Growing America Through
Entrepreneurship), sponsored by the U.S. Department of Labor, was
designed to evaluate the impact of offering tuition-free
entrepreneurship training services (GATE services) on helping clients
create, sustain or expand their own business.
(https://www.doleta.gov/reports/projectgate/)
The cornerstone is complete randomization. Members of the
treatment group were offered GATE services; members of the control
group were not.
n = 4, 198 participants
p = 105 covariates
1 Introduction 35
1.5 Biological, psychological, and agricultural
research
Controlled experiments were mainly developed in these areas in
1900-1950.
1 Introduction 36
Road Map of this course:
(i) The history of experiment design;
(ii) A/B testing in medical studies;
(iii) Online controlled experiments (A/B testing).
2 The history of experiment design 37
2 The history of experiment design
2.1 Experiment design before Fisher
Statistical experiments, following Charles S. Peirce Main article:
Frequentist statistics See also: Randomization A theory of statistical
inference was developed by Charles S. Peirce in ”Illustrations of the
Logic of Science” (1877–1878) and ”A Theory of Probable Inference”
(1883), two publications that emphasized the importance of
randomization-based inference in statistics.
2 The history of experiment design 38
Randomized experiments: Charles S. Peirce randomly assigned
volunteers to a blinded, repeated-measures design to evaluate their
ability to discriminate weights. Peirce’s experiment inspired other
researchers in psychology and education, which developed a research
tradition of randomized experiments in laboratories and specialized
textbooks in the 1800s.
2 The history of experiment design 39
Optimal designs for regression models:
Charles S. Peirce also contributed the first English-language
publication on an optimal design for regression models in 1876. A
pioneering optimal design for polynomial regression was suggested by
Gergonne in 1815. In 1918, Kirstine Smith published optimal designs
for polynomials of degree six (and less).
2 The history of experiment design 40
2.2 Fisher’s principles
A methodology for designing experiments was proposed by Ronald
Fisher, in his innovative books: The Arrangement of Field Experiments
(1926) and The Design of Experiments (1935). Much of his pioneering
work dealt with agricultural applications of statistical methods. As a
mundane example, he described how to test the lady tasting tea
hypothesis, that a certain lady could distinguish by flavour alone
whether the milk or the tea was first placed in the cup. These
methods have been broadly adapted in biological, psychological, and
agricultural research.
2 The history of experiment design 41
2.2.1 Comparison
In some fields of study it is not possible to have independent
measurements to a traceable metrology standard. Comparisons
between treatments are much more valuable and are usually preferable,
and often compared against a scientific control or traditional
treatment that acts as baseline.
2 The history of experiment design 42
2.2.2 Randomization
Random assignment is the process of assigning individuals at random
to groups or to different groups in an experiment, so that each
individual of the population has the same chance of becoming a
participant in the study. The random assignment of individuals to
groups (or conditions within a group) distinguishes a rigorous, ”true”
experiment from an observational study or ”quasi-experiment”. There
is an extensive body of mathematical theory that explores the
consequences of making the allocation of units to treatments by means
of some random mechanism (such as tables of random numbers, or the
use of randomization devices such as playing cards or dice). Assigning
units to treatments at random tends to mitigate confounding, which
makes effects due to factors other than the treatment to appear to
result from the treatment.
2 The history of experiment design 43
The risks associated with random allocation (such as having a serious
imbalance in a key characteristic between a treatment group and a
control group) are calculable and hence can be managed down to an
acceptable level by using enough experimental units. However, if the
population is divided into several subpopulations that somehow differ,
and the research requires each subpopulation to be equal in size,
stratified sampling can be used. In that way, the units in each
subpopulation are randomized, but not the whole sample. The results
of an experiment can be generalized reliably from the experimental
units to a larger statistical population of units only if the experimental
units are a random sample from the larger population; the probable
error of such an extrapolation depends on the sample size, among
other things.
2 The history of experiment design 44
2.2.3 Statistical replication
Measurements are usually subject to variation and measurement
uncertainty; thus they are repeated and full experiments are replicated
to help identify the sources of variation, to better estimate the true
effects of treatments, to further strengthen the experiment’s reliability
and validity, and to add to the existing knowledge of the topic.
2 The history of experiment design 45
However, certain conditions must be met before the replication of the
experiment is commenced: the original research question has been
published in a peer-reviewed journal or widely cited, the researcher is
independent of the original experiment, the researcher must first try to
replicate the original findings using the original data, and the write-up
should state that the study conducted is a replication study that tried
to follow the original study as strictly as possible.
2 The history of experiment design 46
2.2.4 Blocking
Blocking is the non-random arrangement of experimental units into
groups (blocks) consisting of units that are similar to one another.
Blocking reduces known but irrelevant sources of variation between
units and thus allows greater precision in the estimation of the source
of variation under study.
2 The history of experiment design 47
2.2.5 Orthogonality
Orthogonality concerns the forms of comparison (contrasts) that can
be legitimately and efficiently carried out. Contrasts can be
represented by vectors and sets of orthogonal contrasts are
uncorrelated and independently distributed if the data are normal.
Because of this independence, each orthogonal treatment provides
different information to the others. If there are T treatments and T–1
orthogonal contrasts, all the information that can be captured from
the experiment is obtainable from the set of contrasts.
2 The history of experiment design 48
Example 2.1. Measurement Error: We would like to measure
the weight of a subject A by using a scale. We know that there is a
error of scale. Suppose that the error follows a normal distribution
with mean 0 and variance σ2. Mathematically, we may write:
w1 = A+ e1,
where wA is the true weight, YA is the observed weight and e1 is the
measurement error.
2 The history of experiment design 49
Figure 1: A scale to measure subject A
2 The history of experiment design 50
Now we would like to measure the weights of two subjects A and B by
using the same scale twice. What should we do?
2 The history of experiment design 51
Method 1:
w1 = A+ e1 and w2 = B + e2.
2 The history of experiment design 52
Figure 2: Subject B
2 The history of experiment design 53
Method 2:
w3 = A+B + e3 and w4 = A?B + e4.
2 The history of experiment design 54
Figure 3: A + B
2 The history of experiment design 55
Figure 4: A - B
2 The history of experiment design 56
The measurement errors:
Method 1:
Subject A: e1 ～ N(0, σ2).
Subject B: e2 ～ N(0, σ2).
Method 2:
Subject A: (e3 + e4)/2 ～ N(0, σ2/2).
Subject B: (e3 ? e4)/2 ～ N(0, σ2/2).
2 The history of experiment design 57
Use of factorial experiments instead of the one-factor-at-a-time
method. These are efficient at evaluating the effects and possible
interactions of several factors (independent variables). Analysis of
experiment design is built on the foundation of the analysis of
variance, a collection of models that partition the observed variance
into components, according to what factors the experiment must
estimate or test.
2 The history of experiment design 58
2.2.6 Avoiding false positives
False positive conclusions, often resulting from the pressure to publish
or the author’s own confirmation bias, are an inherent hazard in many
fields. A good way to prevent biases potentially leading to false
positives in the data collection phase is to use a double-blind design.
When a double-blind design is used, participants are randomly assigned
to experimental groups but the researcher is unaware of what
participants belong to which group. Therefore, the researcher can not
affect the participants’ response to the intervention.
2 The history of experiment design 59
Experimental designs with undisclosed degrees of freedom are a
problem. This can lead to conscious or unconscious ”p-hacking”:
trying multiple things until you get the desired result. It typically
involves the manipulation – perhaps unconsciously – of the process of
statistical analysis and the degrees of freedom until they return a
figure below the p?.05 level of statistical significance.
2 The history of experiment design 60
So the design of the experiment should include a clear statement
proposing the analyses to be undertaken. P-hacking can be prevented
by preregistering researches, in which researchers have to send their
data analysis plan to the journal they wish to publish their paper in
before they even start their data collection, so no data manipulation is
possible.
2 The history of experiment design 61
Another way to prevent this is taking the double-blind design to the
data-analysis phase, where the data are sent to a data-analyst
unrelated to the research who scrambles up the data so there is no
way to know which participants belong to before they are potentially
taken away as outliers.
2 The history of experiment design 62
2.2.7 Causal attributions
In the pure experimental design, the independent (predictor) variable is
manipulated by the researcher – that is – every participant of the
research is chosen randomly from the population, and each participant
chosen is assigned randomly to conditions of the independent variable.
Only when this is done is it possible to certify with high probability
that the reason for the differences in the outcome variables are caused
by the different conditions. Therefore, researchers should choose the
experimental design over other design types whenever possible.
2 The history of experiment design 63
However, the nature of the independent variable does not always allow
for manipulation. In those cases, researchers must be aware of not
certifying about causal attribution when their design doesn’t allow for
it. For example, in observational designs, participants are not assigned
randomly to conditions, and so if there are differences found in
outcome variables between conditions, it is likely that there is
something other than the differences between the conditions that
causes the differences in outcomes, that is – a third variable. The same
goes for studies with correlational design. (Ade′r Mellenbergh, 2008).
2 The history of experiment design 64
2.2.8 Statistical control
It is best that a process be in reasonable statistical control prior to
conducting designed experiments. When this is not possible, proper
blocking, replication, and randomization allow for the careful conduct
of designed experiments. To control for nuisance variables, researchers
institute control checks as additional measures. Investigators should
ensure that uncontrolled influences (e.g., source credibility perception)
do not skew the findings of the study. A manipulation check is one
example of a control check. Manipulation checks allow investigators to
isolate the chief variables to strengthen support that these variables
are operating as planned.
2 The history of experiment design 65
One of the most important requirements of experimental research
designs is the necessity of eliminating the effects of spurious,
intervening, and antecedent variables. In the most basic model, cause
(X) leads to effect (Y). But there could be a third variable (Z) that
influences (Y), and X might not be the true cause at all. Z is said to
be a spurious variable and must be controlled for. The same is true for
intervening variables (a variable in between the supposed cause (X)
and the effect (Y)), and anteceding variables (a variable prior to the
supposed cause (X) that is the true cause). When a third variable is
involved and has not been controlled for, the relation is said to be a
zero order relationship. In most practical applications of experimental
research designs there are several causes (X1, X2, X3). In most
designs, only one of these causes is manipulated at a time.
2 The history of experiment design 66
2.3 Experimental designs after Fisher
Some efficient designs for estimating several main effects were found
independently and in near succession by Raj Chandra Bose and K.
Kishen in 1940 at the Indian Statistical Institute, but remained little
known until the Plackett–Burman designs were published in
Biometrika in 1946. About the same time, C. R. Rao introduced the
concepts of orthogonal arrays as experimental designs. This concept
played a central role in the development of Taguchi methods by
Genichi Taguchi, which took place during his visit to Indian Statistical
Institute in early 1950s. His methods were successfully applied and
adopted by Japanese and Indian industries and subsequently were also
embraced by US industry albeit with some reservations.
2 The history of experiment design 67
In 1950, Gertrude Mary Cox and William Gemmell Cochran published
the book Experimental Designs, which became the major reference
work on the design of experiments for statisticians for years afterwards.
Developments of the theory of linear models have encompassed and
surpassed the cases that concerned early writers. Today, the theory
rests on advanced topics in linear algebra, algebra and combinatorics.
2 The history of experiment design 68
As with other branches of statistics, experimental design is pursued
using both frequentist and Bayesian approaches: In evaluating
statistical procedures like experimental designs, frequentist statistics
studies the sampling distribution while Bayesian statistics updates a
probability distribution on the parameter space.
2 The history of experiment design 69
Some important contributors to the field of experimental designs are
C. S. Peirce, R. A. Fisher, F. Yates, R. C. Bose, A. C. Atkinson, R. A.
Bailey, D. R. Cox, G. E. P. Box, W. G. Cochran, W. T. Federer, V. V.
Fedorov, A. S. Hedayat, J. Kiefer, O. Kempthorne, J. A. Nelder,
Andrej Pa′zman, Friedrich Pukelsheim, D. Raghavarao, C. R. Rao,
Shrikhande S. S., J. N. Srivastava, William J. Studden, G. Taguchi
and H. P. Wynn.
2 The history of experiment design 70
The textbooks of D. Montgomery, R. Myers, and G. Box/W.
Hunter/J.S. Hunter have reached generations of students and
practitioners.
Some discussion of experimental design in the context of system
identification (model building for static or dynamic models) is given
in[35] and [36].
2 The history of experiment design 71
2.4 Sequences of experiments
The use of a sequence of experiments, where the design of each may
depend on the results of previous experiments, including the possible
decision to stop experimenting, is within the scope of sequential
analysis, a field that was pioneered by Abraham Wald in the context of
sequential tests of statistical hypotheses. Herman Chernoff wrote an