Degenerate State

Causal Inference With Python Part 3 - Frontdoor Adjustment

2018-09-03T00:00:00+01:00

$$ \newcommand{P}[1]{\mathrm{P}\left( #1 \right)} \newcommand{Pc}[2]{\mathrm{P}\left( #1 \mid #2 \right)} \newcommand{In}[2]{ #1 \perp\!\!\!\perp #2} \newcommand{Cin}[3]{ #1 \perp\!\!\!\perp #2 \, | \, #3} \newcommand{do}[1]{\mathrm{do}\left( #1 \right)} \newcommand{E}[1]{\mathrm{E}\left[ #1 \right]} \newcommand{Ec}[2]{\mathrm{E}\left[ #1 \mid #2 \right]} $$

In a previous post I looked at how we can describe causal systems mathematically using the framework of causal graphical model. These models give us the ability to be precise about how we can phrase and answer causal questions of the form:

"What effect does changing $X$ have on $Y$?"

These causal graphical model show us exactly why causality is difficult: if there exist "backdoor paths" - or confounding variables, common causes for both $X$ and $Y$, then it is possible that any observed correlation between $X$ and $Y$ is due to these confounding paths, and not a direct causal relationship between $X$ and $Y$.

In situations where we observe the confounding variables in a causal graphical model we can overcome this limitation by "adjusting" for the backdoor path (sometimes called "covariate adjustment", "backdoor adjustment, "controlling for variables"). My first post contained a number of techniques for doing this.

This leaves the question: if we do not observe enough variables to adjust for a backdoor, can we still make causal inferences?

In some situations the answer to this question is yes. We will be examining one of these situations in this post. As in my previous post I will be using the causalgraphicalmodels python package for my examples.

Before we begin, we need to introduce some slightly new notation:

Unobserved Variables¶

When we describe our system as a causal graphical model, the effect of unmeasured we would like to capture are those that cause confounding relationships between variables because these change the conditional independence structure implied by the model. Unobserved variables which are parents of only one variable, or which act as mediators between variables can be absorbed into the non-parametric structure of the model without modification.

To represent confounding relationships between two variables, we add a dashed edge between, with double headed arrows. We can create such a structure using the following:

In [1]:

# first load some standard libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
%load_ext autoreload
%autoreload 2

In [2]:

from causalgraphicalmodels import CausalGraphicalModel

system_with_hidden_confounder = CausalGraphicalModel(
    nodes=["x", "y", "z"],
    edges=[("x", "z"), ("z", "y")],
    latent_edges=[("x", "y")]
)

system_with_hidden_confounder.draw()

Out[2]:

It's important to realise that this notation is only a convenience: we could simply add another variable as a parent to each pair of confounded variables, and reason about the resulting graph. The reason we include them as hidden variables is that it allows us to automatically reason about whether or not it is possible to infer the results of interventions.

Consider the graph above. The introduction of the hidden variable means that it is no longer possible to always use the backdoor adjustment to reason about about interventions - there is no valid backdoor adjustment for the graph above because the confounding variable is unobserved:

In [3]:

system_with_hidden_confounder.get_all_backdoor_adjustment_sets("x", "y")

Out[3]:

frozenset()

But all is not lost. The fact we observe variable $Z$ allows us to take another approach

Front Door Adjustment¶

Let's begin by writing our target:

$\Pc{Y}{\do{X}}$

We can expand this by including variable $Z$ (as in my previous post, I'm assuming all variables are discrete - if they are continuous, replace the sum with a integral)

$ = \sum_{Z}\Pc{Y}{\do{X}, Z}\Pc{Z}{\do{X}}$

Because there is only one directed path from $X$ to $Z$, their conditional distribution is the same as their interventional distribution and we can replace the $\do{\dots}$ in the second conditional with the observational value:

$ = \sum_{Z}\Pc{Y}{\do{X}, Z}\Pc{Z}{X}$

We can't make a similar adjustment to the first term, because there is a backdoor path between $X$ and $Y$. We can however replace the observed $Z$ with its intervential value $\do{Z}$:

$ = \sum_{Z}\Pc{Y}{\do{X}, \do{Z}}\Pc{Z}{X}$

Doing this, we can remove $\do{X}$ from the first conditional, because when we intervene on $Z$, we "screen off" the intervention on $X$:

$ = \sum_{Z}\Pc{Y}{\do{Z}}\Pc{Z}{X}$

It might not look like we've achieved much yet by replacing one intervention with another, but $\Pc{Y}{\do{Z}}$ is something we know how to manipulate: we can use backdoor adjustment. Conditioning again on $X$ blocks the back door between $Y$ and $Z$, allowing us to write:

$ = \sum_{Z}\Pc{Z}{X}\left(\sum_{X'}\Pc{Y}{Z, X'}\P{X'}\right)$

This is pretty amazing: we've managed to convert a causal statement into one about observations despite there being unmeasured confounders. There are still some strong assumptions here - we need to have observed mediation mechanism between $X$ and $Y$ that is somehow free from direct influence of the confounders, but it is not impossible to imagine situations where this might hold. I found it surprising that this kind of manipulation was possible.

We can verify that in the causal graphical model above, $Z$ is a valid variable for front door adjustment using the following code:

In [4]:

# {"z"} is a valid frontdoor adjustment set
system_with_hidden_confounder.is_valid_frontdoor_adjustment_set("x", "y", {"z"})

Out[4]:

True

In [5]:

# but the empty set is not
system_with_hidden_confounder.is_valid_frontdoor_adjustment_set("x", "y", {})

Out[5]:

False

And we can compute every valid frontdoor adjustment set using the following:

In [6]:

system_with_hidden_confounder.get_all_frontdoor_adjustment_sets("x", "y")

Out[6]:

frozenset({frozenset({'z'})})

The set of manipulations I carried out above to derived the frontdoor adjustment are not at all obvious. In deriving them I use a few imprecise arguments. The manipulation I carried out can be made precise, and are known as do-calculus - but that is a story for another blog post.

Estimating the ATE using the Front Door Adjustment¶

In the previous section we derived the relationship

$\Pc{Y}{\do{X}} = \sum_{Z}\Pc{Z}{X}\left(\sum_{X'}\Pc{Y}{Z, X'}\P{X'}\right)$

for valid frontdoor adjustment sets $Z$.

The remaining question is: given observational data, how do we calculate it?

To simplify things a bit, I am going to look at the simpler question:

$\Ec{Y}{\do{X}} = \sum_{Z}\Pc{Z}{X}\left(\sum_{X'}\Ec{Y}{Z, X'}\P{X'}\right)$

$ = \sum_{X', Y, Z} Y \, \Pc{Y}{Z, X'} \Pc{Z}{X} \P{X'}$

Which is about the expectation of $Y$, under intervention $X$.

We can reduce this to a single number, the average treatment effect (ATE):

$ATE = \Ec{Y}{\do{X=1}} - \Ec{Y}{\do{X=0}}$

The most direct way to estimate this is to estimate each separate conditional probability and combine the results.

Let's start by defining a StructuralCausalModel we can sample from:

In [7]:

from causalgraphicalmodels import StructuralCausalModel
from causalgraphicalmodels.csm import discrete_model, linear_model

fd0 = StructuralCausalModel({
    "u": lambda n_samples: np.random.binomial(n=1, p=0.5, size=n_samples),
    "x": discrete_model(["u"], {(0,): (0,1), (1,): (1,0)}),
    "z": discrete_model(["x"], {(0,): (0.95, 0.05), (1,): (0.05, 0.95)}),
    "y": discrete_model(["z", "u"], {
        (0, 0): (0.95, 0.05), 
        (0, 1): (0.95, 0.05),
        (1, 0): (0.50, 0.50), 
        (1, 1): (0.95, 0.05),
    }),
})

fd0.cgm.draw()

Out[7]:

This model relates four binary variables: $X$, $Z$, $Y$ and $U$. For the rest of this post I will assume that $U$ is unobserved. If it was we could adjust for it directly. Instead we have to use the frontdoor.

Before that, we should have some idea what the actual ATE is. The following functions allow us to empirically estimate the ATE from a StructuralCausalModel:

In [8]:

def estimate_conditional_expectation(df, x, y):
    """
    Estimate E[Y|X=1] - E[Y|X=0]
    
    from a dataframe `df` of samples.
    
    Arguments
    ---------
    df: pandas.DataFrame
    
    x: str
    
    y: str
    
    Returns
    -------
    estimates: tuple[float, float]
        estiamted difference and standard deviation
    """
    a = df.loc[lambda df: df[x] == 0]
    b = df.loc[lambda df: df[x] == 1]
    
    n_a = a.shape[0]
    n_b = b.shape[0]
    
    delta = b[y].mean() - a[y].mean()
    delta_std = np.sqrt(a[y].var() / n_a + b[y].var() / n_b)
    
    return (delta, delta_std)


def ab_test(scm, x, y, n=10000):
    """
    Simulate an A/B on StructuralCausalModel scm
    to estimate the quantity:
    
    E[Y|do(X) = 1] - E[Y|do(X) = 0]
    
    Arguments
    ---------
    scm: StructuralCausalmodel
    
    x: str
    
    y: str
    
    Returns
    -------
    estimate: tuple[estimates, estiamte_std]
    
    """
    scm_do = scm.do(x)
    
    n_a = int(n / 2)
    n_b = n - n_a
    set_variable = np.array([0]*n_a + [1]*n_b)
    
    samp = (
        scm_do
        .sample(
            set_values={x: set_variable}, 
            n_samples=n)
    )
    
    return estimate_conditional_expectation(samp, x, y)

In [9]:

print("Estimated ATE: {:.3f}".format(ab_test(fd0, "x", "y")[0]))

Estimated ATE: 0.208

We now need a function to make the same calculation using our frontdoor adjustment formula:

In [10]:

def estiamte_ate_frontdoor_direct(df, x, y, zs):
    """
    Estiamte the ATE of a system from a dataframe of samples `ds`
    using frontdoor adjustment directly on ML estimates of probability.
    
    E[Y|do(X) = x'] = \sum_{x,y,z} y P[y|x,z] P(z|x') P(X)
    
    Arguments
    ---------
    df: pandas.DataFrame
    x: str
    y: str
    zs: list[str]
    
    Returns 
    -------
    ATE: float
    """
    zs_unique = [tuple(a) for a in np.unique(df[zs].values, axis=0)]
    y_unique = np.unique(df[y].values, axis=0)    
    
    # P(X)
    p_x = {
        x_: np.mean(df[x] == x_)
        for x_ in (0,1)
    }
    
    # P(Z|X)
    p_z_x = {
        (x_, z_): np.mean(df
            .loc[lambda df_: df_[x] == x_]
            [zs]
            .values == z_)
        for x_ in (0,1)
        for z_ in zs_unique
    }
    
    # P(Y|X,Z)
    p_y_xz = {
        (x_, y_, z_): np.mean(df
             .loc[lambda df_: df_[x] == x_]
             .loc[lambda df_: (df_[zs].as_matrix() == z_).squeeze()]
             [y]
             .values == y_)
        for x_ in (0,1)
        for y_ in y_unique
        for z_ in zs_unique
    }
    
    
    # ATE
    E_y_do_x_0 = 0.0
    E_y_do_x_1 = 0.0
        
    for y_ in y_unique:
        for zs_ in  zs_unique:
            for x_ in [0,1]:
                E_y_do_x_0 += y_ * p_y_xz[(x_, y_, zs_)] * p_z_x[(0, zs_)] * p_x[x_]
                E_y_do_x_1 += y_ * p_y_xz[(x_, y_, zs_)] * p_z_x[(1, zs_)] * p_x[x_]
    
    return E_y_do_x_1 - E_y_do_x_0

In [11]:

ds = fd0.sample(10000)

estiamte_ate_frontdoor_direct(ds, "x", "y", ["z"])

Out[11]:

0.20228124029189454

It works!

But what happens when we have continuous values? Or sparse discrete values? In these cases directly estimating the conditional probabilities is impractical.

There are a number of approaches you can take - which ones you want to use will depend on exactly which assumptions you are will to make about you data. Let's start by looking at what we can do when assume the data is linear. In doing so we will get a feeling for what the front door adjustment is doing.

Linear Relationship¶

The linear approximation assumes that there are linear relationships between $X$, $Z$ and $Y$, or that:

$Z = a_{z} + b_{z, x} X + \epsilon_{z}$

$Y = a_{y} + b_{y, z} Z + b_{y, u} U + \epsilon_{y}$

If these are the true data generating equations then the causal effect of changing $X$ on $Y$ is given by the product of coefficients $ b_{z, x} b_{y, z}$. Simple regression doesn't allow us to estimate this value because the unmeasured variable $U$ is a parent of both $X$ and $Y$ - and if $X$ and $U$ end up collinear, our estimate of $b_{y, z}$ cannot be trusted.

One way to side-step this issue is to perform a double regression:

First we regress $Z$ on $X$. This not only gives us the coefficient $b_{z, x}$, but the way to estimate $\Ec{Z}{X}$. If this estimate of $\Ec{Z}{X}$ is accurate, we can use it to find the values of the noise in the first equation. Specifically: $ Z - \Ec{Z}{X} = \epsilon_{z}$
This noise is only introduced at the node for $Z$, which means that by assumption it is independent of the confounding variable $U$. Regressing $\epsilon_{z}$ on $Y$ then gives us an estimate of the coefficient $b_{y, z}$ where we don't have to worry about the collinearity problem.

Together this procedure gives us the following estimate of

In [18]:

import statsmodels.api as sm

def estimate_ate_frontdoor_linear(df, x, y, z):
    """
    Estiamte the ATE of a system from a dataframe of samples `df`
    using frontdoor adjustment and assuming linear models.
    
    Arguments
    ---------
    df: pandas.DataFrame
    x: str
    y: str
    z: str
    
    Returns 
    -------
    ATE: float
    """
    x = df[x].values
    y = df[y].values
    z = df[z].values
    
    z_x_model = sm.OLS(z, sm.add_constant(x)).fit()
    
    z_bar = z_x_model.predict(sm.add_constant(x))
    z_prime = z - z_bar

    y_z_model = sm.OLS(y, sm.add_constant(z_prime)).fit()
    
    return y_z_model.params[1] * z_x_model.params[1]

Let's see how it does:

In [13]:

# first define a linear model
fd1 = StructuralCausalModel({
    "u": lambda n_samples: np.random.binomial(n=1, p=0.5, size=n_samples),
    "x": discrete_model(["u"], {(0,): (0,1), (1,): (1,0)}),
    "z": linear_model(["x"], [8], -3),
    "y": linear_model(["z", "u"], [1, 8], -2),
})

# sample from the model
n = int(10e4)
ds = fd1.sample(n)

# evaluate the model
print("Different estimates of the ATE for a linear system")
print("--------------------------------------------------")
print("Naive estimator: {:.3f}".format(estimate_conditional_expectation(ds, "x", "y")[0]))
print("AB test: {:.3f}".format(ab_test(fd1, "x", "y", n=n)[0]))
print("Linear estimator: {:.3f}".format(estimate_ate_frontdoor_linear(ds, "x", "y", "z")))

Different estimates of the ATE for a linear system
--------------------------------------------------
Naive estimator: 0.004
AB test: 8.033
Linear estimator: 8.012

It looks reasonably good.

It is worth reflecting on what this process is doing: we are using the randomness introduced by the intermediate variable $Z$ to try and disentangle the confounded relationship between $X$ and $Y$. In some ways this is exactly what the more general frontdoor adjustment is doing: we combine our knowledge of the effect of an intervention at $X$ on $Z$ with the effect of an intervention on $Z$ using $X$ to remove it's bias.

Beyond linear models¶

Linear models are powerful when the data is approximately linear, but often this is too restrictive an assumption. We may also want to introduce categoric variables which may not fit into a linear model framework.

There are a number of approaches we can take. The one I am going to look at here is the semi-parametric estimator inspired by this paper. It requires us to make some assumptions about how we can model parts of our data, and leaves others to be estimated non-parametrically from the data. It is not only estimator that exists.

The estimator is:

$\Ec{Y}{\do{X=x}} = \frac{1}{N}\sum_{i} \frac{g(z_{i}) I(x_{i} = x)}{\hat{p}(x_{i})}$

Where

$I(x_{i} = x)$ is the indicator function: one when $x_{i} = x$ and zero otherwise
$\hat{p}(x_{i})$ is an estimate of the propensity of $x_{i}$ being observed
$g(z_{i})$ is an estimate of $\Ec{\Ec{y}{x,z}}{z}$
the sum runs over all the samples

To understand how this estimator works, let's start expanding the probability distribution of our samples using Bayes theorem to get:

$\P{x,y,z} = \Pc{y}{x,z}\Pc{z}{x}\P{x}$

Averaging our estimator over our samples gives us an estimate of

$\E{\frac{g(z_{i}) I(x_{i} = x')}{\hat{p}(x_{i})}} = \sum_{x,y,z} \Pc{y}{x,z}\Pc{z}{x}\P{x} \frac{g(z) I(x = x')}{\hat{p}(x)}$

If we assume that our model $\hat{p}(x_{i})$ is accurate we can simplify this to

$\sum_{x,y,z} \Pc{y}{x,z}\Pc{z}{x} g(z) I(x = x')$

The summing over $x$ and keeping only the non-zero values we get

$\sum_{z} \Pc{z}{x'} g(z)$

And replacing $g$ with the thing it is trying to estimate, we get

$\sum_{x,y,z} y \Pc{y}{x,z}\P{x}\Pc{z}{x'}$

Which, if you compare it with the definition of the frontdoor adjustment, is exactly the quantity we were trying to estimate.

It is important to note that in demonstrating that this estimator matches the frontdoor adjustment we have ignored a number of import questions around asymptotic efficiency of the estimator. This is important and will affect the quality of your results, but they are also a question for another blog post.

We have also not discussed how to calculate $g(z)$ or $\hat{p}(x)$ - this is a much easier problem to address. For binary $x$, $\hat{p}(x)$ can be estimated from the sample averages and $g(z)$ can be calculated using you favorite machine learning technique to estimate a conditional mean (although we must be careful to use stacking when computing the estimate to avoid training on the same data we are predicting on).

Some example code is below

In [14]:

from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.model_selection import KFold

def stack_predict_and_sum(x, z, y, model, n_splits=3):
    """
    Estimate E[ E[Y|X,Z] |Z]
    
    Assumes that X is a binary value, and uses `model` to
    predict the conditional mean. To prevent overfitting on 
    the data itself, we use stacking to predict on data that 
    wasn't used for fitting.
    
    Arguments
    ---------
    x: numpy.array [Nx1] binary array
    y: numpy.array [Nx1] target array
    z: numpy.array [NxM] feature array
    model: sklearn.estimator
    
    Returns 
    -------
    numpy.array [Nx1]
    """
    kf = KFold(n_splits=n_splits, shuffle=True)
    output = np.zeros_like(y, dtype=np.float)
    p_x = np.mean(x)
    
    for train_index, test_index in kf.split(x):
        x_train = x[train_index]
        x_test  = x[test_index]
        
        z_train = z[train_index]
        z_test  = z[test_index]
        
        y_train = y[train_index]
        y_test  = y[test_index]      
        
        # train model on train set
        X = np.concatenate([x_train, z_train], axis=1)
        model.fit(X, y_train.squeeze())
        
        # predict model on z from test set and x=0, x=1
        X_pred_x_0 = np.concatenate([np.zeros_like(x_test), z_test], axis=1)
        X_pred_x_1 = np.concatenate([np.ones_like(x_test), z_test], axis=1)
        
        y_pred_x_0 = model.predict(X_pred_x_0).reshape(-1,1)
        y_pred_x_1 = model.predict(X_pred_x_1).reshape(-1,1)
        
        output[test_index] = (p_x * y_pred_x_1 + (1-p_x)*y_pred_x_0)
       
    return output


def estimate_intervention_frontdoor_nonlinear(ds, x, y, z, model_y_zx):
    """
    Estiamte the ATE of a system from a dataframe of samples `df`
    using a semi-parametric frontdoor adjustment with a arbritary 
    model of the conditional expectation. 
    
    Stacking is used to prevent overfitting on the data.
    
    Arguments
    ---------
    df: pandas.DataFrame
    x: str
    y: str
    z: List[str]
    model_y_zx: sklearn.estimator 
    
    Returns 
    -------
    ATE: float
    """
    x = ds[[x]].values
    y = ds[[y]].values
    z = ds[z].values

    mask_x0 = ds.x.values == 0

    # P(x)
    p_x_0 = np.mean(mask_x0)
    p_x_1 = 1 - p_x_0
    p_x = np.where(mask_x0, p_x_0, p_x_1)

    # E[ E[Y|X,Z] |Z]
    y_z_mean = stack_predict_and_sum(x=x, y=y, z=z, model=model_y_zx).squeeze()

    # sum over samples and estimate:
    E_y_do_x0 = np.mean((y_z_mean * mask_x0) / (1 - p_x))
    E_y_do_x1 = np.mean((y_z_mean * (~mask_x0)) / (p_x))
    
    return E_y_do_x1 - E_y_do_x0

Let's see how it performs.

To evaluate the estimator we are going to use the following non-linear frontdoor-like system.

In [15]:

# Define the model
fd2 = StructuralCausalModel({
    "u": lambda n_samples: np.random.binomial(n=1, p=0.5, size=n_samples),
    "x": discrete_model(["u"], {(0,): (1.00,0.00), (1,): (0.00, 1.00)}),
    "z": lambda x, n_samples: np.where(
        x == 0,
        np.random.triangular(0,0,1, size=n_samples),
        np.random.triangular(0,1,1, size=n_samples),
    ),
    "y": lambda z, u, n_samples: np.where(
        u == 0, 
        1-(4*(z - 0.5)**2),
        (4*(z - 0.5)**2)
    ),
})

# sample from the model
n = int(10e3)
ds = fd2.sample(n)

# evaluate the model
print("Different estimates of the ATE")
print("------------------------------")
print("Naive estimator: {:.3f}".format(estimate_conditional_expectation(ds, "x", "y")[0]))
print("AB test: {:.3f}".format(ab_test(fd2, "x", "y", n=n)[0]))
print("Linear test: {:.3f}".format(estimate_ate_frontdoor_linear(ds, "x", "y", "z")))
print("Non-linear: {:.3f}".format(
    estimate_intervention_frontdoor_nonlinear(
        ds, "x", "y", ["z"], model_y_zx=RandomForestRegressor())))

Different estimates of the ATE
------------------------------
Naive estimator: -0.338
AB test: -0.001
Linear test: 0.272
Non-linear: -0.012

The actual ATE is zero, which our estimate predicts correctly, but the linear ATE model predict incorrectly.

This looks good - but it important to realise that there are limitations. Let's go back to the second StructuralCausalModel we created: the linear system and see how our estimator performs:

In [16]:

# sample from the model
n = int(10e3)
ds = fd1.sample(n)

print("Different estimates of the ATE for a linear system")
print("--------------------------------------------------")
print("Naive estimator: {:.3f}".format(estimate_conditional_expectation(ds, "x", "y")[0]))
print("AB test: {:.3f}".format(ab_test(fd1, "x", "y", n=n)[0]))
print("Linear estimator: {:.3f}".format(estimate_ate_frontdoor_linear(ds, "x", "y", "z")))
print("Non-linear estimator: {:.3f}".format(
    estimate_intervention_frontdoor_nonlinear(
        ds, "x", "y", ["z"], model_y_zx=RandomForestRegressor())))

Different estimates of the ATE for a linear system
--------------------------------------------------
Naive estimator: 0.050
AB test: 8.092
Linear estimator: 8.054
Non-linear estimator: 1.424

It doesn't do well. The reason for this is that we have violated one of the assumptions of the model we're using: good overlap of the covariates. We are using a random forest model to estimate the conditional expectation. This kind of model will do well when we have samples that have good overlap, but fail badly if they attempt to extrapolate beyond that range.

Conversely, if a system is actually linear, a well fitted linear model will be able to extrapolate well beyond the range it was trained on. This is why the results of the linear ATE are so good in this case.

If we plot the value of our $Z$ variable for the linear model, the lack of overlap becomes clear:

In [17]:

z1 = ds.loc[lambda df: df.x == 1].z
z0 = ds.loc[lambda df: df.x == 0].z

sns.distplot(z1, label="x=1")
sns.distplot(z0, label="x=0")

plt.legend();

It is important to realise that this trade-off between ability-to-extrapolate and quality-of-the-model (note: I am not using these terms in a precise way) is nothing specific to front door adjustment: we saw exactly the same thing in my first post which focused on techniques that applied backdoor adjustment. This is not even specific to causal inference - any model building approach will require some trade-off at this level.

The main difference between using models to make causal inferences and using them for predictions is that if our goal is just predictions we can use cross-validation to assess the accuracy of our model. When doing causal inference we usually have to rely assumptions about the system - and any conclusions we draw from our models will only be as good as the assumptions we put in.

Conclusion¶

In this post I've covered how we can use the front door adjustment approach to estimate the average treatment effect even in situations where we do not have access to the confounding variables of a system. While this is powerful, it does require other assumptions about our ability to observe mediating variables.

At a more general level, I have hoped to demonstrate that there is more to causal inference then just the back door adjustment, and that graphical models are an excellent way to express our knowledge of a system. It should also raise the question: are there more general causal inference approaches than just frontdoor and backdoor adjustment?

In turns out there are. The system of rule for deriving such expressions is known as do-calculus and I plan to write my next post on this topic.

A separate but more practical question is once we have these expressions about causal interventions, how do we actually estimate there values? This is a fascinating topic in itself, and one I hope to tackle in another post.

Code¶

You can find the notebook for this post on github here.

Causal Inference With Python Part 2 - Causal Graphical Models

2018-07-10T00:00:00+01:00

In a previous blog post I discussed how we can use the idea of potential outcomes to make causal inferences from observational data. Under the Potential Outcomes framework we treat the counterfactual outcome as if it were missing data and attempt to estimate these missing values from the observed data. To do this we needed to make strong assumptions about the data generating process, specifically "Strong Ignorability"

$\Cin{Y_{i}}{X}{Z}$

Where $Y_{i}$ are the potential outcomes we are trying to estimate, $X$ is the intervention we are trying to measure and $Z$ are a set of covariates which allow us to "correct" our estimate. This statement should be read: "$Y_{i}$ is conditional independent of $X$ given $Z$".

This is a strong statement about the process which generated our data. In order to understand where strong ignorability hold, we need to make some assumptions about the structure of the data generating process itself. The language we will be using to express this structure is that of Causal Graphical Models. In this post I will try to give an light overview of causal graphical model using a python package of the same name.

Compared to my previous post, this post will be less about techniques to make causal inferences and more on gaining intuition about how we can describe data generating structure and what statements we can make once we have such a description. I am also not going to be playing fast and loose with some of the maths

What is Structure?¶

One way we can try and model the world is through the idea of Structural Causal Models, or Structural Equation Models: that we can model the relationships between different variables is described by functions.

For example, imagine a system of three variables, $x_{1}, x_{2}, x_{3}$. We could imagine they are related in the following way:

$x_{1} \sim \hbox{Bernoulli}(0.3)$

$x_{2} \sim \hbox{Normal}(x_{1}, 0.1)$

$x_{3} = x_{3}^{2}$

$X_{1}$ and $X_{2}$ are samples from random variables, and $X_{3}$ is a deterministic function of $X_{2}$.

It is easy to simulate this system in python:

In [1]:

# first, import some standard libraries

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

In [2]:

def f1(): 
    return np.random.binomial(n=1,p=0.3)

def f2(x1):
    return np.random.normal(loc=x1, scale=0.1)
    
def f3(x2):
    return x2 ** 2

x1 = f1()
x2 = f2(x1)
x3 = f3(x2)

print("x1 = {}, x2 = {:.2f}, x3 = {:.2f}".format(x1, x2, x3))

x1 = 0, x2 = -0.04, x3 = 0.00

Structure is the description of which variables are functions of which other variables. In the above example, x1 can only influence x3 though x2, no matter what the actual functions are.

This way of modeling structure is appealing because it has a natural way to describe interventions: we reach into the system and replace the value of one variable with one we choose. If we replace x2 with some value x2_intervention the system would now be updated according to

x1 = f1()
x2 = x2_intervention
x3 = f3(x2)

This has the very measurable consequence that if we intervene on x2, x3 will be independent of x1. This wouldn't be the case if the structure had looked something like

x1 = f1()
x2 = f2(x1)
x3 = f3(x2, x1)

Here there is no ambiguity about what intervention means, and what the structure of the system is. It might seem too far fetched to assume that all relationships been things we measure can be described this way, but ultimately the justification is the universe is governed by the laws of physics. These functions might be very complicated, stochastic, or unknown, but they exist.

The question then becomes: what can we say about interventions when we don't know the true functional relationships, only the structure.

Note: This isn't entirely true. Quantum mechanics has some interesting things to say about causality, and if I get the chance, I'll try and write about them. For most causal inference tasks, we can safely ignore quantum effects.

Causal Graphical Models¶

Let us begin with a classical example of a causal system: the sprinker. It is a system of five variable which indicate the conditions on a certain day:

$season$: indicates which season it is
$rain$: indicates whether it is raining
$sprinkler$: indicates whether our sprinkler is on
$wet$: indicates whether the group is wet
$slippery$: indicates whether the ground is slippery

We know that when it rains, the ground will become wet, however the making the ground wet doesn't cause it to rain. This is exactly the kind of direct relationship that could be described by a function. In the absence of this actual function, we are left with a set of variables and directed relationships between then. A natural way to represent this structure is a directed graph, specifically a Directed Acyclic Graph. We require the graph to be acyclic to prevent "causal loops".

We can create a causal graphical model of this system by specifying the nodes and edges of this graph:

In [3]:

from causalgraphicalmodels import CausalGraphicalModel

sprinkler = CausalGraphicalModel(
    nodes=["season", "rain", "sprinkler", "wet", "slippery"],
    edges=[
        ("season", "rain"), 
        ("season", "sprinkler"), 
        ("rain", "wet"),
        ("sprinkler", "wet"), 
        ("wet", "slippery")
    ]
)

# draw return a graphviz `dot` object, which jupyter can render
sprinkler.draw()

Out[3]:

This is a Probabilistic Graphical Model description of the system (for more information, I recommend this excellent Coursera Course on probabilistic graphical models): a non-parametric model of the structure which generates data.

Describing a system in the way implies that the joint probability distribution over all variables can be factored in the following way:

$\P{\mathbf{X}} = \prod_{i}\Pc{X_{i}}{\hbox{PA}(X_{i})}$

Where $\hbox{PA}(X_{i})$ is the set of parents of the variable $X_{i}$, with respect to the graph.

We can get the join probability distribution implied by our causal graphical model using

In [4]:

print(sprinkler.get_distribution())

P(season)P(sprinkler|season)P(rain|season)P(wet|rain,sprinkler)P(slippery|wet)

This factorization of the joint probability distribution in implies certain conditional independence relationships between variables. For example, if we know whether or not the ground is wet, then whether or not it is slippery is independent of the season. In the language of probabilistic graphical models, two variables are conditionally independent given other variables if they are d-separated.

I'm not going to go into a full proof of d-separation, but to get some intuition about how it is calculated, consider the skeleton of our DAG (the graph with the same nodes and edges, but no notion of "direction"). Two variables can only be related if there are paths between them, so we can limit our attention to the paths between variables. If there is only a single edge between the variables, they cannot be conditionally independent.

For paths of three nodes, there are three possible situations, a fork, a chain and a collider, shown below:

In [5]:

from causalgraphicalmodels.examples import fork, chain, collider

print("Implied conditional Independence Relationship: ", 
      fork.get_all_independence_relationships())
fork.draw()

Implied conditional Independence Relationship:  [('x1', 'x3', {'x2'})]

Out[5]:

In [6]:

print("Implied conditional Independence Relationship: ", 
      chain.get_all_independence_relationships())
chain.draw()

Implied conditional Independence Relationship:  [('x1', 'x3', {'x2'})]

Out[6]:

In the fork and the chain imply the same independence relationships: $X_{1}$ and $X_{3}$ are not independent, unless we condition on $X_{2}$ when they become conditionally independent. (Although I should note they imply very different causal structures: In a chain $X_{1}$ has causal influence on $X_{3}$, but in a fork there is no causal influence).

In [7]:

print("Implied conditional Independence Relationship: ", 
      collider.get_all_independence_relationships())
collider.draw()

Implied conditional Independence Relationship:  [('x1', 'x3', set())]

Out[7]:

For the collider, $x_{1}$ and $x_{3}$ are independent, unless $x_{2}$ or any of it's descendants are in the group we condition on. This is sometimes called Berkson's Paradox.

For paths longer then length 3, it turns out we can use the previous results to decide if two nodes are d-separated by examining each three structure along the paths: a path is d-separated if all sets of consecutive 3-nodes are d-separated.

Consider the following path between $X_{1}$ and $X_{5}$:

In [8]:

path = CausalGraphicalModel(
    nodes = ["x1", "x2", "x3", "x4", "x5"],
    edges = [("x1", "x2"), ("x3", "x2"), ("x4", "x3"), ("x4", "x5")]
)

path.draw()

Out[8]:

If we condition on nothing, they are d-separated because the collider ($X_{1}$, $X_{2}$, $X_{3}$) leaves the path blocked. However, if we condition on $X_{2}$ or any of it's descendants the path becomes unblocked, because the rest of the path is made up of forks ($X_{3}$, $X_{4}$, $X_{5}$) and chains ($X_{2}$, $X_{3}$, $X_{4}$). If we condition on $X_{2}$ and $X_{3}$ the path becomes blocked again because the chain ($X_{2}$, $X_{3}$, $X_{4}$) is blocked.

We can check this with the following code:

In [9]:

print("Are x1 and x5 unconditional independent? {} "
      .format(path.is_d_separated("x1", "x5", {})))

print("Are x1 and x5 conditional independent when conditioning on x2? {} "
      .format(path.is_d_separated("x1", "x5", {"x2"})))

print("Are x1 and x5 conditional independent when conditioning on x2 and x3? {} "
      .format(path.is_d_separated("x1", "x5", {"x2", "x3"})))

Are x1 and x5 unconditional independent? True 
Are x1 and x5 conditional independent when conditioning on x2? False 
Are x1 and x5 conditional independent when conditioning on x2 and x3? True

We can read off all independence relationships implied by the graph in the sprinkler system using:

In [10]:

sprinkler.get_all_independence_relationships()

Out[10]:

[('slippery', 'rain', {'wet'}),
 ('slippery', 'rain', {'sprinkler', 'wet'}),
 ('slippery', 'rain', {'season', 'wet'}),
 ('slippery', 'rain', {'season', 'sprinkler', 'wet'}),
 ('slippery', 'season', {'wet'}),
 ('slippery', 'season', {'sprinkler', 'wet'}),
 ('slippery', 'season', {'rain', 'wet'}),
 ('slippery', 'season', {'rain', 'sprinkler'}),
 ('slippery', 'season', {'rain', 'sprinkler', 'wet'}),
 ('slippery', 'sprinkler', {'wet'}),
 ('slippery', 'sprinkler', {'season', 'wet'}),
 ('slippery', 'sprinkler', {'rain', 'wet'}),
 ('slippery', 'sprinkler', {'rain', 'season', 'wet'}),
 ('rain', 'sprinkler', {'season'}),
 ('season', 'wet', {'rain', 'sprinkler'}),
 ('season', 'wet', {'rain', 'slippery', 'sprinkler'})]

At this point it is worth emphasising that causal graphical models are non-parametric: they do not make any assumptions about the functional form of relationships between variables, only that they exist. Because of this the only testable assumption these models make are the conditional independence relationships between the variables. Unfortunately, testing conditional independence, in the general case, is impossible. Combined with the fact that there are many possible DAGs for even a reasonable number of variables, discovering causal structure from observational data alone is very difficult.

There are still some interesting approaches to identifying causal structure, but for these notes, it is best to think of the main use of causal graphical models as a way of explicitly encoding prior knowledge about the structure of a system, and to use this structure combined with observational data to make predictions about the effect of causal interventions.

From Bayesian networks to Causal Graphical Models¶

So far, our description of causal graphical models has been the same as those of general Bayesian Networks. To endow these structures with a notion of causality, we some assumptions about what happens when an intervention occurs. In causal graphical models, this is the notion of "Stability" or "Invariance" - that when we make an intervention on one variable, the structure of the causal graph and the functional relationships between the remaining variables remain the same.

If the assumption of invariance holds, the effect of an intervention on variable $X$ to remove the edges between the variable and it's parents. We typically denote a node with an intervention with a node with a double outline.

Imagine that we had the power the control the weather. If use it to make an intervention on the "rain" node of our sprinkler model, we get the following system:

In [11]:

sprinkler_do = sprinkler.do("rain")

print(sprinkler_do.get_distribution())

sprinkler_do.draw()

P(season)P(sprinkler|season)P(wet|do(rain),sprinkler)P(slippery|wet)

Out[11]:

Because causal graphical models are non-parametric, they cannot tell us what the relationship between two variables are, they only give us an idea if there is a relationship between the two variables through the notion of conditional independence. It does this using the idea of "paths" between variables: if there are no unblocked paths between two variables, they are independent. It also means that if two causal graphical models share the same paths between two variables, the conditional relationship between these two variables are the same.

For example, in the graph of out sprinkler system, $\Pc{slippery}{wet}$ is the same whether or not we make an intervention on $rain$, but $\Pc{slippery}{season}$ is not.

Describing the interventions in the way has some immediate consequences:

$X$ can only have some causal inference on $Y$ if there is at least one directed path between $X$ and $Y$. This is because if there is no directed path, with respect to the interventional graph the parents of $X$ have been removed, so $\In{X}{Y}$ in the intervential graph.
If there are only directed paths between $X$ and $Y$, then the the causal influence of $X$ on $Y$ is given by the simply by the conditional distribution $\Pc{Y}{X}$. This is because the interventional graph has the same paths between $X$ and $Y$ as the observational distribution.
If there is a unblocked, but not completely directed path between $X$ and $Y$, it means that both $X$ and $Y$ share a common ancestor. This common ancestor is what is called a confounder, and will mean that if we try to estimate $\Pc{Y}{\do{X}}$ from $\Pc{Y}{X}$ of estimates will be biased.

None of these results should be too surprising, but causal graphical models give us a way to quantify these understanding.

Causal Inference with Causal Graphical Models¶

Now that we have a way of describing how both observational and interventional distributions are generated and how they relate to each other, we can ask under what circumstances it is possible to make causal inferences from a system we only have observational samples from. This problem is often called "Identifiability".

To be specific, the question is under what circumstances can we estimate

$\Pc{Y}{\do{X}}$

from observational data, given some assumed causal graphical model?

It turns out we can do this via a set of manipulations known as do-calculus, a set of rules which together with the standard rules of manipulating probability distributions can allow expressions involving distributions conditioned by $\do{\dots}$ to be transformed into a form which involves no interventions.

I'm not going to cover the full score of do-calculus here, instead I'm going to explore some examples.

Let's start with the following causal graphical model:

In [12]:

from causalgraphicalmodels.examples import simple_confounded

simple_confounded.draw()

Out[12]:

Under intervention on $X$, the causal graphical model generating the data is

In [13]:

simple_confounded.do("x").draw()

Out[13]:

We are going to try and estimate the quantity

$\Pc{Y}{\do{X}}$

We can start by expanding this distribution using the standard rules of marginalization:

$\Pc{Y}{\do{X}} = \sum_{Z} \Pc{Y}{\do{X}, Z} \Pc{Z}{\do{X}}$

And because in the interventional distribution $\do{X}$ is independent of $Z$ we get

$\Pc{Y}{\do{X}} = \sum_{Z} \Pc{Y}{\do{X}, Z} \P{Z}$

Appealing to the assumption of stability, we assume that the conditional distribution of $Y$ given it's parents is takes the same form when an intervention takes place, we can write:

$\Pc{Y}{\do{X}} = \sum_{Z} \Pc{Y}{X, Z} \P{Z}$

Where there are no more references to $\do{\dots}$ on the right hand side of the equation: we succeeded writing an interventional distribution in terms of a observational distribution. It is worth comparing this with a similar formula for the observational conditional distribution:

$\Pc{Y}{X} = \sum_{Z} \Pc{Y}{X, Z} \Pc{Z}{X}$

When can we use the Adjustment Formula?¶

The formula we found in the previous section appears in a number of areas, and is something called the adjustment formula, or g-formula or backdoor adjustment formula. It states that under certain circumstances, for a set of variables $W$, we can estimate the the causal influence of $X$ on $Y$ with respect to a causal graphical model using the equation

$\Pc{Y}{\do{X}} = \sum_{W} \Pc{Y}{X, W} \P{W}$

The criterion for $W$ to exist is sometimes called the backdoor criterion. Graphically it states that

$W$ blocks all backdoor paths between $X$ and $Y$ (all paths with arrows going into $X$)
$W$ does not contain any descendants of $X$

Like in the previous section, these criteria are met when $W$ are the parents of $X$, but these aren't the only variables which can be used as an adjustment set. Consider the following graph:

In [14]:

from causalgraphicalmodels.examples import big_csm

example_cgm = big_csm.cgm
example_cgm.draw()

Out[14]:

There are two backdoor paths between $X$ and $Y$:

In [15]:

example_cgm.get_all_backdoor_paths("x", "y")

Out[15]:

[['x', 'a', 'h', 'y'], ['x', 'b', 'd', 'e', 'y']]

But because $h$ acts as a collider in the first path, it is blocked unless conditioned on. To find a valid adjustment set, we need a set which blocks this path. Any of the variables $B$, $D$, $E$ would work, as well as any combination of the above. We can also include any other variable in this set, as long as it doesn't create new paths. Adding $H$, $F$ or $C$ to the adjustment set would create a new path, making the adjustment set invalid:

In [16]:

example_cgm.is_valid_backdoor_adjustment_set("x", "y", {"b", "d", "e"})

Out[16]:

True

In [17]:

example_cgm.is_valid_backdoor_adjustment_set("x", "y", {"b", "d", "e", "h"})

Out[17]:

False

We can compute all valid adjustment sets using the following:

In [18]:

example_cgm.get_all_backdoor_adjustment_sets("x", "y")

Out[18]:

frozenset({frozenset({'a', 'd', 'e'}),
           frozenset({'b', 'd'}),
           frozenset({'a', 'e'}),
           frozenset({'a', 'b'}),
           frozenset({'a', 'b', 'e'}),
           frozenset({'d'}),
           frozenset({'e'}),
           frozenset({'d', 'e'}),
           frozenset({'a', 'd'}),
           frozenset({'b'}),
           frozenset({'a', 'b', 'd'}),
           frozenset({'b', 'e'}),
           frozenset({'b', 'd', 'e'}),
           frozenset({'a', 'b', 'd', 'e'})})

For a full proof of the backdoor criteria, I suggest chapter 6 of the excellent book Elements of Causal Inference, but some intuition about these requirements is that blocking all backdoor path accounts for any bias introduced by confounding variables, and the requirement that no descendants are conditioned on prevents any new paths from being created.

When all variables in a causal graphical model are observed, is alway a set which can be used for adjustment. If not all variables are observed, there can be causal statements which cannot be estimated from the observed data.

Relationship with Potential Outcomes¶

Given the focus on potential outcomes that my previous post had, it is worth asking what the relationship between causal graphical models and them is. It turns out that the two approaches are exactly equivalent to each other, and a statement in one can be related to a statement in the other.

There is a slight subtle point that so far I've treated causal graphical models as a way to make statements about causal inference (how data would be generated if there is an intervention on the system), whereas potential outcomes describe counterfactual inference (what would have happened to a system which had already been observed, if a different treatment had been applied), but it is possible to use CGMs/SCMs to reason about counterfactuals, although it is beyond these notes.

That said, one of the main benefits of explicitly drawing out a causal graphical model of your system is that it makes clear whether or not conditional independence statements like

$\Cin{Y_{i}}{X}{Z}$

are valid.

To see how statements like this fit into CGMs, we need to start by expanding our system to include the potential outcomes:

In [19]:

from causalgraphicalmodels.examples import simple_confounded_potential_outcomes

simple_confounded_potential_outcomes.draw()

Out[19]:

Here $Y_{i}$ is the potential outcome for $X=i$, and the node $Y$ now the deterministic function $Y(X, Y_{1}, Y_{0}) = Y_{X}$.

By examination, it should be clear that $\Cin{Y_{i}}{X}{Z}$ holds.

Looking at the graph, we see that by construction there are no direct paths from $X$ to the potential outcomes $Y_{i}$, only backdoor paths. This means the conditional independence statement holds as long as the set $Z$ blocks all backdoor paths between $Y$ and $X$, and does not create any new one. This is exactly the backdoor criteria.

Actually Estimating Causal Effects¶

In our example of the simple_confounding causal graphical model, we showed that to make estimates of $\Pc{Y}{\do{X}}$ we need a way of estimating the quantity $\Pc{Y}{X, Z}$ from our observational data.

Causal graphical models don't offer a way of doing this. They are a tool for answering question about if we can make causal inferences, given an assumed structure, and how we would do this if we can estimate statistical quantities like $\Pc{Y}{X, Z}$. However, there are a number of statistical techniques we can use to make this estimate. For example, my previous most covered a number of ways to estimate the quantity $E[Y|X,Z]$ from potentially biased observations, which is often what we care about instead of the full distribution.

Example Use Case¶

We've been though a lot to reach this point, so it is good to reflect on how we can actually use causal graphical models to do something useful.

Let's take another look at the following structural causal model:

In [20]:

big_csm.cgm.draw()

Out[20]:

The structure in this model are just linear relationships between variables with Gaussian noise, with the exception of $X$, which is Bernoulli distributed. The whole thing is encapsulated in a StructuralCausalModel object which makes it easy to sample observational data from:

In [21]:

big_csm.sample(5)

Out[21]:

	a	b	c	d	e	f	h	x	y
0	0.696485	0.849004	0.512328	-1.877430	-2.981484	3.024886	6.432555	0	4.690296
1	-0.960686	-2.621972	1.094445	3.105839	7.202387	-3.319097	-8.013940	0	-4.438694
2	-1.240862	0.825594	1.555216	-0.872669	-3.293294	4.543829	7.927069	1	7.820964
3	-1.794861	-0.960123	0.257391	0.759459	1.073854	-3.124381	-6.344803	0	-2.290865
4	0.458812	-1.653600	0.474086	2.501353	4.918685	-2.474993	-3.763032	0	-5.151883

Our goal will be to estimate the Average Treatment Effect of $X$ on $Y$. I will be using the 'causalinference' package to do this. The main question is which covariants should we use to adjust?

Let's start by defining a function which calculates the ATE given variable adjustment sets:

In [22]:

from causalinference import CausalModel

def estimate_ate(dataset, adjustment_set=None, method="matching"):
    """
    Estimate the ATE of X on Y from from dataset when 
    adjusting using adjustment_set.
    
    Arguments
    ---------
    dataset: pd.DateFrame
        dataframe of observations
        
    adjustment_set: iterable of variables or None
    
    method: str
        adjustment method to use.    
    """
    
    if adjustment_set is None:
        y0 = dataset.loc[lambda df: df.x == 0].y.mean()
        y1 = dataset.loc[lambda df: df.x == 1].y.mean()
        
        y0_var = dataset.loc[lambda df: df.x == 0].y.var()
        y1_var = dataset.loc[lambda df: df.x == 1].y.var()
        
        y0_n = dataset.loc[lambda df: df.x == 0].shape[0]
        y1_n = dataset.loc[lambda df: df.x == 1].shape[0]
        
        return {
            "ate": y1 - y0,
            "ate_se": 2 * np.sqrt(y0_var/y0_n + y1_var/y1_n)
        }

    cm = CausalModel(
        Y=dataset.y.values, 
        D=dataset.x.values, 
        X=dataset[adjustment_set].values
    )

    cm.est_via_ols()
    cm.est_via_matching()
    cm.est_propensity_s()
    cm.est_via_weighting()

    cm.stratify_s()
    cm.est_via_blocking()
    
    results = {
        "ate": cm.estimates[method]["ate"],
        "ate_se": cm.estimates[method]["ate_se"]
    }

    return results

In [23]:

n_samples = 10000

ds = big_csm.sample(n_samples)

# this allows us to generate samples from an interventional distribution
# where the value of X is assigned randomly as in an A/B test.
ds_intervention = (
    big_csm
    .do("x")
    .sample(
        n_samples=1000000, 
        set_values={"x": np.random.binomial(p=0.5, n=1, size=1000000)})
)

true_ate = estimate_ate(ds_intervention)["ate"]

# generate results for a number of different adjustment sets
results = {
    "no_adjustment": estimate_ate(ds),
    "adjustment_b": estimate_ate(ds, ["b"]),
    "adjustment_bde": estimate_ate(ds, ["b", "d", "e"]),
    "adjustment_bh": estimate_ate(ds, ["b", "h"]),
    "adjustment_bc": estimate_ate(ds, ["b", "c"]),
    "adjustment_everything": estimate_ate(ds, ["a", "b", "c", "d", "e", "f", "h"]),

}

# plot the results
x_label = list(results.keys())
x = np.arange(len(x_label))
y = [results[l]["ate"] for l in x_label]
yerr = [results[l]["ate_se"] for l in x_label]

plt.figure(figsize=(10,6))
plt.errorbar(x=x, y=y, yerr=yerr, linestyle="none", capsize=5, marker="o")
plt.xticks(x, x_label, rotation=45, fontsize=16)
plt.title("Estimated ATE Size", fontsize=18)
xmin, xmax = plt.xlim()
plt.hlines(true_ate, xmin, xmax, linestyles="dashed")

Out[23]:

<matplotlib.collections.LineCollection at 0x7fce58abb908>

We can see that when we use the valid adjustment sets we get close to the true value, but excluding or including extra variables can make things worse. This is hopefully a warning about why it is not a good idea to use all the features you have to correct for observational bias - you should always considering the structure that generated the data. Causal Graphical Models are an excellent way to do this.

Conclusions¶

Hopefully these notes have been a useful introduction Causal Graphical Models. There is still a lot that I haven't had a change to cover. Very loosely, work with them can be categorised in two area:

Identification: Given assumptions about the causal structure of a system, can we estimate influence of one variable on another.
Discovery: Given data, how can we estimate the causal structure of a system.

The backdoor adjustment method is one example of identification discussed in these notes - it is not the only one. In my next post I will look at how we can use mediating variables to make causal inferences. More generally, it has been shown that for partly observed causal graphical models it is possible to decide exactly whether or not a causal statement is identifiable and the rules of do-calculus are complete.

In the other direction, learning causal structure is much more difficult. There are some tools out there to estimate structure, but in my experience, when guided only by the data they tend to produce nonsensical results. I suspect this is because given only limited observational data there often is not enough information to accurately estimate structure. There is some interested work on how to use combinations of observational data from different setting to try and infer structure, but I think in general this is an open problem.

I hope to have a chance to discuss more of these areas in future posts.

Code¶

You can find the notebook for this post on github here.

Causal Inference With Python Part 1 - Potential Outcomes

2018-03-24T00:00:00+00:00

In [1]:

from __future__ import division

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style("whitegrid")
sns.set_palette("colorblind")

%matplotlib inline

import datagenerators as dg

In this post, I will be using the excellent CausalInference package to give an overview of how we can use the potential outcomes framework to try and make causal inferences about situations where we only have observational data. The author has a good series of blog posts on it's functionality.

Because most datasets you can download are static, throughout this post I will be using be using my own functions to generate data. This has two advantages: we can and will generate datasets with specific properties, and we have the ability to "intervene" in the data generating system directly, giving us the ability to check whether our inferences are correct. These data generators all generate i.i.d. samples from some distribution, returning the results as a pandas dataframe. You can find the functions which generate these datasets in the accompanying file datagenerators.py on github here.

To begin, let's look at a motivating example.

Introduction¶

One day a team lead notices that some members of their team wear cool hats, and that these members of the team tend to be less productive. Being data drive, the Team Lead starts to record whether or not a team member wears a cool hat ($X=1$ for a cool hat, $X=0$ for no cool hat) and whether or not they are productive ($Y=1$ for productive, $Y=0$ for unproductive).

After making observations for a week, they end up with a dataset like the following:

In [2]:

observed_data_0 = dg.generate_dataset_0()

observed_data_0.head()

Out[2]:

	x	y
0	0	1
1	1	0
2	1	0
3	1	0
4	0	1

The first question the team lead asks is: are people wearing cool hats more likely to be productive that those who don't? This means estimating the quantity

$P(Y=1|X=1) - (Y=1|X=0)$

which we can do directly from the data:

In [3]:

def estimate_uplift(ds):
    """
    Estiamte the difference in means between two groups.
    
    Parameters
    ----------
    ds: pandas.DataFrame
        a dataframe of samples.
        
    Returns
    -------
    estimated_uplift: dict[Str: float] containing two items:
        "estimated_effect" - the difference in mean values of $y$ for treated and untreated samples.
        "standard_error" - 90% confidence intervals arround "estimated_effect"
        
        
    """
    base = ds[ds.x == 0]
    variant = ds[ds.x == 1]
    
    delta = variant.y.mean() - base.y.mean()
    delta_err = 1.96 * np.sqrt(
        variant.y.var() / variant.shape[0] + 
        base.y.var() / base.shape[0])
    
    return {"estimated_effect": delta, "standard_error": delta_err}

estimate_uplift(observed_data_0)

Out[3]:

{'estimated_effect': -0.16372191280967924,
 'standard_error': 0.086646375506673382}

It looks like people with cool hats are less productive.

To be sure, we can even run a statistical test:

In [4]:

from scipy.stats import chi2_contingency

contingency_table = (
    observed_data_0
    .assign(placeholder=1)
    .pivot_table(index="x", columns="y", values="placeholder", aggfunc="sum")
    .values
)

_, p, _, _ = chi2_contingency(contingency_table, lambda_="log-likelihood")

# p-value
p

Out[4]:

0.00034453601398614649

That's one small p-value. Staticians would be proud.

We can use this information to make statements about what we might think about someone's probability if we see them wearing a cool hat. As long as we believe that they are "drawn from the same distribution" as our previous observations, we expect the same correlations to exist.

The problem comes if we try to use this information as an argument about whether or not the team lead should force people to wear cool hats. If the team lead does this they fundamentally change the system we are sampling from, potentially altering or even reversing any correlations we observed before.

The cleanest way to actually measure the effect of some change in a system is by running a randomized control trial. Specifically, we want to randomize who gets cool hats and who doesn't, and look at the different values of $y$ we receive. This removes the effect of any confounding variables which might be influencing the metric we care about.

Because we generated our dataset from a known process (in this case a function I wrote), we can intervene in it directly and measure the effect of an A/B test:

In [5]:

def run_ab_test(datagenerator, n_samples=10000, filter_=None):
    """
    Generates n_samples from datagenerator with the value of X randomized
    so that 50% of the samples recieve treatment X=1 and 50% receive X=0,
    and feeds the results into `estimate_uplift` to get an unbiased 
    estimate of the average treatment effect.
    
    Returns
    -------
    effect: dict
    """
    n_samples_a = int(n_samples / 2)
    n_samples_b = n_samples - n_samples_a
    set_X = np.concatenate([np.ones(n_samples_a), np.zeros(n_samples_b)]).astype(np.int64)
    ds = datagenerator(n_samples=n_samples, set_X=set_X)
    if filter_ != None:
        ds = ds[filter_(ds)].copy()
    return estimate_uplift(ds)

run_ab_test(dg.generate_dataset_0)

Out[5]:

{'estimated_effect': 0.2026, 'standard_error': 0.019195426971237748}

Suddenly, it looks like the direction of the effect of wearing cool hats has reversed.

What's going on?

Note: In the above example, and in all following examples, I'm assuming that our samples are i.i.d., and obey the Stable unit treatment value assumption (SUTVA). Basically this means that when one person chooses, or is forced to wear a really cool hat they have no influence on the choice or effect of another person wearing a really cool hat. By construction, the synthetic datagenerators I use all have this property. In reality it is yet another thing you have to assume to be true.

Definitions of Causality¶

The previous example demonstrates the old statistics saying:

Correlation Does Not Imply Causation.

"Causality" is a vague, philosophical sounding word. In the current context, I am using it to mean "What is the effect on $Y$ of changing $X$?"

To be precise, $X$ and $Y$ are random variables and the "effect" we want to know is how the distribution of $Y$ will change when we force $X$ to take a certain value. This act of forcing a variable to take a certain value is called an "Intervention".

In the previous example, when we make no intervention on the system, we have an observational distribution of $Y$, conditioned on the fact we observe $X$:

$P(Y|X)$

When we force people to wear cool hats, we are making an intervention. The distribution of $Y$ is then given by the interventional distribution

$P(Y|\hbox{do}(X))$

In general these two are not the same.

The question these notes will try and answer is how we can reason about the interventional distribution, when we only have access to observational data. This is a useful question because there are lots of situations where running an A/B test to directly measure the effects of an intervention is impractical, unfeasable or unethical. In these situations we still want to be able to say something about what the effect of an intervention is - to do this we need to make some assumptions about the data generating process we are investigating.

Potential Outcomes¶

One way to approach this problem is to introduce two new random variables to our system: $Y_{0}$ and $Y_{1}$, known as the Potential Outcomes. We imagine that these variables exist, and can be treated as any other random variable - the only difference is that they are never directly observed. $Y$ is defined in terms of

$Y = Y_{1}$ when $X=1$
$Y = Y_{0}$ when $X=0$

This shifts the problem from one about how distributions change under the intervention, to one about data drawn i.i.d. from some underlying distribution with missing values. Under certain assumptions about why values are missing, there is well developed theory about how to estimate the missing values.

Goals¶

Often we do not care about the full interventional distribution, $P(Y|\hbox{do}(X))$, and it is enough to have an estimate of the difference in means between the two groups. This is a quantity known as the Average Treatment Effect:

$\Delta = E[Y_{1} - Y_{0}]$

When we run and A/B test and compare the means of each group, this is directly the quantity we are measuring

If we just try and estimate this quantity from the observational distribution, we get:

$\Delta_{bad} = E[Y|X=1] - E[Y|X=0] \\ = E[Y_{1}|X=1] - E[Y_{0}|X=0] \\ \neq \Delta$

This is not generally equal to the true ATE because:

$E[Y_{i}|X=i] \neq E[Y_{i}]$

Two related quantities are

$ATT = E[Y_{1} - Y_{0}|X=1]$, the "Average Treatment effect of the Treated"
$ATC = E[Y_{1} - Y_{0}|X=0]$, the "Average Treatment effect of the Control"

One way to interpret ATC is as a measure of the effect of treating only samples which wouldn't naturally be treated, and vice versa for ATT. Depending on your use case, they may be more natural measures of what you care about. The following techniques will allow us to estimate them all.

$\def\ci{\perp\!\!\!\perp}$

Making Assumptions¶

When we A/B test, we randomize the assignment of $X$. This has the effect of allowing us to choose which variable of $Y_{1}$ or $Y_{0}$ is revealed to us. This makes the outcome independent of the value of $X$. We write this as

$Y_{1}, Y_{0} \ci X$

Which means that the distribution of $X, Y_{0}, Y_{1}$ factorizes as

$P(X, Y_{0}, Y_{1}) = P(X)P(Y_{0}, Y_{1})$

If this independence holds then

$E[Y_{1}|X=1] = E[Y_{1}]$

If we want to estimate the ATE using observational data, we need to use other information we have about the samples - specifically we need to assume that we have enough additional information to completely explain the choice of treatment each subject.

If we call the additional information the random variable $Z$, we can write this assumption as

$Y_{1}, Y_{0} \ci X \, | \, Z$

$P(X, Y_{0}, Y_{1}| Z) = P(X|Z)P(Y_{0}, Y_{1}|Z)$

This means that the observed treatment a sample receives, $X$, is completely explained by $Z$. This is sometimes called the "ignorability" assumption.

In our motivating example about cool hats this would mean that there is some other factor - let's call it "skill" - which impacts both the productivity of the person and whether or not they wear a cool hat. In our example above, skilled people are more likely to be productive and also less likely to were cool hats. These facts together could explain why the effect of cool hats seemed to reverse when ran an A/B test.

If we split our data on whether or not the person is skilled, we find that for each subgroup there is a positive relationship between wearing cool hats and productivity:

In [6]:

observed_data_0_with_confounders = dg.generate_dataset_0(show_z=True)

print(estimate_uplift(observed_data_0_with_confounders.loc[lambda df: df.z == 0]))
print(estimate_uplift(observed_data_0_with_confounders.loc[lambda df: df.z == 1]))

{'estimated_effect': 0.21970895286301803, 'standard_error': 0.16384029461180688}
{'estimated_effect': 0.10869565217391297, 'standard_error': 0.19384986576424423}

Unfortuntly, because we never observe $Y_{0}$ and $Y_{1}$ for the same sample, we cannot test the assumption that

$Y_{1}, Y_{0} \ci X \, | \, Z$

It is something we have to use our knownledge of the system we are investigating to evaluate.

The quality of any prediction you make depends on exactly how well this assumption holds. Simpson's Paradox is an extreme example of the fact that if $Z$ does not contain all confounding variables, then any inference we make could be wrong. Facebook has a good paper comparing different causal inference approaches with direct A/B test that show how effects can be overestimated when conditional independence doesn't hold.

Once we have made this assumption there are a number of techniques for approaching this. I will outline a few of simpler approaches in the rest of the post, but keep in mind that this is an area of ongoing research.

Modeling the Counterfactual¶

From the above, it should be clear that if know $Y_{0}$ and $Y_{1}$, we can estimate the ATE. So why not just try and model them directly? Specifically we can build estimators:

$\hat{Y}_{0}(Z) = E[Y|Z, X=0]$
$\hat{Y}_{1}(Z) = E[Y|Z, X=1]$.

If we can model these two quantities, we can estimate the ATE as:

$\Delta = \frac{1}{N}\sum_{i}(\hat{Y}_{1}(z_{i}) - \hat{Y}_{0}(z_{i}))$

The success of this approach depends on how well we can model the potential outcomes. To see it in action, let's use the following data generating process:

In [7]:

observed_data_1 = dg.generate_dataset_1()

observed_data_1.plot.scatter(x="z", y="y", c="x", cmap="rainbow", colorbar=False);

Before jumping into modelling the counterfactual, let's look at the data. If we look at how $Y$ is distributed, there appears to be a small difference between the two groups:

In [8]:

sns.kdeplot(observed_data_1.loc[lambda df: df.x == 0].y, label="untreated")
sns.kdeplot(observed_data_1.loc[lambda df: df.x == 1].y, label="treated")

Out[8]:

<matplotlib.axes._subplots.AxesSubplot at 0x7ff4b50a9630>

We can confirm this by looking at the difference in means between the two groups

In [9]:

print("Observed ATE: {estimated_effect:.3f} ({standard_error:.3f})".format(**estimate_uplift(observed_data_1)))

Observed ATE: 0.230 (0.095)

However, if we look at the distribution of the covariance, $Z$, it is clear that there is a difference between the groups.

In [10]:

sns.kdeplot(observed_data_1.loc[lambda df: df.x == 0].z, label="untreated")
sns.kdeplot(observed_data_1.loc[lambda df: df.x == 1].z, label="treated")

Out[10]:

<matplotlib.axes._subplots.AxesSubplot at 0x7ff4b50a90b8>

If we believe that $Z$ has some infulance on the metric $Y$, this should concern us. We need some way to disentangle the effect of $X$ on $Y$ and the effect of $Z$ on $Y$.

We can check the actually ATE using our simulated A/B test and confirm that it is difference of the observed value:

In [11]:

print("Real ATE:  {estimated_effect:.3f} ({standard_error:.3f})".format(**run_ab_test(dg.generate_dataset_1)))

Real ATE:  -0.477 (0.026)

But what happens if we cannot run this A/B test? We need to resort to modelling the system.

The simplest type of model we can use is a linear model. Specifically we could assume

$Y_{0} = \alpha + \beta Z + \epsilon$

$Y_{1} = Y_{0} + \gamma$

If this is accurate, fitting the model

$Y = \alpha + \beta Z + \gamma X$

to the data using linear regression will give us an estimate of the ATE.

The causalinference package gives us a simple interface to do this:

In [12]:

from causalinference import CausalModel

cm = CausalModel(
    Y=observed_data_1.y.values, 
    D=observed_data_1.x.values, 
    X=observed_data_1.z.values)

cm.est_via_ols(adj=1)

print(cm.estimates)

Treatment Effect Estimates: OLS

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE     -0.440      0.033    -13.275      0.000     -0.505     -0.375

causalinference returns an estimate of the ATE, along with some statistical properties of the estimate. It is important to realise that the confidence intervals reported for the estimates are the confidence intervals if we assume the model accurately describes the counterfactual, not confidence intervals about how well the the model describes the counterfactual.

In this case the package has done well in identifying the correct ATE - which is good, but the data generating process was specifically designed to meet the assumptions. Let's look at a few cases where it might fail.

The first is when the effect is not simply additive:

In [13]:

observed_data_2 = dg.generate_dataset_2()

observed_data_2.plot.scatter(x="z", y="y", c="x", cmap="rainbow", colorbar=False)

print("Observed ATE: {estimated_effect:.3f} ({standard_error:.3f})".format(**estimate_uplift(observed_data_2)))
print("Real ATE:  {estimated_effect:.3f} ({standard_error:.3f})".format(**run_ab_test(dg.generate_dataset_2)))

Observed ATE: 0.768 (0.111)
Real ATE:  0.572 (0.030)

In [14]:

cm = CausalModel(
    Y=observed_data_2.y.values, 
    D=observed_data_2.x.values, 
    X=observed_data_2.z.values)

cm.est_via_ols(adj=1)

print(cm.estimates)

Treatment Effect Estimates: OLS

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE      0.349      0.082      4.241      0.000      0.188      0.510

Usually this can be overcome by using more powerful estimators . A simple, non-parametric approach, is the technique of matching. The idea is to find for each sample which received the treatment, a similar samples which did not receive the treatment, and to directly compare these values. Exactly what you mean by "similar" will depend on your specific usecase.,

The package causalinference implements matching by selecting for each unit, with replacement, the most similar unit from the other treatment group and using the difference between these two units to calculate the ATE. By default, the choice of match is chosen to be the nearest neighbour in covariate space $Z$, with the distances weighted by inverse variance of each dimension.

There are options to change the number of units compared and the weighting of each dimension in the match. For more details, see the documentation.

We can compute the matching estimate with the following code

In [15]:

cm = CausalModel(
    Y=observed_data_2.y.values, 
    D=observed_data_2.x.values, 
    X=observed_data_2.z.values)

cm.est_via_matching()

print(cm.estimates)

Treatment Effect Estimates: Matching

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE      0.597      0.139      4.310      0.000      0.326      0.869
           ATC      1.032      0.150      6.878      0.000      0.738      1.326
           ATT      0.176      0.184      0.959      0.338     -0.184      0.537

The confidence intervals around our estimate now contain the true ATE.

Covariate Imbalance¶

A more difficult problem to deal with is when the covariates you are using are imbalanced: when there are areas of covariate space which contains only the treated or untreated samples. Here we have to extrapolate the effect of the treatment - which will depend heavily on assumptions model we use.

The example below demonstrates this:

In [16]:

observed_data_3 = dg.generate_dataset_3()

observed_data_3.plot.scatter(x="z", y="y", c="x", cmap="rainbow", colorbar=False)

# actual response curves
z = np.linspace(0,1,100)
y0 =  np.where(z >= 0.4, -4*(z - 0.4), 0)
y1 =  np.where(z < 0.6,  -4*(z - 0.6), 0) + 1
plt.plot(z,y0, "b")
plt.plot(z,y1, "r")


print("Observed ATE: {estimated_effect:.3f} ({standard_error:.3f})".format(**estimate_uplift(observed_data_3)))
print("Real ATE:  {estimated_effect:.3f} ({standard_error:.3f})".format(**run_ab_test(dg.generate_dataset_3)))

Observed ATE: 1.348 (0.083)
Real ATE:  2.402 (0.033)

In [17]:

# OLS estimator
cm = CausalModel(
    Y=observed_data_3.y.values, 
    D=observed_data_3.x.values, 
    X=observed_data_3.z.values)

cm.est_via_ols()

print(cm.estimates)

Treatment Effect Estimates: OLS

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE      1.981      0.055     36.048      0.000      1.874      2.089
           ATC      2.002      0.063     31.971      0.000      1.879      2.124
           ATT      1.967      0.069     28.517      0.000      1.831      2.102

In [18]:

# Matching estimator
cm = CausalModel(
    Y=observed_data_3.y.values, 
    D=observed_data_3.x.values, 
    X=observed_data_3.z.values)

cm.est_via_matching()

print(cm.estimates)

Treatment Effect Estimates: Matching

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE      1.991      0.179     11.113      0.000      1.639      2.342
           ATC      2.106      0.257      8.185      0.000      1.602      2.611
           ATT      1.906      0.242      7.866      0.000      1.431      2.381

The OLS estimator fails to capture the true effect, and while the matching estimator improves things a bit, there just isn't enough information in the data to extrapolate fully into areas where there isn't overlap.

This example might seem contrived - that's because it is, but once we start looking covariates with higher dimensionality this issue can become much more common.

causalinference provides a useful tool to quickly assess the overlap of the variables using the summary_stats property:

In [19]:

print(cm.summary_stats)

Summary Statistics

                       Controls (N_c=211)         Treated (N_t=289)             
       Variable         Mean         S.d.         Mean         S.d.     Raw-diff
--------------------------------------------------------------------------------
              Y       -0.154        0.469        1.193        0.471        1.348

                       Controls (N_c=211)         Treated (N_t=289)             
       Variable         Mean         S.d.         Mean         S.d.     Nor-diff
--------------------------------------------------------------------------------
             X0        0.271        0.203        0.688        0.192        2.106

Where the Normalized difference is defined as

$\frac{\bar{X}_{T} - \bar{X}_{T}}{ (\sigma^{2}_{T} + \sigma^{2}_{C})/2 } $

While it isn't a strict statistical test, it provides some indication how much overlap there is between each covariate. Values greater than one suggest there isn't much overlap.

Propensity Score¶

The Propensity score is a estimate of how likely it is for a subject to have ended up with the treatment, given the covariates:

$\hat{p}(Z) = P(X|Z)$

We can estimate this however we like, but once we have it there are a number of things we can do with it.

Inverse Propensity Score Weighting¶

Remember that the problem of measuring causal inference is that we want to know the quantity $E[Y_{i}]$, but we only have access to samples from $E[Y_{i}|X=i]$.

The probability of a potential outcome can be expanded to give

$P(Y_{i}) = P(Y_{i}| X = i)P(X = i)$

This suggests that we can estimate the true

$E[Y_{i}] = E[\frac{Y_{i}}{P(X=i|Z)}P(X=i|Z)] = E[\frac{Y_{i}}{P(X=i|Z)}|X=i, Z]$

So if we weight each point by it's inverse propensity, we can recover the potential outcomes. The result is the inverse propensity score weight estimator:

$\Delta_{IPS} = \frac{1}{N}\left(\sum_{i \in 1} \frac{y_{i}}{\hat{p}(z_{i})} - \sum_{i \in 0} \frac{y_{i}}{1 - \hat{p}(z_{i})}\right)$

Let's see how it does one of our previous datasets:

In [20]:

observed_data_1 = dg.generate_dataset_1()

observed_data_1.plot.scatter(x="z", y="y", c="x", cmap="rainbow", colorbar=False)

print("Observed ATE: {estimated_effect:.3f} ({standard_error:.3f})".format(**estimate_uplift(observed_data_1)))
print("Real ATE:  {estimated_effect:.3f} ({standard_error:.3f})".format(**run_ab_test(dg.generate_dataset_1)))

Observed ATE: 0.084 (0.096)
Real ATE:  -0.506 (0.026)

We can estimate the propensity using the CausalInference package's methods est_propensity_s or est_propensity, which uses logistic regression on the covariate to estimate propensity:

In [21]:

cm = CausalModel(
    Y=observed_data_1.y.values, 
    D=observed_data_1.x.values, 
    X=observed_data_1.z.values)

cm.est_propensity_s()

propensity = cm.propensity["fitted"]

df = observed_data_1

df["ips"] = np.where(
    df.x == 1, 
    1 / propensity,
    1 / (1 - propensity))
df["ipsw"] = df.y * df.ips

ipse = (
      df[df.x == 1]["ipsw"].sum() 
    - df[df.x == 0]["ipsw"].sum()
) / df.shape[0]

ipse

Out[21]:

-0.54484467207335452

This does well in our situation - by is very dependent on how good our estimate of the propensity score is - for the data generator we're using for this example the relationship can be described well by plain logistic regression. If we tried to estimate the propensity using, say, sklean's logistic regression function, which by default uses regularization, we would have got the wrong answer:

In [22]:

from sklearn.linear_model import LogisticRegression

lg = LogisticRegression()
X = df.z.values.reshape(-1,1)
y = df.x.values
lg.fit(X,y)

propensity = lg.predict_proba(X)[:,1]

df["ips"] = np.where(
    df.x == 1, 
    1 / propensity,
    1 / (1 - propensity))
df["ipsw"] = df.y * df.ips

ipse = (
      df[df.x == 1]["ipsw"].sum() 
    - df[df.x == 0]["ipsw"].sum()
) / df.shape[0]

ipse

Out[22]:

-0.32827891787280261

It does better than our naive estimator, but is not correct.

Doubly Robust Weighted Estimator¶

We can combine the inverse propensity score weighting estimators and the linear estimator of effect size together to try and reduce the flaws in either model. This is done by preforming weighted linear regression on the data, with each point weighted by the inverse propensity score. The result is the doubly robust weighted estimator.

The idea is that points because there is an bias in which samples are treated in the observational data, the samples which were treated, but were unlikely to have been, are more import and should be given more weight.

We can apply it using the following:

In [23]:

observed_data_1 = dg.generate_dataset_1()

observed_data_1.plot.scatter(x="z", y="y", c="x", cmap="rainbow", colorbar=False)

print("Observed ATE: {estimated_effect:.3f} ({standard_error:.3f})".format(**estimate_uplift(observed_data_1)))
print("Real ATE:  {estimated_effect:.3f} ({standard_error:.3f})".format(**run_ab_test(dg.generate_dataset_1)))

Observed ATE: 0.114 (0.099)
Real ATE:  -0.502 (0.026)

In [24]:

cm = CausalModel(
    Y=observed_data_1.y.values, 
    D=observed_data_1.x.values, 
    X=observed_data_1.z.values)

cm.est_propensity_s()
cm.est_via_weighting()

print(cm.estimates)

Treatment Effect Estimates: Weighting

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE     -0.502      0.037    -13.609      0.000     -0.575     -0.430

In [25]:

cm.estimates

Out[25]:

{'weighting': {'ate': -0.50245509578049252, 'ate_se': 0.036921043251613585}}

Unconfoundedness and the Propensity Score¶

In the previous sections, we assumed that the outcomes and the treatment were independent given our covariates:

$Y_{1}, Y_{0} \ci X \, | \,Z$

We can also assume something slightly stronger: that the outcomes are independent of the treatment, conditioned on the probability of the propensity:

$Y_{1}, Y_{0} \ci X \, | \,\hat{p}(Z)$

With this assumption, we potentially reduce the dimensionality of the confounding variables. This allows us to perform several techniques which may not work in higher dimensional settings.

Trimming¶

We previously saw that imbalances in covariates can create issues. A simple solution is to only make predictions for the counterfactual in regions where there is a good overlap, or "trim" points where there is not good overlap. For high dimensional data "good overlap" can be difficult to define - using just the propensity score to define overlap is one way to solve this.

Let's look at a dataset with low overlap:

In [26]:

observed_data_3 = dg.generate_dataset_3()

observed_data_3.plot.scatter(x="z", y="y", c="x", cmap="rainbow", colorbar=False)

# actual response curves
z = np.linspace(0,1,100)
y0 =  np.where(z >= 0.4, -4*(z - 0.4), 0)
y1 =  np.where(z < 0.6,  -4*(z - 0.6), 0) + 1
plt.plot(z,y0, "b")
plt.plot(z,y1, "r")


print("Observed ATE: {estimated_effect:.3f} ({standard_error:.3f})".format(**estimate_uplift(observed_data_3)))
print("Real ATE:  {estimated_effect:.3f} ({standard_error:.3f})".format(**run_ab_test(dg.generate_dataset_3)))

Observed ATE: 1.390 (0.076)
Real ATE:  2.422 (0.033)

CausalInference offers a method to trim the data based on the propensity score

In [27]:

# OLS estimator
cm = CausalModel(
    Y=observed_data_3.y.values, 
    D=observed_data_3.x.values, 
    X=observed_data_3.z.values)

cm.est_propensity_s()
cm.trim_s()
cm.est_via_matching()

print(cm.estimates)

Treatment Effect Estimates: Matching

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE      1.923      0.105     18.272      0.000      1.717      2.129
           ATC      2.016      0.195     10.332      0.000      1.634      2.399
           ATT      1.840      0.073     25.219      0.000      1.697      1.983

We can look at the remaining data in the following way

In [28]:

# mask out data ignored by the 
propensity = cm.propensity["fitted"]
cutoff = cm.cutoff
mask = (propensity > cutoff) &  (propensity < 1 - cutoff)

# plot the data
observed_data_3[mask].plot.scatter(x="z", y="y", c="x", cmap="rainbow", colorbar=False)

# actual response curves
z = np.linspace(0,1,100)
y0 =  np.where(z >= 0.4, -4*(z - 0.4), 0)
y1 =  np.where(z < 0.6,  -4*(z - 0.6), 0) + 1
plt.plot(z,y0, "b")
plt.plot(z,y1, "r")

# filter out data in regions we have trimmed when we calculate the true uplift
filter_ = lambda df: (df.z > 0.2) & (df.z < 0.7)

print("Observed ATE: {estimated_effect:.3f} ({standard_error:.3f})".format(**estimate_uplift(observed_data_3[mask])))
print("Real ATE:  {estimated_effect:.3f} ({standard_error:.3f})".format(
    **run_ab_test(dg.generate_dataset_3, filter_= filter_)))

Observed ATE: 1.694 (0.110)
Real ATE:  2.007 (0.031)

It does quite well in there cases.

When we apply trimming, we are explicitly saying that it is only possible to make causal inferences for samples in some part of the covariate space. For samples outside these regions, we cannot say anything about the ATE.

Stratification¶

Another use of the propensity score is the stratification, or blocking, estimator. It consist of grouping the data points into groups of similar propensity, and to estimate the ATE within these groups. Again, CausalInference provides a nice interface to achieve this.

We use the stratify (for user defined stata boundaries) or stratify_s (to automatically choose the boundaries) methods to determine the strata:

In [29]:

observed_data_1 = dg.generate_dataset_1()

cm = CausalModel(
    Y=observed_data_1.y.values, 
    D=observed_data_1.x.values, 
    X=observed_data_1.z.values)

cm.est_propensity_s()
cm.stratify_s()

print(cm.strata)

Stratification Summary

              Propensity Score         Sample Size     Ave. Propensity   Outcome
   Stratum      Min.      Max.  Controls   Treated  Controls   Treated  Raw-diff
--------------------------------------------------------------------------------
         1     0.146     0.180        56         8     0.160     0.165    -0.502
         2     0.180     0.231        44        18     0.202     0.208    -0.414
         3     0.231     0.444        87        38     0.318     0.336    -0.536
         4     0.444     0.604        31        32     0.514     0.515    -0.410
         5     0.607     0.644        10         6     0.626     0.626    -0.451
         6     0.647     0.701         3        12     0.651     0.678    -0.761
         7     0.706     0.774         7        24     0.732     0.735    -0.560
         8     0.775     0.884        10        52     0.846     0.833    -0.587
         9     0.888     0.955         4        58     0.908     0.930    -0.310

and the est_via_blocking method to combine the estimates of these strata into one overall ATE:

In [30]:

cm.est_via_blocking()
print(cm.estimates)

Treatment Effect Estimates: Blocking

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE     -0.511      0.034    -15.222      0.000     -0.577     -0.445
           ATC     -0.521      0.038    -13.612      0.000     -0.596     -0.446
           ATT     -0.501      0.040    -12.382      0.000     -0.580     -0.421

Which again works well.

Stratify the data into groups by propensity score is useful when we don't have any prior knowledge of what constitutes "similar" units, however it is not the only way. If you have prior knowledge the different groups of your samples are likely to be affected by the intervention in similar ways, it makes sense to split you samples into these groups before estimating the ATE, then pooling the results to get a global ATE.

Which Technique to Use?¶

I've now covered most the of the common techniques for causal inference from observational data. The remaining question is how to decide which method to use? This is not an easy question. While there are some automated techniques, like this paper, I haven't had a change to try them out.

Ultimately, to choose your technique you need to make some assumptions about how you contruct you counter factual. If you trust you data to have good over in covariate space, matching is a good approach because there is always some nearby point with the opposite treatment. When this isn't the case, you need to either use a model you trust to extrapolate well into unexplored areas or make the assumption that something like the propensity score captures enough information to assume ignorability.

To highlight that all these methods can fail, I have one more example. Unlike the previous examples, there is more than one covariate. Like all the previous datagenerators, this one also obeys the assumption

$Y_{1}, Y_{0} \ci X \, | \,Z$

By design.

Let's blindly try the methods we've discussed so far and see what happens

In [31]:

data_gen = dg.generate_exercise_dataset_2
ds = data_gen()

print("Observed ATE: {estimated_effect:.3f} ({standard_error:.3f})".format(**estimate_uplift(ds)))
print("Real ATE:  {estimated_effect:.3f} ({standard_error:.3f})".format(**run_ab_test(data_gen)))

zs = [c for c in ds.columns if c.startswith("z")]

cm = CausalModel(
    Y=ds.y.values, 
    D=ds.x.values, 
    X=ds[zs].values)

cm.est_via_ols()
cm.est_via_matching()
cm.est_propensity_s()
cm.est_via_weighting()

cm.stratify_s()
cm.est_via_blocking()

print(cm.estimates)

Observed ATE: -0.103 (0.695)
Real ATE:  4.629 (0.181)

Treatment Effect Estimates: OLS

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE     -0.221      0.343     -0.645      0.519     -0.894      0.451
           ATC      0.174      0.368      0.474      0.636     -0.547      0.896
           ATT     -0.407      0.350     -1.164      0.245     -1.094      0.279

Treatment Effect Estimates: Matching

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE      1.334      0.579      2.305      0.021      0.200      2.469
           ATC      1.067      0.642      1.662      0.096     -0.191      2.326
           ATT      1.460      0.680      2.147      0.032      0.127      2.793

Treatment Effect Estimates: Weighting

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE     -0.076      0.369     -0.207      0.836     -0.800      0.647

Treatment Effect Estimates: Blocking

                     Est.       S.e.          z      P>|z|      [95% Conf. int.]
--------------------------------------------------------------------------------
           ATE      0.126      0.468      0.270      0.787     -0.790      1.043
           ATC     -0.105      0.433     -0.242      0.809     -0.953      0.744
           ATT      0.235      0.563      0.417      0.676     -0.868      1.338

In [32]:

y = []
yerr = []
x_label = []

for method, result in dict(cm.estimates).items():
    y.append(result["ate"])
    yerr.append(result["ate_se"])
    x_label.append(method)
    
x = np.arange(len(y))

plt.errorbar(x=x, y=y, yerr=yerr, linestyle="none", capsize=5, marker="o")
plt.xticks(x, x_label)
plt.title("Estimated Effect Size", fontsize=18)
plt.hlines(4.6, -0.5, 3.5, linestyles="dashed")
plt.xlim(-0.5,3.5);

The horizontal dashed line shows the true ATE for this dataset.

Not only do we have a range of different results from each technique, they all miss the true value.

This should be a warning about the limitations of this kind of technique. It might be an intesteresting exercise for the reader to try and work about what properties of the dataset cause these methods to miss the true value.

The Structure of Causal Inference¶

Hopefully by this point, you will have realised the importance of the ignorability assumption

$Y_{1}, Y_{0} \ci X \, | \,Z$

What I haven't talked about is how we choose $Z$ so that this is true. Ultimately this needs to come from domain knowledge about the system being studied. There are a set of powerful tools called Causal Graphical Models which allow you to encode knowledge about the system being studied is a graphical model of the system and to reason about conditional independence assumptions like the one above.

I discuss these my next post on causal inference.

Another question this post might raise is whether the only way to make causal inferences is through adjusting for confounding variables. It isn't - in a later post I look at another technique you can use.

Code¶

You can find the notebook for this post on github here.

Reducing the Variance of A/B Tests Using Prior Information

2018-01-04T00:00:00+00:00

In [1]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns

%matplotlib inline

This post contains my notes on A/B testing and how to make it more powerful. They have been sitting around on my computer for a while now, and it has take a number of long train rides to finish them off. As a result they might seem a bit fragmented, but hopefully they make sense. Please let me know if you find any corrections or have any questions

Introduction¶

With the rise of the Internet, we are able to collect more data then ever, faster then ever, in a more controlled environment then ever before. Specifically, it allow us to show two different versions of a product to different customers simultaneously. This allow us to directly measure the impact of what these changes are, and has changed way products are designed. This idea of showing two versions of a product at the same time is known as A/B testing. This paper is a good overview of how websites use it: Practical Guide to Controlled Experiments on the Web.

The faster you can measure any impact of a change, the more useful these techniques are. In traditional A/B testing, this is usually achieved by increasing the number of people in the experiment, however there are a few variance reduction techniques that offer the ability to use additional information to increase the power of these experiments.

In this post I'm going to look at one of these techniques: control variates, and suggest how we can improve it using machine learning.

Classic A/B Testing¶

The goal of controlled experiments is often to estimate the Average Treatment Effect (ATE) of an intervention on some property, $y$, of a population. This is defined as

$\Delta = \hbox{E}[y|I] - \hbox{E}[y|\bar{I}]$

Where $I$ is an indicator for whether or not an individual received the intervention, and the expectation is averaged over the population.

$\Delta$ is the thing we fundamentally what to know - the effect of our treatment. However, because a single individual cannot both receive and not receive the intervention a the same time, we estimate this value by splitting a population into two groups at random, labeled $A$ and $B$, and applying our intervention only to group $B$.

Our estimate of the ATE is then:

$\hat{\Delta} = \bar{y}_{B} - \bar{y}_{A}$

Where $\bar{y}_{i}$ is the sample mean for group $i$. Splitting the samples into groups randomly ensures that no confounding variables influence our measurement.

Because we only have a finite number of samples, there will be some error in this estimate. One way to quantify this error is the confidence intervals around $\hat{\Delta}$. The interpretation of confidence intervals is

$P(\hbox{True value} \in \hbox{confidence interval}) = 1 - \alpha$.

Where $1 - \alpha$ is the coverage of the confidence interval, a value we choose to be between 0 and 1. Common values are 90% or 95%. The exact value you want will depend on how you are using your estimate. Note that here the true value is fixed, and it is the confidence interval which is a function of our data, and therefore a random variable.

To estimate the confidence intervals for our estimator, we use the central limits theorem: $\bar{y}_{i}$ is distributed normally with variance $\hbox{Var}(y_{i})/N_{i}$ in the limit of large $N_{i}$.

Under this approximation the standard deviation of $\hat{\Delta}$ is

$\sqrt{\hbox{Var}(\bar{y}_{B}) + \hbox{Var}(\bar{y}_{A})}$

Which we can estimate from our data using

$s_{\Delta} = z \sqrt{s_{B}^{2}/N_{B} + s_{A}^{2}/N_{A}}$

Where $s_{i}$ is the sample standard deviation of group $i$. And the confidence intervals are just

$se_{\Delta} = z(\alpha) \sqrt{\hbox{Var}(\bar{y}_{B}) + \hbox{Var}(\bar{y}_{A})}$

Where $z(\alpha) = \Phi^{-1}(1 - \frac{\alpha}{2})$ with $\Phi^{-1}$ as the inverse normal CDF.

We now have an estimate of the effect size, and our uncertainty of it. Often however, we need to make a decision with this information. How exactly we make this choice should depend on the cost/benefits of the decision, but it is sometimes enough just to ask whether or not our estimated value of $\Delta$ is "significantly" different from zero. This is usually done by using the language of hypothesis testing.

I don't want to go too much into the details of hypothesis testing, because I feel that confidence intervals are often a better way to present information, but since the language of A/B tests are often phrased as hypothesis tests, I should mention a few points:

Hypothesis testing for a difference in means between two populations answers the specific question "If we assume that the means of the two distributions are the same, is the probability I obtained my data smaller than $1 - \alpha$?". If the answer is yes, we say the results are "statistically significant".
Here $\alpha$ is the same $\alpha$ we used for confidence intervals above. It is a parameter we as experimenters choose (and we choose it before the experiment starts). The larger $1 - \alpha$ is, the more confident we are.
If we ran a lot of A/A tests (tests where there is no intervention), we would expect $\alpha$ of them to be "significant" ($\alpha$ is sometimes called the false positive rate, or type one error).
Once we have decided on a significance level, another question we can ask is: "if there was a real difference between the populations of $\Delta$, how often would we measure an effect?". This value is called the statistical power, and is often denoted as $1 - \beta$.

If we are to run tests, it is in our best interest to make them as powerful as possible. The power of a test is a function of the sample size, the population variance, our significance level and the effect size. Because of this, it is intimately related to the confidence intervals. It turns out that for an effect size equal to the size of the confidence intervals, the power of a z-test is 50%.

For the rest of these notes, I'm going to call an effect size "detectable" for a given experiment setup if it has power of at least 50%:

$\Delta_{detectable} \ge se_{\Delta}$

I have made this term up to simplify these notes. It is not common statistical jargon, and in reality you should always aim for tests more powerful then this. "Detectable effect size" is a function of only the variance of the underlying population we want to measure, and the sample sizes of each group.

Before continuing, I should note:

I've made a lot of approximations here. There will be circumstances where they are all violated.
One specific assumption is the normality of the estimator $\hat{\Delta}$ - for large sample sizes this is a good approximation, but for small samples there are more powerful tests you can run.
Hypothesis tests in one way to approach detecting differences between two groups. There are others.
Statistical significance should not be confused with the actual significance of a result.
When we make an intervention, we don't just change the expectation of a population, we may change the underlying distribution completely. Under our normal approximation, this means that the variance of our sample mean may change as well.

Generate Dataset¶

Having waded through some maths, it is good practice to check our results by simulation. In doing so, we can confirm that even using our ideal assumptions we have an idea what's going on.

To do this we need a mechanism we really generates data. The function below does this.

In [2]:

def generate_dataset(n_samples=100, uplift=0):
    x = np.random.uniform(low=0, high=1, size=n_samples*2)
    y = (
        + 10 * np.abs(x - 0.25)  
        + 2 * np.sin((x * 5) * 2 * np.pi)
        + np.random.normal(size=n_samples*2)
    )
    
    assignment = [1] * n_samples + [0] * n_samples
    np.random.shuffle(assignment)
    
    samples = pd.DataFrame({
        "x": x,
        "y": y,
        "group": assignment
    })
    
    samples.loc[lambda df: df.group == 1, "y"] += uplift
    
    return samples

In [3]:

ds = generate_dataset(n_samples=1000, uplift=1)
ds.head()

Out[3]:

	group	x	y
0	1	0.724798	4.357071
1	0	0.762068	3.006875
2	1	0.264475	0.813114
3	0	0.745901	2.559967
4	1	0.844981	9.910773

It generates variables from a distribution. The metric we care about is $y$, and the group assignment is given by the group column. There is also a covariate $x$, but for now we will ignore it.

This distribution of the metric is show below

In [4]:

sns.kdeplot(ds.y)

Out[4]:

<matplotlib.axes._subplots.AxesSubplot at 0x7fca3ae542e8>

Running a test¶

Now we have a method to generate data, we can apply an uplift to one group, and see if we can detect it. Our data generating function applies an uplift by just adding a constant value to all samples in group 1.

For 1000 samples in each group, we expect the confidence intervals in our effect size estimate to be

$z \sqrt{2 \frac{Var(y)}{N}}$

Where $z$ is a constant we set depending on the width of our confidence intervals. For 95% confidence intervals this is about 1.96.

The numeric value is

In [5]:

from scipy.stats import norm

sample_size = 1000
alpha = 0.05
z = norm.isf(alpha / 2)
estimated_variance = ds.y.var()

detectable_effect_size = z * np.sqrt(2 * estimated_variance / sample_size)
detectable_effect_size

Out[5]:

0.23309489059284458

So, for an uplift of 0.24, we would expect about 50% of our experiment to have a significant uplift.

Let's take a look

In [6]:

from scipy.stats import ttest_ind

n_experiments = 20
uplift = 0.24
sample_size = 1000
alpha = 0.05
z = norm.isf(alpha / 2)

results = []
is_significant = []

# run some experiments
for _ in range(n_experiments):
    ds = generate_dataset(n_samples=sample_size,uplift=uplift)
    base = ds[ds.group == 0]
    variant = ds[ds.group == 1]
    
    # estimated effect size is just the difference in means
    estimated_effect_size = variant.y.mean() - base.y.mean()
    
    # estiamated error is the combined variance of each group
    estimated_effect_err = 2 * np.sqrt(
        variant.y.var()/variant.shape[0] + 
        base.y.var() / base.shape[0]
    )
    results.append((estimated_effect_size, estimated_effect_err))
    
    # t-test to check if significant
    t,p = ttest_ind(base.y.values, variant.y.values)
    is_significant.append(p <= 0.05)
    
    
# plot everything
y = np.arange(n_experiments) + 0.5
x = np.asarray([i[0] for i in results])
xerr = np.asarray([i[1] for i in results])
is_significant = np.array(is_significant)

plt.figure(figsize=(8,8))
plt.vlines(0, 0, len(results), linestyles="dashed")
plt.vlines(uplift, 0, len(results))
plt.ylim(0,len(results))

mask = is_significant
plt.errorbar(x[mask],y[mask],xerr=xerr[mask], ls="none", c="g", marker="o")
mask = ~mask
plt.errorbar(x[mask],y[mask],xerr=xerr[mask], ls="none", c="r", marker="o");

print("Points with significant effect: {}/{}".format(np.sum(mask), n_experiments))
print("Points whose confidence interval covers the true effect size: {}/{}".format(
    np.sum(np.abs(x-uplift) > xerr), n_experiments))

Points with significant effect: 10/20
Points whose confidence interval covers the true effect size: 1/20

This image shows the results of 20 simulated experiments for datasets of 1000 samples in each group, and an uplift of 0.24. The points show the estimates of $\Delta$ and the lines show the confidence intervals around the points. Points in green pass a significance test (so we can "reject the null hypothesis", and say that there is a real difference), points in red do not. The solid line is the true effect size, the dashed line is zero.

As expect by our design, about half the points are measured to have a detectable effect (because the power is about 50%), and only one point has confidence intervals which do not cover the true effect size (because our significance level is 95%).

Improving the Power of our experiment¶

Now that we understand the system we are dealing with, we can ask the question: how can we increase the detectable effect size of our experiments? We are left with a few options:

Increase the effect size
Increase the sample size
Decrease the variance

Increasing the effect size may or may not be possible depending on the effect we are investigating.

Increasing the sample size is often the easiest way to improve the power of a test, however because the detectable effect size scales as $1/\sqrt{N}$, it becomes harder and harder to increase the power of an experiment this way. In reality the sample size is often constrained by cost or time.

This leaves the option of reducing the variance.

Variance Reduction¶

Up until this point, we have assumed that the our samples were drawn IID from some distribution and nothing else was known about them. In reality, this is often not the case: we might have observed our metric of interest for each subject in some period before the experiment started or we might know some characteristic about the subject. Provided this informations is unaffected by the intervention made during our experiment, it provides us with the ability to reduce the variance.

There are a number of approaches to include this information, but the clearest idea I've seen is the method of control variates.

Control Variates¶

I came across the idea of using control variates in the paper Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data, and it states that if we have a variable which is correlated with our metric of interest, we can improve the variance by simply subtracting a weighted value of it.

Mathematically, if we have a variable $x$ that we want to use as a control variate, and our variable of interest $y$, then we can define a new variable $\tilde{y}$ so that

$\tilde{y} = y - \theta x$

Where $\theta$ is a constant.

The expectation of this new variable is

$E[\tilde{y}] = E[y] - \theta E[x]$

If we choose $x$ such that it's expectation is 0 (for example, by subtracting the mean) then the expectation of $\tilde{y}$ is just the expectation of $y$. The variance of $\tilde{y}$ is given by:

$\hbox{Var}(\tilde{y}) = \hbox{Var}(y) + \theta^{2}\hbox{Var}(x) - 2 \theta \hbox{Cov}(y,x)$

Minimising the variance with respect to $\theta$, we get

$\theta = \frac{\hbox{Cov}(y,x)}{\hbox{Var}(x)}$

Plugging this back into our original expression, we get

$\hbox{Var}(\tilde{y}) = \hbox{Var}(y)\left(1 - \rho^{2}\right)$

Where

$\rho^{2} = \frac{\hbox{Cov}(y,x)^{2}}{\hbox{Var}(y)\hbox{Var}(x)}$

which is just the correlation between our metric and the control variate. This means that if there's some correlation between the our control variate and our metric, we can reduce the variance.

Control Variates in A/B tests¶

This all looks good, but it relies on us knowing the mean of $x$. In A/B tests, we no longer need to know this. If $x$ is not affected by the intervention, then the mean's of $x$ for each group will be identical, and will canceled out. We can define a new estimator:

$\tilde{\Delta} = \frac{1}{N_{B}}\sum_{i} (y_{B,i} - \theta x_{B,i}) - \frac{1}{N_{A}}\sum_{i} (y_{A,i} - \theta x_{A,i})$

$ = (\hat{y}_{B} - \hat{y}_{A}) - \theta (\bar{x}_{B} - \bar{x}_{A})$

$ = \hat{\Delta}$

Which has the same expectation value as our original estimator, but with variance reduced by a factor of $\left(1 - \rho^{2}\right)$.

Let's see how it performs in simulations.

Control Variate Simulations¶

When we created our initial dataset, I didn't talk about the column $x$. Let's see how $y$ varies with $x$

In [7]:

ds = generate_dataset(n_samples=200)
sns.lmplot(data=ds, x="x", y="y", hue="group", fit_reg=False)

Out[7]:

<seaborn.axisgrid.FacetGrid at 0x7f3b0f1c4ef0>

There is definitely a lot of structure there - $y$ appears to be some non-linear function of $x$. The exact structure isn't important, what's important to us is that it is correlated with $x$:

In [8]:

correlation = ds.corr().loc["x","y"]
correlation

Out[8]:

0.6768194939711959

Before jumping into it, let's define some helper functions. The just compare different estimators by evaluating them on many realizations of the dataset.

In [9]:

def run_many_times(dataset_generator, estimator, n_runs):
    return np.array([estimator(dataset_generator()) for _ in range(n_runs)])

def compare_estimators(dataset_generator, estimators, true_uplift, n_runs=500):
    results = {}
    
    for estimator_name, estimator in estimators.items():
        res = run_many_times(dataset_generator, estimator, n_runs)
        sns.kdeplot(res, shade=True, label=estimator_name)
        
        print("Estimator: {}".format(estimator_name))
        print("\tConfidence Interval Width: {:.3f}".format(2 * np.std(res)))
        print("\tBias: {:.3f}".format(np.mean(res - true_uplift)))
        results[estimator_name] = res
        
    ymin, ymax = plt.ylim()
    plt.vlines(uplift, 0, ymax)
    plt.vlines(0, 0, ymax, linestyles="dashed")
    plt.ylim(0, ymax)
    
    return results

Let's start by looking at our first estimator: the difference in means between each group.

In [10]:

def base_estimator(ds):
    a = ds[ds.group == 0]
    b = ds[ds.group == 1]
    return b.y.mean() - a.y.mean()

And checking how it is distributed

In [11]:

uplift = 0.5
n_samples = 1000
generator = lambda : generate_dataset(uplift=uplift, n_samples=n_samples)

results = compare_estimators(
    dataset_generator=generator,
    estimators={
        "base": base_estimator,
    },
    true_uplift=uplift
)

Estimator: base
	Confidence Interval Width: 0.239
	Bias: -0.002

As expected, it is centered around the uplift, with the expected standard deviation.

Now let's define our control variate estimator and see how it compares:

In [12]:

def cv_estimator(ds):
    theta = ds.cov().loc["x", "y"] / ds.cov().loc["x", "x"]
    
    ds_copy = ds.copy()
    ds_copy.y = ds.y - theta * ds.x
    
    return base_estimator(ds_copy)

In [13]:

uplift = 0.5
n_samples = 1000
generator = lambda : generate_dataset(uplift=uplift, n_samples=n_samples)

results = compare_estimators(
    dataset_generator=generator,
    estimators={
        "base": base_estimator,
        "control variate estimator": cv_estimator
    },
    true_uplift=uplift
)

Estimator: base
	Confidence Interval Width: 0.230
	Bias: 0.004
Estimator: control variate estimator
	Confidence Interval Width: 0.188
	Bias: 0.003

We see a reduction in the standard deviation of the estimator.

Using the calculated correlation, we can check whether it agrees with the reduction we expect:

In [14]:

np.sqrt((1 - correlation ** 2) * estimated_variance)

Out[14]:

2.0255125675111496

Looks good. Without changing anything about our experiment, we have increased it's power. Specifically:

We've reduced our detectable effect size by 21%
To get the same decrease by collecting more samples, this would take about 60% more samples

An Alternative View¶

Having reassured ourselves that using control variates can increase the power, it's worth thinking again about what we are doing. The form of how we calculate the control variate weighting parameter $\theta$ might seem familiar: this is because it appears in the same form in the coefficient for ordinary least-square regression.

This provides an interesting interpretation of what we are doing: fitting a model to $y$ given $x$, and then estimating the effect size from the residuals of this fit. Because which of our subjects gets a treatment is randomized and not part of our model, from the point of view of our model it is "noise".

Given the powerful function approximation techniques available in machine learning, it is tempting to ask whether we can improve things further.

Let's see what happens when we try the same trick, using random forests. To do this, I will use sklearn's default implementation.

In [15]:

from sklearn.ensemble import RandomForestRegressor

def rf_estimator(ds):
    X = ds.x.values.reshape(-1,1)
    y = ds.y.values
    reg = RandomForestRegressor()
    reg.fit(X,y)
    
    ds_copy = ds.copy()
    ds_copy.y = ds.y - reg.predict(X)
    
    return base_estimator(ds_copy)

In [16]:

uplift = 0.5
n_samples = 1000
generator = lambda : generate_dataset(uplift=uplift, n_samples=n_samples)

results = compare_estimators(
    dataset_generator=generator,
    estimators={
        "base": base_estimator,
        "control variate estimator": cv_estimator,
        "random forest estimator": rf_estimator
    },
    true_uplift=uplift
)

Estimator: base
	Confidence Interval Width: 0.225
	Bias: 0.001
Estimator: control variate estimator
	Confidence Interval Width: 0.186
	Bias: -0.003
Estimator: random forest estimator
	Confidence Interval Width: 0.043
	Bias: -0.316

The confidence intervals are smaller, but the estimate is nowhere near the true effect. What's happened?

Overfitting and Bias¶

In machine learning language, we have "overfitted" the data. Our estimator has learned not just the underlying signal in our data, but also the noise. If all we cared about is the prediction accuracy of our estimator, this might not be a bad thing - the is a tradeoff between variance and bias when fitting a model to data, and sometimes it is worth trading a bit of bias for a better overall prediction.

In our situation, things are different. Because we are ultimately using our model to estimate an effect size, any bias will degrade the performance on our final result, leading to a biased estimate.

This is not something specific to the use of random forests. We can see the same behavior in our control variant estimator. The reason we didn't see it before is that in our control variate estimator, we have only one free parameter to fit: the covariance between $x$ and $y$, which is much smaller than the number of datapoints we were using to fit it. If we fit to only a small number of datapoints, we start to see the same thing occurring:

In [17]:

uplift = 10
n_samples = 2
generator = lambda : generate_dataset(uplift=uplift, n_samples=n_samples)

results = compare_estimators(
    dataset_generator=generator,
    estimators={
        "base": base_estimator,
        "control_variate": cv_estimator,
    },
    true_uplift=uplift
)

Estimator: base
	Confidence Interval Width: 5.173
	Bias: 0.236
Estimator: control_variate
	Confidence Interval Width: 7.252
	Bias: -3.360

So, if bias is an intrinsic part of this estimators, is there any way we can avoid it and still get that sweet sweet variance reduction without compromising our results?

One approach that I've found is to use the idea of stacking.

We start by splitting our data into $K$ disjoint groups. For each of these groups, $i$, we fit an estimator on all the data not in group $i$, and then use this estimator to make a prediction for group $i$. The idea is that even if each of our estimators overfits, because it is not predicting on the data it was trained on, the overfit translates to increased variance.

This might make more sense with an implementation. Let's see how it does

In [18]:

from sklearn.model_selection import KFold

def stacked_estimator(ds):
    kfold = KFold(n_splits=10)

    X = ds.x.values.reshape(-1,1)
    y = ds.y.values
    
    reg = RandomForestRegressor()
    
    ds_copy = ds.copy()
    ds_copy["y_pred"] = 0

    for train_index, test_index in kfold.split(X):
        
        X_train, X_test = X[train_index], X[test_index]      
        y_train, y_test = y[train_index], y[test_index]
        
        reg.fit(X_train,y_train)
        ds_copy.loc[test_index, "y_pred"] = reg.predict(X[test_index])
    
    ds_copy.y = ds_copy.y - ds_copy["y_pred"]
    
    return base_estimator(ds_copy)

In [19]:

uplift = 0.5
n_samples = 1000
generator = lambda : generate_dataset(uplift=uplift, n_samples=n_samples)

results = compare_estimators(
    dataset_generator=generator,
    estimators={
        "base": base_estimator,
        "control variate estimator": cv_estimator,
        "stacked random forest estimator": stacked_estimator
    },
    true_uplift=uplift
)

Estimator: base
	Confidence Interval Width: 0.237
	Bias: 0.005
Estimator: control variate estimator
	Confidence Interval Width: 0.181
	Bias: -0.002
Estimator: stacked random forest estimator
	Confidence Interval Width: 0.107
	Bias: 0.003

It looks good. We've reduced the confidence intervals by another 40% over using linear control variates.

This means compared to our base estimator, we have

We've reduced the size of the confidence intervals by 55%
To get the same increase by increasing the sample size, we would need 4.8 times as many samples

It is worth keeping in mind that when we use models like this, there is some chance that they can be very bad. When the models are bad, we have the risk of increasing the variance of our estimator. For this reason I think that it is useful to add a layer of standard control variate correction on top of the stacked estimator. If our predictions do turn out to be bad, the lack of correlation between the stacked predictions should mean that the linear control variates do not alter the results. This will introduce some small bias, but given a large dataset, this bias should be small.

I implement this below

In [20]:

from sklearn.model_selection import KFold

def stacked_cv_estimator(ds):
    kfold = KFold(n_splits=10)

    X = ds.x.values.reshape(-1,1)
    y = ds.y.values
    
    reg = RandomForestRegressor()
    
    ds_copy = ds.copy()
    ds_copy["y_pred"] = 0

    for train_index, test_index in kfold.split(X):
        
        X_train, X_test = X[train_index], X[test_index]      
        y_train, y_test = y[train_index], y[test_index]
        
        reg.fit(X_train,y_train)
        ds_copy.loc[test_index, "y_pred"] = reg.predict(X[test_index])
    
    ds_copy.x = ds_copy["y_pred"]
    
    return cv_estimator(ds_copy)

In [21]:

uplift = 0.5
n_samples = 1000
generator = lambda : generate_dataset(uplift=uplift, n_samples=n_samples)

results = compare_estimators(
    dataset_generator=generator,
    estimators={
        "base": base_estimator,
        "control variate estimator": cv_estimator,
        "stacked random forest estimator": stacked_estimator,
        "stacked random forest CV estimator": stacked_cv_estimator,
    },
    true_uplift=uplift
)

Estimator: base
	Confidence Interval Width: 0.240
	Bias: -0.006
Estimator: control variate estimator
	Confidence Interval Width: 0.180
	Bias: -0.002
Estimator: stacked random forest estimator
	Confidence Interval Width: 0.110
	Bias: 0.001
Estimator: stacked random forest CV estimator
	Confidence Interval Width: 0.112
	Bias: 0.001

In our case, it doesn't change things, but there are situations where it might.

It is also worth pointing out that we are getting close to the fundamental limit of how good an estimator could be. In the function which generates the data, I have included a noise term which add a sample from the unit normal distribution to each datapoint. Because of this, the smallest we can make the size of our confidence intervals is

In [22]:

z * np.sqrt(2 / 1000)

Out[22]:

0.087652254057658169

Control Variates is not the only methodology to improve the efficiency of randomized control trials. Another Common approach is know as covariate adjustment. In it's simplest form, it involves fitting a linear model to the experiment data:

$y = \alpha + \beta x + \beta_{z} z$

Where $z$ is an indicator variable, with $z = 0$ when the sample is in the base group and $z=1$ when the sample is in the variant group and $\alpha$, $\beta$ and $\beta_{z}$ are the parameters to be fitted.

The fitted parameter for $\beta_{z}$ is then our estimator for the effect size. There are a number of ways to produce this fit, but most seem to involve minimising the residuals as in OLS. This paper has a good overview of the asymptotic properties of covariate adjustment, and suggest that his approach is almost identical to linear control variate approaches.

The Bayesian Elephant in the Room¶

The goal of this post has been to outline some ideas about how we can use prior information about subjects in a random controlled trial to improve our estimate of the effect of the intervention. The standard way to take into account prior information in models is by using Bayesian techniques: there is a good overview of the basic ideas in these posts.

If we stick with linear models, we could frame the problem in a bayesian way, but it feels like this approach is very similar to covariate adjustment. It's not clear if you could extend it to arbitrary models, however if someone has an idea, I'd very interested to hear about it.

Code¶

You can find the notebook for this post on github here.

Generating Examples of Simpson's Paradox

2017-10-22T00:00:00+01:00

Simpson' Paradox is a example of one of the counter intuitive properties of probability distributions. It occurs when an observed relationship between two variables is reversed when you take into account another variable.

The paradox itself has been described very well elsewhere, so I'm not going to describe it in too much detail here. Instead I'm going to try and answer another question: how do we write a program to generate examples of Simpson's paradox?

Let's start with an example of the paradox:

Imagine we are trying to work out whether a certain drug is an effective treatment for a disease. To decide whether it is effective, we compare people who took the drug (call this $x$, if someone took the drug, $x=1$, if not $x=0$), by examining how many of each recovered from the disease (call this $y$ - $y=1$ mean they got better, $y=0$ mean they did not).

When we just look at these two numbers, we find that of 250 people who took the drug, 110 recovered (44%), whereas out of the 250 people who did not take the drug 177 recovered (71%). From these results it looks like there is a clear advantage to not taking the drug. Unfortunately, this drug was not administered as part of a random controlled trial. This means that the decision of whether or not to take the drug may have been confounded.

In fact, if we look at the recovery rate for different age groups (denoted by variable $z$), we find that for each age group more of those who took the drug recovered than those who did not, as shown in the table below

In [1]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline

import utils as u
from IPython.core.display import HTML

HTML(u.create_table(pd.read_csv("binary_example.csv")))

Out[1]:

z	X=0		X=1
0	1/22	0.05	18/109	0.17
1	1/5	0.20	13/36	0.36
2	2/4	0.50	3/5	0.60
3	61/91	0.67	51/72	0.71
4	112/128	0.88	25/28	0.89
total:	177/250	0.71	110/250	0.44

When we look at different age groups, our conclusion about the drug effectiveness is reversed.

What's worse, is that it is possible that if we take into account another variable the correlation might be reversed again. This is why random controlled trials are so important for separating correlation and causation.

How can we generate examples of the paradox? let's start by defining it precisely.

Binary Case¶

$$ \newcommand{P}[1]{\mathrm{P}\left( #1 \right)} \newcommand{Pc}[2]{\mathrm{P}\left( #1 \mid #2 \right)} $$

Consider the situation where we have three random variables, two binary: $x$, $y$ and one discrete, $z$. Simpson's paradox is observed if when considering just the $x$ and $y$ variables, there appears to be a negative correlation between $x=1$ and $y = 1$.

Specifically

$\Pc{Y=1}{X=1} < \Pc{Y=1}{X=0}$

but when we look at each subgroup, the relationship is the other way round:

$\Pc{Y=1}{X=1, Z=i} > \Pc{Y=1}{X=0, Z=i}$

For all $i$.

To find a distribution which obeys these inequalities, let's start by parameterising it by:

$\Pc{Y=1}{X, Z} = p_{X,Z}$

And

$\Pc{Z}{X} = q_{X,Z}$

The marginal distribution can be written as

$\Pc{Y=1}{X=j} = \sum_{i\in Z} \Pc{Y=1}{X=j, Z=i} \Pc{Z=i}{X=j} = \sum_{i\in Z} p_{j,i}q_{j,i}$

So we are looking for $p$ and $q$ which obey

$p_{1,i} > p_{0,i}$

$\sum_{i\in Z} p_{1,i}q_{1,i} < \sum_{i\in Z} p_{0,i}q_{0,i}$

$\sum_{i\in Z} q_{j,i} = 1$

I will also choose the labels so that:

$p_{j,i} < p_{j,k}$

if $i < k$.

Before we look for a solution, let's write a function which takes descriptions $p$ and $q$ and checks whether they obey this inequality:

In [2]:

def assert_paradox_binary_distribution(p, q):
    """
    Asserts whether a distribution described by p,q demonstrates 
    Simpson's paradox. 
    
    :param p: dictionary, p[x][z] = p(Y=1|X,Z)
    :param q: dictionary, q[x][z] = p(Z|X)
    :return: None
    """
    n_samples = len(p[0])
    for i in range(n_samples):
        assert(p[0][i] < p[1][i])
            
    assert(all(0 < qq < 1 for qq in q[1].values()))
    assert(all(0 < qq < 1 for qq in q[0].values()))
    
    b0 = sum(p[0][i] * q[0][i] for i in range(n_samples))
    b1 = sum(p[1][i] * q[1][i] for i in range(n_samples))
    
    assert(b0 > b1)

To generate a solution, note that the terms $\sum_{i\in Z} p_{j,i}q_{j,i}$ generate a convex combination of the $p$ values. Given a free choice of $q$, we can make this term lie anywhere between the largest and smallest $p_{j,i}$ value. To meet the second inequality, we require that

$p_{1,0} < p_{0,n}$

Where $n$ is the largest value $Z$ can take.

We now have the ingredients to a solution. We can put them together in the following way:

Choose $p$ values so that $p_{1,i} > p_{0,i}$ for all $i$ and $p_{1,0} < p_{0,n}$
Choose two values in the region $(p_{1,0}, p_{0,n})$, $b_{1}$ and $b_{0}$ so that $b_{1}$ < $b_{0}$
Find $q$'s so that $\sum_{i\in Z} p_{j,i}q_{j,i} = b_{j}$

Programmatically, we can carry out the the first step with the following function

In [3]:

from collections import defaultdict

def generate_p_values(n_subgroups):
    """
    Generates a set of conditional probabilities that obey
    
     - p(Y=1|x=1, z) > p(Y=1|x=0, z) for all z
     - p(Y=1|x=1, z=j) > p(Y=1|x=1, k) when j > k
     - p(Y=1|x=1, z=0) < p(Y=1|x=0, z=n) where n = max(z)
     
    :param n_subgroups: int. The number of values $Z$ can take.
    :return: dictionary, p[x][z] = p(Y=1|x,z)
    """
    p = defaultdict(dict)

    boundaries = np.random.uniform(0, 1, size=2 * n_subgroups)
    boundaries = [(n + b) / (2*n_subgroups) for n, b in enumerate(boundaries)]

    for i in range(n_subgroups):
        p[0][i] = boundaries[i*2]
        p[1][i] = boundaries[i*2+1]

    return p

Which generates a solution to the inequalities by generating a set of ordered random numbers between (0,1) and assigning them to $p_{i,j}$ ordered by ($i$, $j$). The method we use to generate the random numbers ensure they are spread across the whole range.

We now need a function which for a given set of $p$'s, and a target values, find the weights which obeys

$\sum_{i} p_{i}q_{i} = target$

$\sum_{i} q_{i} = 1$

When there are only two subgroups, this has a single solution. When there are more, there are a range of solutions. We choose one by finding pairs of solutions recursively. The following function does this

In [4]:

def get_q_weights(ps, target):
    """
    Generates a mixture of the values in ps which is the solution to
    
    \sum_{i} p[i]q[i] = target
    
    :param ps: list of number
    :param target: goal of the sum
    :return: qs: list of weightings of ps
    """
    if len(ps) <= 1:
        raise ValueError("ps cannot be shorter than 2")

    if len(ps) == 2:
        p0, p1 = ps
        q0 = (p1 - target) / (p1 - p0)
        return q0, (1 - q0)

    rest, last = ps[:-1], ps[-1]
    mid_target = np.random.uniform(low=rest[0], high=min(target, rest[-1]))

    q0, q1 = get_q_weights([mid_target, last], target)
    remaining_qs = get_q_weights(rest, mid_target)
    qs = [q0 * q for q in remaining_qs] + [q1]

    return qs

With these two functions, we can generate the compete distribution and check that it meets our original requirements

In [5]:

def generate_paradox_binary_distribution(n_subgroups=3):
    """
    Generates a distribution which demonstrates Simpson's paradox 
    
    q[x][z] = p(Z|X)
    p[x][z] = p(Y=1|X,Z)
    
    :param n_subgroups: int
    :return: p, q: dicts
    """

    p = generate_p_values(n_subgroups)
    
    p_low, p_high= p[1][0], p[0][n_subgroups-1]

    b1, b0 = np.random.uniform(size=2)
    b1, b0 = p_low+((1+b1)/5)*(p_high-p_low), p_low+((3+b0)/5)*(p_high-p_low) 
    
    
    q = {}
    q[0] = {k: v for k, v in enumerate(get_q_weights(sorted(p[0].values()), b0))}
    q[1] = {k: v for k, v in enumerate(get_q_weights(sorted(p[1].values()), b1))}

    return p, q

p, q = generate_paradox_binary_distribution(4)

assert_paradox_binary_distribution(p, q)

We have generated a distribution which violates Simpson's paradox. Now let's draw some samples from it.

It is worth noting that up until this point we haven't talked about how the $X$ values are distributed. It is because they don't matter for the current discussion. As long as there is some variability in the outcome, we can demonstrate the paradox. However, now that we want to actually generate counts we do need to specify it.

In [6]:

def realise_binary_paradox(p, q, x, n_approx_samples=100):
    """
    Realises the mean outcome from the provided distribution
    :param p: dict, p[x][z] = p(Y=1|X,Z)
    :param q: dict, q[x][z] = p(Z|X)
    :param x: dict, x[i] = p(x)
    :param n_approx_samples: int
    :return: Dataframe of outcomes
    """
    records = []
    for xx in x.keys():
        for zz in q[xx].keys():
            p_y_eq_0 = x[xx] * q[xx][zz] * (1 - p[xx][zz])
            p_y_eq_1 = x[xx] * q[xx][zz] * p[xx][zz]
            records.append({"x": xx, "z": zz, "y": 1, 
                            "count": round(n_approx_samples * p_y_eq_1)})
            records.append({"x": xx, "z": zz, "y": 0, 
                            "count": round(n_approx_samples * p_y_eq_0)})
    return pd.DataFrame.from_records(records)

Because this function involves rounding the number of samples to integers, it is possible for some groups to end up with equal ratios, however on average it does produce the required paradox.

Let's take a look at the outcome

In [14]:

p, q = generate_paradox_binary_distribution(n_subgroups=3)
x = {0:0.3, 1:0.7}

df = realise_binary_paradox(p, q, x, n_approx_samples=500)
HTML(u.create_table(df))

Out[14]:

z	X=0		X=1
0	5/33	0.15	53/200	0.27
1	4/10	0.40	76/118	0.64
2	82/106	0.77	31/33	0.94
total:	91/149	0.61	160/351	0.46

Continuous Example¶

It is possible to define Simpson's paradox for continuous variables. A simple example would be look at the Covariance between two variables, $x$ and $y$, which is one sign for a collection of subgroups, but a different sign for the whole population.

If we consider the variables $x$ and $y$ as being drawn from a mixture distribution then by the law of total covariance, the covariance of the total population is given by two components:

a weighted sum of the subpopulation covariances
the covariance of the subpopulation means

If the sign of the covariance for the subpopulations are all in one direction, the first term will be in that direction. If the overall population is going to be the opposite sign, it needs to be compensated by the second term.

Using multivariate normal distributions, this is straightforward to simulate:

In [8]:

def generate_gaussian_simpsons_paradox(n_subgroups=3, n_samples=1000):

    overall_cov = 3*np.array([[1,0.9], [0.9,1]])

    means = np.random.multivariate_normal(mean=[0,0], cov=overall_cov, size=n_subgroups)
    
    weights = np.random.uniform(size=n_subgroups)
    weights /= np.sum(weights)
    covs = [np.random.uniform(0.2,0.8) for _ in range(n_subgroups)]
    covs = [np.array([[1,-c], [-c,1]]) for c in covs]


    samples = []

    for sg, (mean, cov, w) in enumerate(zip(means, covs, weights)):
        n = int(round(n_samples * w))
        sample = np.random.multivariate_normal(mean=mean, cov=cov, size=n)
        sample = pd.DataFrame(sample, columns=["x", "y"])
        sample["z"] = sg
        samples.append(sample)
        
    df = pd.concat(samples)
    
    return df

Because we are generating the subgroup means and covariances randomly, it is possible to end up with a sample which does not demonstrate the paradox, but on average it will.

In [9]:

df = generate_gaussian_simpsons_paradox()

print("Total Covariance: {:.3f}".format(df[["x", "y"]].cov().iloc[0,1]))
for z in df.z.unique():
    print("Subgroup {} covariance: {:.3f}".format(z,df[df.z==z][["x", "y"]].cov().iloc[0,1]))

Total Covariance: 1.629
Subgroup 0 covariance: -0.414
Subgroup 1 covariance: -0.633
Subgroup 2 covariance: -0.647

This is clearer if we plot what's going on.

For the overall population, there is a positive correlation between $x$ and $y$:

In [10]:

sns.regplot(data=df, x="x", y="y");

But for each subpopulation, the correlation is negative

In [11]:

fig, ax = plt.subplots()

plt.xlim(-6,6);
plt.ylim(-6,6);

for z in df.z.unique():
    sns.regplot(data=df[df.z==z], x="x", y="y", ax=ax)

Code¶

You can find the notebook for this post on github here.

Map Projections

2017-09-30T00:00:00+01:00

The animation above was built to demonstrate how different the map would look if you changed the point used as (0,0) for longitude, latitude. Click anywhere on the map to rotate to the view given by using that point as the origin, or press "Start Roaming" to rotate between random points.

When we create 2d maps of the globe, we have to decide how to take points on a sphere and embed them in a plane. How we do this is know as a map projection, and no matter which one we chose, something about the representation is always distorted (Relevant XKCD). The goal of this animation is to explore how these distortions look as we change the origin of the map.

The whole thing was built using d3, and is only a slight modification Mike Bostock's World Tour animation. You can find the full code used on github here.

Making A Prophet

2017-07-24T00:00:00+01:00

I have recently been using facebook's timeseries prediction package Prophet. Prophet add some much needed timeseries prediction power to python. It is simpler to use that statsmodels, the other major package in python for timeseries predictions, but this simplicity comes at a cost of obscuring what's going inside prophet. This post is my notes on understanding how Prophet works, and comes fom my own reading of the Prophet source code and accompanying paper.

If you've read the documentation around the package, there isn't much new here, but hopefully it will be of use to someone.

Introduction¶

Let's start by making a prediction with Prophet. We'll do this using daily data from London's cycle hire scheme. We begin by loading this data

In [25]:

# load some libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

%matplotlib inline

# load and clean the data
tfl = (
    pd.read_excel("tfl-daily-cycle-hires.xls", sheetname="Data")
    .rename(columns={"Day":"ds", "Number of Bicycle Hires": "y"})
    [["ds", "y"]]
    .copy()
)

tfl.ds = pd.to_datetime(tfl.ds)

tfl.head()

Out[25]:

	ds	y
0	2010-07-30	6897
1	2010-07-31	5564
2	2010-08-01	4303
3	2010-08-02	6642
4	2010-08-03	7966

In [2]:

tfl.set_index("ds").plot(figsize=(10,4))

Out[2]:

<matplotlib.axes._subplots.AxesSubplot at 0x7f693a0a97f0>

It is more or less what we would expect: yearly trends, weekly trends and a slow upwards growth, with lots of outliers along the way.

We can now import the package:

In [3]:

import fbprophet

And, following the instructions in the quickstart, we can immediately fit a model to our data:

In [4]:

model = fbprophet.Prophet()
model.fit(tfl);

Prediction is also straight forward:

In [21]:

future = model.make_future_dataframe(periods=365)
forecast = model.predict(future)
model.plot(forecast);

As well as predictions, we can quickly identify potential outliers:

In [23]:

# merge real points with forcast
real_and_forecast = pd.merge(left=forecast, right=tfl, on="ds")

# get the difference between prediction and forcast
real_and_forecast["residual"] = real_and_forecast.y - real_and_forecast.yhat

# get the range between 80% confidence intervals
real_and_forecast["uncertainty"] = real_and_forecast.yhat_upper - real_and_forecast.yhat_lower

# define an outlier as more than two intervals away from the forcast
v = 2
(
    real_and_forecast
    [real_and_forecast.residual.abs() > v * real_and_forecast.uncertainty]
    [["ds", "residual"]]
)

Out[23]:

	ds	residual
1805	2015-07-09	35940.798939
1833	2015-08-06	26185.523156

Some quick googling suggests that the two points with an unusually large number of bike rentals happed on two days when there were tube strikes on: see here and here. This suggests that potentially 35,000 extra people rented bikes as a result of a strike. It also gives us some idea of data we might want to include to make the predictions better.

And that is Prophet.

Let's see what's going on under the hood.

How it works¶

At it's heart, Prophet is fitting the following model to the data:

$y(t) = g(t) + s(t) + h(t) + \epsilon_{t}$

Where

$y(t)$ is the variable we are predicting
$g(t)$ is "growth/trend" - a slowly varying function
$s(t)$ is "seasonality" - periodic functions to take into account monthly/weekly patterns
$h(t)$ is "holiday" - takes into account the impact of "special" days that may not follow regular patterns
$\epsilon_t$ is the residuals, which are assumed to be i.i.d. and normally distributed

Fitting to this function is carried out by STAN, a powerful statistical modeling package. A lot of my interest in understanding the insides of Prophet came from want to see STAN used in action.

Components¶

We can directly look at the components which make up the forecast using the plot_components method

In [7]:

model.plot_components(forecast);

Setting up the dataframe¶

Before fitting the various components, the dataframe is prepared. This is done by the setup_dataframe method, which converts the dates into a numbers between 0 and 1 and the $y$ values into scaled values between 0 and 1.

In [8]:

prepared = model.setup_dataframe(tfl)
prepared.head()

Out[8]:

	ds	y	t	y_scaled
0	2010-07-30	6897	0.000000	0.094358
1	2010-07-31	5564	0.000406	0.076121
2	2010-08-01	4303	0.000811	0.058869
3	2010-08-02	6642	0.001217	0.090869
4	2010-08-03	7966	0.001622	0.108983

In [9]:

prepared.y_scaled.describe()

Out[9]:

count    2467.000000
mean        0.333190
std         0.122545
min         0.037814
25%         0.248810
50%         0.327824
75%         0.418078
max         1.000000
Name: y_scaled, dtype: float64

Trend¶

The growth/trend function $g(t)$ can either be modeled as linear or logistic function. In what follows, I'm going to explain the linear how the linear growth is formed. The logistic growth follows almost the same pattern.

When we initialised the model, the type of growth function defaulted to linear. However, if you look at the trend component of the forecast above, you notice that the trend is not just one linear component, but a sequence of piecewise linear components chained together. If prior information about when these changepoints might occur is available, the user can specify them. If not, prophet by default assumes that there are 25 evenly spaced changepoints spread over the first 80% of the data, and tried to choose their size optimally.

We can visualise them directly:

In [26]:

plt.figure(figsize=(10,4))

x = forecast.ds.values
y = forecast.trend.values
cp = model.changepoints

plt.plot(x,y)
ymin, ymax = plt.ylim()
plt.vlines(cp.values, ymin, ymax, linestyles="dashed")

Out[26]:

<matplotlib.collections.LineCollection at 0x7f69360e4438>

You'll notice that while there are 25 change point, they do not always change the trend line. This comes from placing a sparse prior on the induced change.

Mathematically, the linear trend component is described by

$g(t) =(k + \sum_{i} a_{ti} \delta_{i}) \times t + (b + \sum_{i} a_{ti} \gamma_{i})$

Where

$a$ is a matrix of shape |T| x |S| (where $T$ is the set of times we are training on and $S$ is a set of changepoints).
$\mathbf{\delta}^{T} = (\delta_{0} \, \delta_{1} \, \delta_{2} \dots)$, a vector of trend changes. Each $\delta_{i}$ is the change of the trend at changepoint $i$, these are parameters we fit.
$\gamma$ is a vector whose value for component $i$ is $- \delta_{i} t_{i}$, where $s_{i}$ is the time of checkpoint $i$. This correction to the offset ensures that the trend is continuous.

We can check the value for $a$ that the model uses:

In [11]:

model.get_changepoint_matrix()

Out[11]:

array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       ..., 
       [ 1.,  1.,  1., ...,  1.,  1.,  1.],
       [ 1.,  1.,  1., ...,  1.,  1.,  1.],
       [ 1.,  1.,  1., ...,  1.,  1.,  1.]])

The values of $\delta$ used come from fitting our model. In fitting, a Laplace distribution is used as a prior. The scale parameter of this prior can be adjusted by setting the changepoint_prior_scale parameter on the constructor. This prior is equivalent to $L_{1}$ regularisation.

Seasonality¶

Seasonality refers to patterns in the data which repeat periodically. For timeseries measured on a daily level over years, we often see seasonality on the scale of weeks and years. Prophet attempts to models this by including weighted terms with the expected periodicity. Currently, Prophet only supports weekly and yearly seasonality.

The periodic components used are the lowest $n$ components of the fourier series associated with a periodicity. We can look at a few of these components below

In [12]:

# plot individual components
seasons = model.make_all_seasonality_features(tfl)

seasons.yearly_delim_1.plot(figsize=(10,3))
seasons.yearly_delim_2.plot(figsize=(10,3))
plt.show()

seasons.yearly_delim_5.plot(figsize=(10,3))
seasons.yearly_delim_6.plot(figsize=(10,3))
plt.show()

seasons.yearly_delim_10.plot(figsize=(10,3))
seasons.yearly_delim_11.plot(figsize=(10,3))
plt.show()

Mathematically, we write the seasonality components as:

$s(t) = \sum_{i} \beta_{t} X_{i, t}$

Where

$\beta_{i}$ are the weighting of each term, which we will fit
$X_{i}$ are the time series associated with component $i$

And the $\beta_{i}$ terms are fitted with a normal prior, which is equivalent to $L_{2}$ regularization. The width can be adjusted with the parameter seasonality_prior_scale in the models constructor.

We can take a look at the output of a subset of fitted seasonsal components with the following

In [13]:

subset = [4,10]
beta = np.array(model.params["beta"])[0]

(beta[subset] * seasons[seasons.columns[subset]]).sum(axis=1).plot()

Out[13]:

<matplotlib.axes._subplots.AxesSubplot at 0x7f693a1f6748>

Holidays¶

In time series, there are often days which fall outside the regular pattern of trend and seasonality due to holidays or other large scale events. For example, let's see what happens to bike rentals around the end of December 2015:

In [14]:

tfl.set_index("ds").ix["2015-11-01":"2016-02-15"].plot()
plt.vlines("2015-12-25", 0, 35000, linestyle="dashed")

Out[14]:

<matplotlib.collections.LineCollection at 0x7f693a1f4860>

The days around Christmas and new years have much lower rental rates than the weeks surrounding it. Interestingly given this decrease, Christmas day (the dashed line) has a higher rental rate then we would expect. We can find similar patterns

To the effects of holidays into account, you can pass a list of holiday dates to Prophet's constructor:

In [15]:

holidays = pd.DataFrame({
    "holiday":"chirstmas",
    "ds": ["20" + str(i) + "-12-25" for i in range(11,17)],
    "lower_window":-1,
    "upper_window":2
})

holidays

Out[15]:

	ds	holiday	lower_window	upper_window
0	2011-12-25	chirstmas	-1	2
1	2012-12-25	chirstmas	-1	2
2	2013-12-25	chirstmas	-1	2
3	2014-12-25	chirstmas	-1	2
4	2015-12-25	chirstmas	-1	2
5	2016-12-25	chirstmas	-1	2

In [16]:

# we fit as before
model_with_holidays = fbprophet.Prophet(holidays=holidays)
model_with_holidays.fit(tfl)

# and predict
forecast_with_holiday = model_with_holidays.predict(future)

# and look can examine the holiday components
model_with_holidays.plot_holidays(forecast_with_holiday);

When you pass Prophet holiday dates, it takes them into account in a very similar way to seasonality: each holiday is assigned a parameter $\kappa_{i}$, which is used as a weight for an identity vector $Z$ which is 1 on all days the holiday falls, and 0 on all other days. In fact, for the purposes of predicting and fitting, holiday components are combined seasonal components.

The total holiday component is the described by

$h(t) = \sum_{i} \kappa_{i} Z_{t,i}$

This model assumes that the effects of the holidays are independent of each other.

And the $\kappa_{i}$ terms are fitted with a normal prior, whose width can be adjusted with the parameter holidays_prior_scale in the models constructor.

Fitting Parameters¶

All parameters of the model are fitted using STAN. There are already very good introductions to what STAN is and how it works, so I won't go into too much detail here.

The STAN code itself is particularly easy to read, so I'm going to reproduce the code here. For example, the following code is used to fit a model with linear trend:

data {
  int T;                                // Sample size
  int<lower=1> K;                       // Number of seasonal vectors
  vector[T] t;                            // Day
  vector[T] y;                            // Time-series
  int S;                                // Number of changepoints
  matrix[T, S] A;                   // Split indicators
  real t_change[S];                 // Index of changepoints
  matrix[T,K] X;                // season vectors
  real<lower=0> sigma;              // scale on seasonality prior
  real<lower=0> tau;                  // scale on changepoints prior
}

parameters {
  real k;                            // Base growth rate
  real m;                            // offset
  vector[S] delta;                       // Rate adjustments
  real<lower=0> sigma_obs;               // Observation noise (incl. seasonal variation)
  vector[K] beta;                    // seasonal vector
}

transformed parameters {
  vector[S] gamma;                  // adjusted offsets, for piecewise continuity

  for (i in 1:S) {
    gamma[i] = -t_change[i] * delta[i];
  }
}

model {
  //priors
  k ~ normal(0, 5);
  m ~ normal(0, 5);
  delta ~ double_exponential(0, tau);
  sigma_obs ~ normal(0, 0.5);
  beta ~ normal(0, sigma);

  // Likelihood
  y ~ normal((k + A * delta) .* t + (m + A * gamma) + X * beta, sigma_obs);
}

The code blocks in the above can be interpreted in the following way:

data{...} defines the input to the model that we provide as users.
parameters{...} defines the parameters which are to be fitted by the model
transformed parameters{...} defines any transformed parameters which are to be used
model{...} defines the probabilistic model that is to be fitted by STAN.

The addative model that underlies Prophet is defined in the second to last line:

  y ~ normal((k + A * delta) .* t + (m + A * gamma) + X * beta, sigma_obs);

Which tells us that we expect our timeseries, $y$, to follow the additive model up to normally distributed error terms. The rest of the STAN text is just setting things up for this.

Calling .fit()¶

When we call the .fit(df) method, Prophet creates and formats the various pieces of data required for the model, and the calls optimizing on pySTAN. Once called, pySTAN attempts to fit the parameter values which estimate the posterior distribution defined by the model.

By setting the constructor argument mcmc_samples to a non-zero value, the fitting process will return not just the optimal value for each of the parameters, but also full a MCMC approximation of the posterior. This will make the fitting process take longer.

It is straightforward to view the parameters fitted by the model

In [28]:

model.params

Out[28]:

{'beta': array([[ 0.        , -0.02789544, -0.09795023,  0.0032827 , -0.01191055,
         -0.00079181, -0.01053872,  0.00922425, -0.00067823, -0.00432156,
         -0.0033903 ,  0.00964615, -0.00980707,  0.00425593, -0.00587887,
          0.00095963, -0.00622742,  0.00722454, -0.00857349,  0.00143998,
         -0.00183064, -0.03130348,  0.03356159,  0.01697193, -0.00058229,
         -0.00331231, -0.00206514]]),
 'delta': array([[ -6.28393636e-04,  -4.18861965e-01,  -7.86392209e-02,
          -2.92435593e-08,   4.56688711e-02,   2.57600035e-07,
          -1.52769503e-08,  -2.63458486e-07,  -4.79167582e-01,
          -5.73366291e-01,  -1.33463203e-07,   4.65374113e-07,
           2.31001984e-01,   4.15488010e-01,   4.45814979e-01,
           2.03386100e-08,  -7.04230890e-08,  -3.18685970e-01,
          -3.23622173e-01,   5.80621068e-08,   2.88568473e-07,
          -7.07708104e-09,   1.61374105e-07,   2.07247415e-05,
           1.99941271e-01]]),
 'gamma': array([[  2.01310208e-05,   2.68370602e-02,   7.55778400e-03,
           3.74734986e-09,  -7.31516791e-03,  -4.95143619e-08,
           3.41965797e-09,   6.74137488e-08,   1.37959847e-01,
           1.83449312e-01,   4.69773155e-08,  -1.78714228e-07,
          -9.61103145e-02,  -1.86177717e-01,  -2.14049041e-01,
          -1.04167334e-08,   3.83243250e-08,   1.83638590e-01,
           1.96719237e-01,  -3.71540975e-08,  -1.93900227e-07,
           4.98208138e-09,  -1.18772912e-07,  -1.59175428e-05,
          -1.59969233e-01]]),
 'k': array([[ 1.02398319]]),
 'm': array([[ 0.16432243]]),
 'sigma_obs': array([[ 0.07177915]])}

Making a Prediction¶

We can now proceed to our goal: predicting the future.

When you call Prophet's predict method on a dataframe of dates, it extrapolates the trend and seasonality components which were previously fitted, and uses them to make predictions using the same additive model we've been working with up to this point.

Behind the scenes Prophet calls three submethods: predict_trend, predict_seasonal_components and predict_uncertainty to achieve this. The first two of these work as you would expect: they extract the parameters obtained by fitting in the previous section, and project them forward in time.

The predict_uncertainty component requires some more discussion. To understand what's going on, we note that there are a few different sources in uncertainty in our output model:

Normally distributed noise on the model
Uncertainty in our fitted parameters
Uncertainty in future trends

The first of these is given by the parameter sigma_obs gives us a measure of the uncertainty related to our prediction once seasonal and trend components have been take into account.

To take into account our uncertainty in parameter values, it is possible to enable MCMC sampling as discussed above. In this case STAN will return samples from the posterior distribution defined by our additive model. When these samples are used to project the components forward, they give us an estimate of the distribution of outcomes, allowing us to estimate the uncertainty. If MCMC sampling is not enabled, the model will use the single values returned.

To take into account the uncertainty in the trend component, many "potential" trends can simulated. This is carried out in the sample_predictive_trend method. To do this, it is assumed that for the prediction period there are changepoints with Laplace distributed sizes and uniform random spacing. By simulating many of these possible trends, the uncertainty of the model is approximated.

We can get an example of a few of these simulated trends using the code below

In [31]:

df = model.setup_dataframe(future)

pred_trend = [model.sample_predictive_trend(df, 0) for _ in range(5)]

for i in pred_trend:
    plt.plot(i[1000:])

Conclusion¶

That's all there is to Prophet. When you dig into it, it's surprising how simple the ideas behind it are. However the combination of this simplicity and robust packaging, together with STAN's sampling abilities result in a very powerful library for fast time series forecasting.

Code¶

You can find the notebook for this post on github here.

Heavy Metal and Natural Language Processing - Part 3

2017-06-13T00:00:00+01:00

In two previous blog posts, I explored some natural language techniques using a set of metal lyrics. The first looked at how we could use word frequencies to understand text and the second looked at how we can generate text from a set of examples.

Back in April I had the opportunity to present this work at PyData Amsterdam. You can find a video of my talk here and the slides for it here. It was great fun, and there were lots of interesting talks there. A huge thank you to the PyData committee for organising it.

As part of the preparation for the talk, I tidied up my code for last two blog posts. You can find it on github here. Unfortunately, I'm still not releasing the lyrics themselves. I don't own the copyright for them and am not comfortable releasing them.

I also spend some time exploring another idea:

Quantifying Emotional Arcs

I came across an interesting paper: The emotional arcs of stories are dominated by six basic shapes. The paper explores the idea that most stories follow one of a few "emotional arc" - the large scale changes in emotions felt by the main characters of the story. Apparently this idea goes back to a rejected masters thesis by Kurt Vonnegut. The paper attempts to quantify this idea by applying sentiment analysis to a collection of books from project Gutenburg. The result is "time-series" measuring the average emotion of the story at any point. Decomposing this time series in various was you end up with the set of canonical emotional arcs.

I'm not entirely convinced that the results necessarily prove that there are canonical story arcs (the arcs themselves look a lot like just the lowest Fourier modes of the system...), they do suggest that there is some large scale structure to "emotion" of a story, and that we are able to measure it.

The way in which the "emotion" of the story was measured is also interesting. They used an idea called the Hedonometer - detecting sentiment by assigning a score to common English words. The average score over many words is used as a measure of the "happiness" of a document (original paper here). This method is simple and computationally easy to apply, but seems powerful when averaged over large quantities of text.

The idea of scoring words, and using there average to measure some property of a piece of text got me interested because from my first blog post on metal lyrics, I had a score for the "metalness" of words. I thought it would be fun to use this to look at how "metalness" and "happyness" intersected in word usage.

The Metal/Happy Plane

For the 6000 words which appear in both our score for "Metalness" and the Hedonometer dataset, we have can place the word at a point the in the two-dimensional "Happy/Metal" Plane. A sample of these words are plotted below.

It matches my intuition surprisingly well. Down at the bottom right corner, the metal-unhappy quadrant, we have a cluster of death-related words, ("death", "kill", "die). Slang words like "gonna" and "yeah" end up being metal-neutral due to their disproportionate use in lyrics (compared to the Brown corpus). In the unmetal-happy quadrant at the top left we have words like "tea", "writing" and "grass", and the bottom left corners, the unhappy-unmetal quadrant, things are mostly empty, with the occasional overused violent word appearing ("attack" and "forced").

We can also use this to explore some of the limitations of this approach. In the top right corner, the metal-happy quadrant, we can see words like "dream", "sky" and "love". This position doesn't seem that unreasonable, but words like "love" can mean take on a range of different emotional and metal meaning depending on the context in which it is used, and this context is something our simple word-scorers do not take into account. Consider the following three examples of how the word love is used (selected at random from texts I had open and in memory)

Hate
I'm your hate
I'm your hate when you want love
Pay
Pay the price
Pay, for nothing's fair

Metallica, Sad But True

They headed down to breakfast, where Mr. Weasley was reading the front page of the Daily Prophet with a furrowed brow and Mrs. Weasley was telling Hermione and Ginny about a love potion she’d made as a young girl. All three of them were rather giggly.

Harry Potter and the Prisoner of Azkaban

She fell in love with his greasy machine
She leaned over wiped his kickstart clean
She'd never seen the beast before
But she left there wanting more more more

Iron Maiden, From here to Eternity

In the first example, love refers to something you can have, in the second it is simply part of the name of a potion, and in the third it is used as some which happened to someone. Each of these is very different, yet in counting words we treat them the same. We make this assumption because taking into account context is very difficult in situations like this. Some approaches that might work are

to use phrases like "fell in love" in place of just "love"
to move to a more powerful representation, like in this example from openai, where they used the recurrent output of a RNN trained to predict the next word, with a linear classifier to predict sentiment
to try and infer the part of speech of each word, and then to apply similar word scoring to the (Word, POS) pairs

Metal Story Arcs

Using the above we can place individual words at a point in a two-dimensional plane. We can extend this to documents by using the average score of each document to define a location. Doing this we can look at Metallica's albums, and explore how they have changed their style over time:

We can see that Metallica's first few albums start deep in the metal, unhappy part of the spectrum, before moving to a happier place. The Black album is almost sentiment-neutral. Since then they've been becoming slowly darker and more metal, moving back to a similar style to their first few albums.

We can do the same for Iron Maiden:

Of course, there is nothing limiting us to applying this technique to Metal lyrics. We can just as easily explore the motion through the Happy/Metal plane of Harry Potter texts:

Where, unsurprisingly, Deathly Hallows is the most metal and unhappy of all the Harry Potter series. It is interesting the most of the books are clustered in similar parts of the plane, but the first and the last books are noticeably more metal than the others.

Conclusion

That's all for now. You can find the code used to produce the plots here.

Degenerate State

Causal Inference With Python Part 3 - Frontdoor Adjustment

Unobserved Variables¶

Front Door Adjustment¶

Estimating the ATE using the Front Door Adjustment¶

Linear Relationship¶

Beyond linear models¶

Conclusion¶

Code¶

Causal Inference With Python Part 2 - Causal Graphical Models

What is Structure?¶

Causal Graphical Models¶

From Bayesian networks to Causal Graphical Models¶

Causal Inference with Causal Graphical Models¶

When can we use the Adjustment Formula?¶

Relationship with Potential Outcomes¶

Actually Estimating Causal Effects¶

Example Use Case¶

Conclusions¶

Code¶

Causal Inference With Python Part 1 - Potential Outcomes

Introduction¶

Definitions of Causality¶

Potential Outcomes¶

Goals¶

Making Assumptions¶

Modeling the Counterfactual¶

Covariate Imbalance¶

Propensity Score¶

Inverse Propensity Score Weighting¶

Doubly Robust Weighted Estimator¶

Unconfoundedness and the Propensity Score¶

Trimming¶

Stratification¶

Which Technique to Use?¶

The Structure of Causal Inference¶

Code¶

Reducing the Variance of A/B Tests Using Prior Information

Introduction¶

Classic A/B Testing¶

Generate Dataset¶

Running a test¶

Improving the Power of our experiment¶

Variance Reduction¶

Control Variates¶

Control Variates in A/B tests¶

Control Variate Simulations¶

An Alternative View¶

Overfitting and Bias¶

Related Approaches¶

The Bayesian Elephant in the Room¶

Code¶

Generating Examples of Simpson's Paradox

Binary Case¶

Continuous Example¶

Code¶

Map Projections

Making A Prophet

Introduction¶

How it works¶

Components¶

Setting up the dataframe¶

Trend¶

Seasonality¶

Holidays¶

Fitting Parameters¶

Calling .fit()¶

Making a Prediction¶

Conclusion¶

Code¶

Heavy Metal and Natural Language Processing - Part 3

Quantifying Emotional Arcs

The Metal/Happy Plane

Metal Story Arcs

Conclusion