Statistical Analysis

Table of Contents

• Summary
• Statistics High Level Overview
  • How are we using the data? Descirbe or Infer
  • What are the data types? Categorical or Continuous
  • Number of Independent and Dependent Variables
• Model Cheatsheet
• SciPy Stats
  • Example: One Sample T-Test
  • Example: Fisher’s Exact Test
• Statistician Tools
  • Example: Simulation with Coin Tosses
  • TODO Example: Shuffling with Two Groups

Summary

When I was younger, I wish someone told me to learn calculus if I wanted to be a rocket scientist, otherwise go learn statistics. Even with that advice, I found it difficult to learn statistics because there seems to be a lot of different naming conventions. Here is my take on the first principles of statistics where we try to break down statistical analysis into its smallest pieces; the idea is that we want to find useful patterns for learning statistics. Statistics is simply outcome = model + error

Statistics High Level Overview

I think data analysis can be broken down into three key parts. This is a very high level overview so it is probably oversimplifying a lot of complex topics, but should give you a sense of what data is.

1. How are we using the data? Do we want to describe or make inferences about the dataset?
   • If we want to just describe the dataset, we look at things like the mean, median, etc. We want to be careful to make sure that our measurements are good in the sense that they are reliable and valid, etc.
   • If we want to make some type of inference, we have additional criteria to consider and can look at it a few different ways.
     • What are we interested in checking, correlation or causality (cause and effect)? Correlation does not directly infer whereas Causality says a variable effects another.
     • How are we measuring? Are we looking at independent measures (aka cross-sectional study, a single point in time snapshot) or repeated measures (aka longitudinal study, measure variables at different points in time).
     • There’s additional considerations like if we have an adequate sample size, what should we consider statistically significant (p-value), one-tailed or two-tailed test, etc.
2. What are the data types? Are the variables categorical (dichotomous, nominal, ordinal) or continuous?
   • If the variables are continuous, we can use parametric tests (that assume a normal distribution) and if the variables are categorical, we use non-parametric tests, which make less assumptions
3. Number of independent variables and dependent variables
   • The number of variables affect the test we can use. I made a quick guide below with some examples.

How are we using the data?

Descriptive Statistics describes the dataset and is pretty straightforward. For example, what is the mode, median, mean? For more details, look up the Stats Primer post.

Inferential Statistics predicts values of an outcome variable based on some kind of model. This usually means we are looking to see if the independent variable(s) cause some kind of change in the dependent variable(s) or if there is a strong relationship between a set of variables. All inferential statistics can be described with:

outcome = model + error

Personal Note: A lot of data analyst jobs out there are just making graphs of descriptive statistics. It’s a great intro to data and is useful, but being able to make inferences from the dataset is a natural (and more difficult) next step.

What are the data types?

Data can be broken down into different levels of measurement that help determine what statistical tests we can do. Ideally we want more specific (continuous) instead of braoder (categorical-binary). Here they are from weakest to strongest in terms of statistical power (so you can find the right effect).

• Categorical (aka Qualitative)
  • Binary (aka Dichotomous) has only two distinct possibilities (e.g. either alive or dead, pregnant or not)
  • Nominal has two or more possibilities (e.g. human, cat, dog)
  • Ordinal has two or more possibilities and there is a logical order (e.g. first, second, third place in a race); you know which is larger and smaller, but not by how much
• Continuous (aka Quantitative)
  • Numerical data that are numbers (e.g. 1,000 cookies)

Number of Independent and Dependent Variables

See chart below on what statistical test to use based on the number (and type) of variables we have.

Model Cheat Sheet

We simply switch out the model, which is made up of variables and parameters. Like all cheat sheets, this is a large oversimplification of statistics, but could be useful as a high level starting point.

SciPy Stats

In Python, there’s an open-source library called SciPy that handles a lot of standard mathematics, science, and engineering calculations. There’s many subpackages, including linear algebra (scipy.linalg), spatial data structures and algorithms to compute triangulations and Voronoi diagrams (scipy.spatial), but for this article we want to look at the statistics side (scipy.stats). FYI there are other libraries like statsmodel that do this and more.

Example: One Sample T-Test

Problem

We want to compare a sample mean to a known population mean of the average American height (177cm). Does this data represent the population mean?

Code

"""
  Definitons:
    Assume statistical significance level (alpha) = .05
    Assume Two tailed
    Degrees of Freedom (dof) = For a simple t-test: number of samples - 1

  Example:
    If we look up degrees of freedom, say for 29 samples, assume a statistical
    significance .05, two-tailed test, we look up a 'critical value' table
    and get a critical value of 2.045; this means if we get a t-statistic of
    less than -2.045 or greater than 2.045, we reject the null hypothesis
"""

from scipy import stats
import numpy as np

data_a = [177, 177, 178, 179, 172, 173, 178, 177, 174, 175, 178, 179,
             178, 179, 178, 177, 173, 177, 175]
data_b = [180, 198, 200, 178, 199, 210, 270, 210, 430, 240, 260,
             180, 190, 188, 300, 210]

print "Running Data A; we accept the null hypothesis" \
      "(i.e. no relationship/difference between measured groups)"
print "Data A has a mean of:", np.mean(data_a)  # avg height: 176.526
dof = len(data_a)-1  # for a t-test, degrees of freedom = number samples-1
print "Data A has a length of:", len(data_a), " and " \
    " degrees of freedom:", len(data_a)-1
result = stats.ttest_1samp(data_a, 177)
print "t-statistic is {} and p-value is {}".format(result[0], result[1])
# t-statistic is -0.940589569294 and p-value is 0.359367763116
critical_value = stats.t.isf(0.05/2, dof)  # critical value for two sided test
print "critical value is:", critical_value  # 2.100

print "\nRunning Data B; we reject the null hypothesis" \
      "(i.e. there is a relationship/difference between measured groups)"
print "Data B has a mean of:", np.mean(data_b)  # avg height: 227.687
dof = len(data_b)-1  # for a t-test, degrees of freedom = number samples-1
print "Data B has a length of:", len(data_b), " and " \
    " degrees of freedom:", len(data_b)-1
result = stats.ttest_1samp(data_b, 177)
print "t-statistic is {} and p-value is {}".format(result[0], result[1])
# t-statistic is 3.13929630004 and p-value is 0.00675255864071
critical_value = stats.t.isf(0.05/2, dof)  # critical value for two sided test
print "critical value is:", critical_value  # 2.131

Explaination of Results

We did a t-test for the mean of ONE group of scores. We created two examples, one dataset that accepts the null hypothesis and the other that rejects the null hypothesis. As a quick overview:

• Null hypothesis is just a way of saying there is no relationship/difference among these variables. We either accept or reject the null hypothesis. If we accept the null hypothesis, this means there is no relationship/difference between variables. If we reject the null hypothesis, this means there is a relationship/difference between the groups.
• The statistical signifiance level (aka alpha) is a number we pick at the beginning of our experiment (usually .05 or .01) and use this as a threshold to determine if we reject the null hypothesis.
• The p-value can be seen as the probability of obtaining the observed sample results. If the p-value is equal or smaller than the statistical signifiance level, then the null hypothesis must be rejected. However, this does not automatically mean that the alternative hypothesis is true.
• The t-statistic used in this t-test assesses whether the size of the difference is significant. The greater this value, the greater the evidence against the null hypothesis. Check this with the critical value.
• The critical value is what we use to compare to the t-statistic to see if we reject the null hypothesis or not.

Example 1 - Similar Data

In data_a we had an average height of 176.526. Just by eyeballing, our data_a is near the mean of 177 that we passed in. For a t-test, we calculate the degrees of freedom as the number of samples-1, which comes out to be 19-1=18. We get a t-statistic that is -0.940 and a two tailed p-value of 0.359. We calculate the critical value to be 2.100. We compare the t-statistic with the critical value and find that the t-statistic value (-0.940) is within the critical value ranges (of -2.1 to 2.1) so we have to accept the null hypothesis (i.e. there is no relationship/difference between these two sets of numbers). With a p-value of .359, this is another indicator that we must reject the null hypothesis since it is greater than our signifiance level of .05.

Example 2 - Different Data

In data_b we had an average height of 227.687. Just by eyeballing, our data_b is not near the mean of 177 that we passed in. For a t-test, we calculate the degrees of freedom as the number of samples-1, which comes out to be 16-1=15. We get a t-statistic that is 3.139 and a two tailed p-value of 0.006. We calculate the critical value to be 2.131. We compare the t-statistic with the critical value and find that the t-statistic value (3.139) is within the critical value ranges (of -2.131 to 2.131) so we reject the null hypothesis (not necessarily have to accept the alternative hypothesis). With a p-value of .006, we can then accept the alternative hypothesis since it is less than our signifiance level of .05 (i.e. there is a relationship/difference between these two sets of numbers).

Example: Fisher’s Exact Test

Code

import pandas as pd
import scipy.stats as stats


if __name__ == '__main__':

    raw_data = {'gender': ['Male', 'Female', 'Male', 'Male', 'Female', 'Male',
                           'Female', 'Male', 'Male', 'Female', 'Female',
                           'Female', 'Female', 'Female', 'Male', 'Female',
                           'Male', 'Female', 'Male', 'Male', 'Female',
                           'Female', 'Female', 'Male', 'Male'],
                'diet': ['Yes', 'No', 'No', 'Yes', 'No', 'Yes', 'Yes', 'No',
                         'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'Yes',
                         'No', 'Yes', 'Yes', 'No', 'Yes', 'Yes', 'Yes',
                         'No', 'No']}
    df = pd.DataFrame(raw_data, columns=['gender', 'diet'])
    print df
    #     gender diet
    # 0     Male  Yes
    # 1   Female   No
    # 2     Male   No
    # 3     Male  Yes
    # 4   Female   No
    # 5     Male  Yes
    # 6   Female  Yes
    # 7     Male   No
    # 8     Male  Yes
    # 9   Female   No
    # 10  Female   No
    # 11  Female  Yes
    # 12  Female  Yes
    # 13  Female   No
    # 14    Male   No
    # 15  Female  Yes
    # 16    Male   No
    # 17  Female  Yes
    # 18    Male  Yes
    # 19    Male   No
    # 20  Female  Yes
    # 21  Female  Yes
    # 22  Female  Yes
    # 23    Male   No
    # 24    Male   No
    
    # See crosstab of counts
    a = df['gender']
    b = df['diet']
    
    ct = pd.crosstab(index=a, columns=b)
    ct = ct[['Yes', 'No']]  # Reorder columns
    print ct
    # diet    Yes  No
    # gender
    # Female    8   5
    # Male      5   7
    
    row_1 = list(ct.ix['Female'])
    row_2 = list(ct.ix['Male'])
    
    fvalue, pvalue = stats.fisher_exact([row_1, row_2])
    print "fvalue is {f}, pvalue is {p}".format(f=fvalue, p=pvalue)
    # fvalue is 2.24, pvalue is 0.433752668115

Statistician’s Tools

With a computer, we can do a lot of computations that can help verify our math (or save us from doing a lot of math). We can:

• Simulate Results
• Shuffle Data
• Bootstrap Data
• Cross Validate Models

Simulation with Coin Tosses

Say you toss a coin 30 times and see 22 heads. Is it a fair coin? Using a probabilistic model (i.e. we know it will be 50/50 chance for heads or tails), we can simulate the results.

Code

"""
    From: Statistics for Hackers by Jake VanderPlas

    Instead of calculating the probability behind a sampling distribution,
    we simply simulate the sampling distribution of say a coin flipping
    using a for-loop.  We count the number of occurrences something happens
    and can check the probability that it occurs.

    Sample Problem: You toss a coin 30 times and you see 22 heads.  Is the
    coin fair?  Assuming that the coin is not fair, we test the Null Hypothesis
    We reject the fair coin hypothesis at p < 0.05
"""


import numpy


if __name__ == '__main__':
    numpy.random.seed(1)
    M = 0
    for i in xrange(10000):  # do 10,000 trials
        trials = numpy.random.randint(2, size=30)  # each trial, flip 30 times
        if (trials.sum() >= 22):
            M += 1
    p = M/10000.0

    print 'Probability is: {0:.5f}'.format(p)  #p=0.00830, reject fair coin hypothesis

Related Posts

• 11 May 2023 DBT
• 30 Oct 2022 Godot
• 16 Oct 2022 AWS Fargate