Skip to content

Overview#

CI Coverage License Version Package Status PyPI Python Versions

tea-tasting is a Python package for statistical analysis of A/B tests that features:

  • Student's t-test and Z-test out of the box.
  • Extensible API: Define and use statistical tests of your choice.
  • Delta method for ratio metrics.
  • Variance reduction with CUPED/CUPAC (also in combination with delta method for ratio metrics).
  • Confidence interval for both absolute and percent change.
  • Sample ratio mismatch check.

tea-tasting calculates statistics within data backends such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Spark, and other of 20+ backends supported by Ibis. This approach eliminates the need to import granular data into a Python environment, though Pandas DataFrames are also supported.

tea-tasting is still in alpha, but already includes all the features listed above. The following features are coming soon:

  • More statistical tests:
    • Bootstrap.
    • Quantile test (using Bootstrap).
    • Asymptotic and exact tests for frequency data.
    • Mann–Whitney U test.
  • Power analysis.
  • A/A tests and simulations.

Package name#

The package name "tea-tasting" is a play of words which refers to two subjects:

  • Lady tasting tea is a famous experiment which was devised by Ronald Fisher. In this experiment, Fisher developed the null hypothesis significance testing framework to analyze a lady's claim that she could discern whether the tea or the milk was added first to a cup.
  • "tea-tasting" phonetically resembles "t-testing" or Student's t-test, a statistical test developed by William Gosset.