{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "e3a691cf",
      "metadata": {
        "id": "e3a691cf"
      },
      "source": [
        "# Application Session 7 (Unit 3)\n",
        "\n",
        "# Analyzing User Behavior in Digital Platforms using Binomial & Poisson Distributions\n",
        "\n",
        "In this practical session, a CSV file extracted from the YOOCHOOSE e-commerce dataset is used to analyze user session behavior using Binomial and Poisson distributions. The dataset contains user click and purchase data from a major online shopping website and helps students in computing fields apply concepts from statistics and probability to understand user activity and system behavior.\n",
        "\n",
        "Each row in the CSV represents one user session (one visit to the website), with:\n",
        "\n",
        "- `session_id` = unique session identifier\n",
        "- `session_start` = time when the session started\n",
        "- `num_clicks` = total number of clicks in that session\n",
        "- `unique_items` = number of different viewed items\n",
        "- `session_duration_sec` = session duration in seconds\n",
        "- `purchased` = 1 if the session ended with a purchase, otherwise 0\n",
        "\n",
        "We focus on **two distributions only**:\n",
        "\n",
        "- **Binomial distribution** → useful when we ask about the number of purchases in a **fixed number of sessions**, for example \\(n=20\\)\n",
        "- **Poisson distribution** → useful when we ask about the number of **session starts (or session arrivals)** in a **fixed time interval**, for example one minute\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f1d200fd",
      "metadata": {
        "id": "f1d200fd"
      },
      "source": [
        "## Practical Questions for This Session\n",
        "\n",
        "By the end of the notebook, we will answer these questions:\n",
        "\n",
        "1. **What is the probability of getting at most 2 purchases in the next 20 user sessions (web visits)?**\n",
        "2. **What is the probability that none of the next 20 sessions results in a purchase?**\n",
        "3. **What is the probability that at least 50 session starts occur in the next minute?**\n",
        "4. **If 35 session starts occur in one minute, is that normal or unusually low?**  \n",
        "\n",
        "These questions help us understand how online users behave on a website and what the system should expect.\n",
        "They also help us decide whether something unusual is happening so the system can respond quickly.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7655f401",
      "metadata": {
        "id": "7655f401"
      },
      "source": [
        "## Session Goals\n",
        "\n",
        "This session aims to help students:\n",
        "\n",
        "- Become familiar with using a real computing dataset (the YOOCHOOSE e-commerce dataset)\n",
        "- Estimate the parameters of the **Binomial** and **Poisson** distributions from data\n",
        "- Compare empirical results with theoretical results\n",
        "- Use statistics and probability to understand system performance and user activity\n",
        "- Understand why these distributions are useful in real digital platforms\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "715669f7",
      "metadata": {
        "id": "715669f7"
      },
      "source": [
        "## Step 1: Import libraries and load the CSV\n",
        "\n",
        "We have already used NumPy, pandas, and Matplotlib before. In this python session, we also use **SciPy**, which is a Python library for scientific and statistical computing. SciPy provides ready-made functions for probability distributions such as the Binomial and Poisson distributions, so we can calculate probabilities without writing the formulas by hand.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9cbc4057",
      "metadata": {
        "id": "9cbc4057"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from scipy.stats import binom, poisson\n",
        "\n",
        "# Load the processed dataset\n",
        "file_name = \"Session_7_user_activity_data.csv\"\n",
        "df = pd.read_csv(file_name)\n",
        "\n",
        "# Convert session_start to datetime and round down to the nearest minute\n",
        "df[\"session_start\"] = pd.to_datetime(df[\"session_start\"], format=\"mixed\").dt.floor(\"min\")\n",
        "\n",
        "print(\"Number of rows in the dataset =\", len(df))\n",
        "\n",
        "# Preview the data\n",
        "df.head()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "32ac12bc",
      "metadata": {
        "id": "32ac12bc"
      },
      "source": [
        "## Step 2: Quick summary of the real data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6f1d70c3",
      "metadata": {
        "id": "6f1d70c3"
      },
      "outputs": [],
      "source": [
        "# Basic summary measures from the session data\n",
        "purchase_rate = df[\"purchased\"].mean()\n",
        "average_clicks = df[\"num_clicks\"].mean()\n",
        "average_duration = df[\"session_duration_sec\"].mean()\n",
        "\n",
        "print(\"Number of sessions =\", len(df))\n",
        "print(\"Purchase rate =\", purchase_rate)\n",
        "print(\"Average number of clicks per session =\", average_clicks)\n",
        "print(\"Average session duration (seconds) =\", average_duration)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3dad34b5",
      "metadata": {
        "id": "3dad34b5"
      },
      "outputs": [],
      "source": [
        "# Plot the distribution of the number of clicks per session\n",
        "plt.hist(df[\"num_clicks\"], bins=40, edgecolor=\"black\")\n",
        "plt.xlabel(\"Number of clicks in a session\")\n",
        "plt.ylabel(\"Frequency\")\n",
        "plt.title(\"Distribution of Session Clicks\")\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "81e75ed1",
      "metadata": {
        "id": "81e75ed1"
      },
      "source": [
        "## Step 3: Binomial Distribution Parameters\n",
        "\n",
        "- **One trial** = one session  \n",
        "- **Success** = a purchase  \n",
        "\n",
        "For 20 sessions,\n",
        "\n",
        "$$\n",
        "X = \\text{number of sessions (out of 20) that end with a purchase}\n",
        "$$\n",
        "\n",
        "So the Binomial parameters are:\n",
        "\n",
        "$$\n",
        "n = 20\n",
        "$$\n",
        "\n",
        "$$\n",
        "p = \\text{purchase probability estimated from the data}\n",
        "$$\n",
        "\n",
        "We also use:\n",
        "\n",
        "$$\n",
        "\\text{mean} = E(X) = np\n",
        "$$\n",
        "\n",
        "$$\n",
        "\\text{standard deviation} = \\sigma = \\sqrt{np(1-p)}\n",
        "$$\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2dbd7b49",
      "metadata": {
        "id": "2dbd7b49"
      },
      "outputs": [],
      "source": [
        "# Set the binomial parameters\n",
        "n = 20\n",
        "p = purchase_rate\n",
        "\n",
        "# Compute the mean and standard deviation\n",
        "mean = n * p\n",
        "std = np.sqrt(n * p * (1 - p))\n",
        "\n",
        "print(\"Estimated p =\", p)\n",
        "print(\"Expected number of purchases in 20 sessions = n*p =\", mean)\n",
        "print(\"Standard deviation =\", std)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0031b2bc",
      "metadata": {
        "id": "0031b2bc"
      },
      "source": [
        "## Step 4: Build and compare the empirical and theoretical PMF for the binomial example\n",
        "\n",
        "We randomly draw many samples of 20 sessions.\n",
        "For each sample, we count how many purchases happened. This gives an empirical distribution for the number of purchases in 20 sessions.\n",
        "Then we compute the theoretical binomial probabilities for the same values of \\(X\\).\n",
        "Finally, we draw a bar chart to compare the empirical PMF with the theoretical binomial PMF.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "34bec6df",
      "metadata": {
        "id": "34bec6df"
      },
      "outputs": [],
      "source": [
        "np.random.seed(7)\n",
        "num_samples = 1000\n",
        "\n",
        "# Build many random samples of 20 sessions and count purchases in each sample\n",
        "X = []\n",
        "for i in range(num_samples):\n",
        "    sample = df[\"purchased\"].sample(n)\n",
        "    X.append(sample.sum())\n",
        "\n",
        "X = np.array(X)\n",
        "\n",
        "# Possible values of X are 0, 1, 2, ..., n\n",
        "x = np.arange(0, n + 1)\n",
        "\n",
        "pmf_empirical_binom = []\n",
        "pmf_theoretical_binom = []\n",
        "\n",
        "# Compare the empirical PMF from the sampled data with the theoretical binomial PMF\n",
        "for i in x:\n",
        "    pmf_empirical_binom.append(np.mean(X == i))\n",
        "    pmf_theoretical_binom.append(binom.pmf(i, n, p))\n",
        "\n",
        "pmf_empirical_binom = np.array(pmf_empirical_binom)\n",
        "pmf_theoretical_binom = np.array(pmf_theoretical_binom)\n",
        "\n",
        "print(\"Sum of empirical PMF =\", pmf_empirical_binom.sum())\n",
        "print(\"Sum of theoretical PMF =\", pmf_theoretical_binom.sum())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9dde3f62",
      "metadata": {
        "id": "9dde3f62"
      },
      "outputs": [],
      "source": [
        "# Plot empirical and theoretical binomial PMFs side by side\n",
        "plt.bar(x - 0.2, pmf_empirical_binom, width=0.4, label=\"Empirical PMF\")\n",
        "plt.bar(x + 0.2, pmf_theoretical_binom, width=0.4, label=\"Theoretical PMF\")\n",
        "plt.xlabel(\"Number of purchase sessions in 20 sessions\")\n",
        "plt.ylabel(\"Probability\")\n",
        "plt.title(\"Empirical vs. Theoretical Binomial PMF\")\n",
        "plt.xticks(np.arange(0, 21, 1))\n",
        "plt.legend()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f76ea5d7",
      "metadata": {
        "id": "f76ea5d7"
      },
      "source": [
        "## Step 5: Answer binomial questions\n",
        "\n",
        "1. **What is the probability of getting at most 2 purchases in the next 20 user sessions?**\n",
        "2. **What is the probability that none of the next 20 sessions results in a purchase?**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "efba5e75",
      "metadata": {
        "id": "efba5e75"
      },
      "outputs": [],
      "source": [
        "# Answer the two binomial questions using the fitted Binomial distribution\n",
        "q1_at_most_2 = binom.cdf(2, n, p)\n",
        "q2_none = binom.pmf(0, n, p)\n",
        "\n",
        "print(\"Question 1: P(X ≤ 2) =\", q1_at_most_2)\n",
        "print(\"Question 2: P(X = 0) =\", q2_none)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "de82ffbd",
      "metadata": {
        "id": "de82ffbd"
      },
      "source": [
        "## Step 6: Poisson Distribution\n",
        "\n",
        "For the Poisson model, we now work with **time intervals**.\n",
        "\n",
        "We now count how many **sessions start** in each minute.\n",
        "\n",
        "Let:\n",
        "\n",
        "$$\n",
        "Y = \\text{number of session starts in one minute}\n",
        "$$\n",
        "\n",
        "This allows us to estimate the Poisson parameter:\n",
        "\n",
        "$$\n",
        "\\lambda = \\text{average number of session starts per minute}\n",
        "$$\n",
        "\n",
        "We will then compare the empirical PMF from the data with the theoretical Poisson PMF.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9397f3be",
      "metadata": {
        "id": "9397f3be"
      },
      "outputs": [],
      "source": [
        "# Choose a specific time window from the dataset\n",
        "start = pd.Timestamp(\"2014-04-01 08:00:00+00:00\")\n",
        "end = pd.Timestamp(\"2014-04-01 14:00:00+00:00\")\n",
        "\n",
        "# Build the list of one-minute intervals in the selected window\n",
        "all_minutes = pd.date_range(start=start, end=end - pd.Timedelta(minutes=1), freq=\"min\")\n",
        "\n",
        "\n",
        "print(\"Number of minutes in the selected window =\", len(all_minutes))\n",
        "\n",
        "# Count how many sessions started in each minute\n",
        "sessions_per_minute = []\n",
        "for m in all_minutes:\n",
        "    count = (df[\"session_start\"] == m).sum()\n",
        "    sessions_per_minute.append(count)\n",
        "\n",
        "# Estimate lambda as the average number of session starts per minute\n",
        "lam = sum(sessions_per_minute) / len(sessions_per_minute)\n",
        "print(\"Estimated lambda =\", lam)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0f5b5a79",
      "metadata": {
        "id": "0f5b5a79"
      },
      "outputs": [],
      "source": [
        "# Convert the list to a NumPy array for easier calculations\n",
        "sessions_per_minute = np.array(sessions_per_minute)\n",
        "\n",
        "# Possible values of Y: 0, 1, 2, ..., maximum observed count\n",
        "y = np.arange(0, sessions_per_minute.max() + 1)\n",
        "\n",
        "# Build the empirical and theoretical PMFs\n",
        "pmf_empirical_poisson = []\n",
        "pmf_theoretical_poisson = []\n",
        "\n",
        "for i in y:\n",
        "    pmf_empirical_poisson.append(np.mean(sessions_per_minute == i))\n",
        "    pmf_theoretical_poisson.append(poisson.pmf(i, lam))\n",
        "\n",
        "pmf_empirical_poisson = np.array(pmf_empirical_poisson)\n",
        "pmf_theoretical_poisson = np.array(pmf_theoretical_poisson)\n",
        "\n",
        "# Plot both PMFs for comparison\n",
        "plt.figure(figsize=(12, 5))\n",
        "plt.bar(y - 0.2, pmf_empirical_poisson, width=0.4, label=\"Empirical PMF\")\n",
        "plt.bar(y + 0.2, pmf_theoretical_poisson, width=0.4, label=\"Theoretical Poisson PMF\")\n",
        "plt.xlabel(\"Number of session starts in one minute\")\n",
        "plt.ylabel(\"Probability\")\n",
        "plt.title(\"Empirical vs. Theoretical Poisson PMF\")\n",
        "plt.xticks(np.arange(y.min(), y.max() + 1, 5))\n",
        "plt.legend()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "27e40f3f",
      "metadata": {
        "id": "27e40f3f"
      },
      "source": [
        "## Step 7: Answer practical Poisson questions\n",
        "\n",
        "3. **What is the probability that at least 50 session starts occur in the next minute?**\n",
        "4. **If 35 session starts occur in one minute, is that normal or unusually low?**  \n",
        "   (probability below 0.05 → unusual)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "29cc7b78",
      "metadata": {
        "id": "29cc7b78"
      },
      "outputs": [],
      "source": [
        "# Answer the two Poisson questions using the fitted Poisson distribution\n",
        "q3_at_least_50 = 1 - poisson.cdf(49, mu=lam)\n",
        "prob_exactly_35 = poisson.pmf(35, lam)\n",
        "\n",
        "print(\"Question 3: P(Y ≥ 50) =\", q3_at_least_50)\n",
        "print(\"Question 4: P(Y = 35) =\", prob_exactly_35)\n",
        "\n",
        "if prob_exactly_35 < 0.05:\n",
        "    print(\"6 session starts in one minute is unusually low.\")\n",
        "else:\n",
        "    print(\"6 session starts in one minute is normal.\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a1c05921",
      "metadata": {
        "id": "a1c05921"
      },
      "source": [
        "## Task\n",
        "\n",
        "Using the same CSV:\n",
        "\n",
        "1. Change the binomial setting from **20 sessions** to **30 sessions**.\n",
        "2. Recompute:\n",
        "   - the expected number of purchases\n",
        "   - the probability of no purchases\n",
        "3. For the Poisson part, answer:\n",
        "   - What is the probability of at most 45 session starts occur in one minute?\n",
        "   - If 60 session starts occur in one minute, does that look unusual?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "324687f6",
      "metadata": {
        "id": "324687f6"
      },
      "source": [
        "## Summary\n",
        "\n",
        "In this session, Python was used to apply Unit 3 to a real-world user activity dataset:\n",
        "\n",
        "- **Binomial distribution** modeled the number of purchases in a fixed number of sessions\n",
        "- **Poisson distribution** modeled the number of session starts in a fixed time interval\n",
        "- The empirical results from the data were compared with the theoretical models\n",
        "\n",
        "This session shows how probability distributions can help analyze user behavior in digital platforms."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}