{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2e1466cc",
   "metadata": {},
   "source": [
    "# Application Session 8 (Unit 4)\n",
    "\n",
    "# Interpreting Computing Data with Normal, Exponential, and Erlang Distributions\n",
    "\n",
    "In this practical session, **two real computing datasets** are used to study three continuous distributions from Unit 4:\n",
    "\n",
    "- **Normal distribution**\n",
    "- **Exponential distribution**\n",
    "- **Erlang distribution**\n",
    "\n",
    "We use:\n",
    "- **Alibaba machine usage data** for the **Normal** part, where we use memory utilization percentages as a continuous measurement that varies around an average.\n",
    "- **IT request timing data** for the **Exponential** and **Erlang** parts, where we use waiting times between support requests to model the time until the next request and the time until the 4th request arrives.\n",
    "\n",
    "From these datasets, we study:\n",
    "\n",
    "- how **memory utilization** varies around an average,\n",
    "- how long we wait until the **next request** arrives,\n",
    "- and how long we wait until **4 requests** arrive.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43e91372",
   "metadata": {},
   "source": [
    "## Practical Questions for This Session\n",
    "\n",
    "By the end of the notebook, we will answer these questions:\n",
    "\n",
    "1. **Using a fitted Normal model, what is the probability that memory utilization is less than 90%?**\n",
    "2. **Using a fitted Normal model, what is the probability that memory utilization is between 86% and 90%?**\n",
    "3. **Using a fitted Exponential model, what is the probability that the next request arrives within 5 minutes?**\n",
    "4. **Using a fitted Exponential model, what is the probability that we wait more than 10 minutes for the next request?**\n",
    "5. **Using a fitted Erlang model, what is the probability that 4 requests arrive within 30 minutes?**\n",
    "6. **Using a fitted Erlang model, what is the probability that waiting for 4 requests takes more than 60 minutes?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "134af15f",
   "metadata": {},
   "source": [
    "## Session Goals\n",
    "\n",
    "This session aims to help students:\n",
    "\n",
    "- use **real computing datasets**,\n",
    "- estimate parameters for **continuous distributions** from data,\n",
    "- compare empirical data with fitted theoretical distributions,\n",
    "- use **SciPy** to compute probabilities,\n",
    "- interpret distributions in a meaningful computing context.\n",
    "\n",
    "We will use the following ideas:\n",
    "\n",
    "- **Normal**: a continuous measurement centered around an average\n",
    "- **Exponential**: waiting time until the **next** event\n",
    "- **Erlang**: waiting time until the **4th** event\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e72f3546",
   "metadata": {},
   "source": [
    "## Step 1: Import libraries and load the CSV files\n",
    "\n",
    "We use:\n",
    "\n",
    "- **pandas** to load and organize the datasets\n",
    "- **NumPy** for calculations\n",
    "- **Matplotlib** for plotting\n",
    "- **SciPy** for probability distributions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7fab8a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import norm, expon, erlang\n",
    "\n",
    "# -----------------------------\n",
    "# Load Alibaba machine usage data\n",
    "# -----------------------------\n",
    "machine_file = \"Session_8_machine_usage.csv\"\n",
    "machine_df = pd.read_csv(machine_file)\n",
    "\n",
    "# -----------------------------\n",
    "# Load processed request dataset\n",
    "# -----------------------------\n",
    "request_file = \"Session_8_request_data.csv\"\n",
    "request_df = pd.read_csv(request_file)\n",
    "\n",
    "print(\"Alibaba machine usage shape:\", machine_df.shape)\n",
    "print(\"Request dataset shape:\", request_df.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c26df91",
   "metadata": {},
   "source": [
    "## Step 2: Prepare the data for the three models\n",
    "\n",
    "Because these are **real datasets**, some variables may contain extreme values.\n",
    "\n",
    "To make the practical session clearer:\n",
    "\n",
    "- for the **Normal** part, we use **memory utilization** from Alibaba machine usage data and keep the central values up to the **99th percentile**\n",
    "- for the **Exponential** part, we keep **positive waiting times** and remove the most extreme **top 1%** of gaps\n",
    "- for the **Erlang** part, we use the waiting time until **4 requests**, again focusing on the central **99%**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59c7c691",
   "metadata": {},
   "outputs": [],
   "source": [
    "# -------- Normal data from Alibaba memory utilization --------\n",
    "normal_raw = pd.to_numeric(machine_df[\"mem_util_percent\"], errors=\"coerce\").dropna()\n",
    "normal_upper = normal_raw.quantile(0.99)\n",
    "normal_lower = normal_raw.quantile(0.01)\n",
    "normal_data = normal_raw[(normal_raw >= normal_lower) & (normal_raw <= normal_upper)]\n",
    "\n",
    "# -------- Exponential data from request log --------\n",
    "exp_raw = request_df[\"minutes_until_next_request\"].dropna()\n",
    "exp_raw = exp_raw[exp_raw > 0]\n",
    "exp_upper = exp_raw.quantile(0.99)\n",
    "exp_data = exp_raw[exp_raw <= exp_upper]\n",
    "\n",
    "# -------- Erlang data from request log --------\n",
    "erlang_raw = request_df[\"minutes_until_4_requests\"].dropna()\n",
    "erlang_raw = erlang_raw[erlang_raw > 0]\n",
    "erlang_upper = erlang_raw.quantile(0.99)\n",
    "erlang_data = erlang_raw[erlang_raw <= erlang_upper]\n",
    "\n",
    "print(\"Normal data size =\", len(normal_data))\n",
    "print(\"Exponential data size =\", len(exp_data))\n",
    "print(\"Erlang data size =\", len(erlang_data))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "185430ed",
   "metadata": {},
   "source": [
    "## Step 3: Normal Distribution for memory utilization\n",
    "\n",
    "We now model:\n",
    "\n",
    "$$\n",
    "X = \\text{memory utilization percentage of a machine}\n",
    "$$\n",
    "\n",
    "We estimate:\n",
    "\n",
    "$$\n",
    "\\mu = \\text{sample mean}, \\qquad \\sigma = \\text{sample standard deviation}\n",
    "$$\n",
    "\n",
    "Because this is real data and can be noisy, we focus on the central part of the distribution to exclude outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7082c87e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Estimate Normal parameters\n",
    "mu = normal_data.mean()\n",
    "sigma = normal_data.std(ddof=1)\n",
    "\n",
    "print(\"Estimated mean (mu) =\", mu)\n",
    "print(\"Estimated standard deviation (sigma) =\", sigma)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbebbb96",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot histogram of the cleaned memory utilization data with fitted Normal PDF\n",
    "x = np.linspace(normal_data.min(), normal_data.max(), 500)\n",
    "pdf_normal = norm.pdf(x, loc=mu, scale=sigma)\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.hist(normal_data, bins=100, density=True, edgecolor=\"black\", alpha=0.7)\n",
    "plt.plot(x, pdf_normal, linewidth=2)\n",
    "plt.xlabel(\"Memory utilization (%)\")\n",
    "plt.ylabel(\"Density\")\n",
    "plt.title(\"Memory Utilization with Fitted Normal Curve\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ba918e",
   "metadata": {},
   "source": [
    "## Step 4: Answer the Normal questions\n",
    "\n",
    "We use the fitted Normal distribution to answer:\n",
    "\n",
    "1. **What is the probability that memory utilization is less than 90%?**\n",
    "2. **What is the probability that memory utilization is between 86% and 90%?**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7547c844",
   "metadata": {},
   "outputs": [],
   "source": [
    "q1_normal = norm.cdf(90, loc=mu, scale=sigma)\n",
    "q2_normal = norm.cdf(90, loc=mu, scale=sigma) - norm.cdf(86, loc=mu, scale=sigma)\n",
    "\n",
    "print(\"Question 1: P(X <= 90) =\", q1_normal)\n",
    "print(\"Question 2: P(86 <= X <= 90) =\", q2_normal)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c73f3aa",
   "metadata": {},
   "source": [
    "## Step 5: Exponential Distribution for waiting until the next request\n",
    "\n",
    "Now let\n",
    "\n",
    "$$\n",
    "Y = \\text{waiting time (in minutes) until the next request arrives}\n",
    "$$\n",
    "\n",
    "For an Exponential model, the parameter is:\n",
    "\n",
    "$$\n",
    "\\lambda = \\frac{1}{\\text{mean waiting time}}\n",
    "$$\n",
    "\n",
    "So we estimate $\\lambda$ from the cleaned waiting-time data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f737ee06",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Estimate Exponential parameter\n",
    "mean_wait = exp_data.mean()\n",
    "lam = 1 / mean_wait\n",
    "\n",
    "print(\"Estimated mean waiting time =\", mean_wait, \"minutes\")\n",
    "print(\"Estimated lambda =\", lam)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f22370f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot histogram with fitted Exponential PDF\n",
    "x = np.linspace(0, exp_data.max(), 500)\n",
    "pdf_exp = expon.pdf(x, scale=1/lam)\n",
    "\n",
    "plt.figure(figsize=(9, 5))\n",
    "plt.hist(exp_data, bins=50, density=True, edgecolor=\"black\", alpha=0.7)\n",
    "plt.plot(x, pdf_exp, linewidth=2)\n",
    "plt.xlabel(\"Minutes until next request\")\n",
    "plt.ylabel(\"Density\")\n",
    "plt.title(\"Waiting Time Until Next Request with Fitted Exponential Curve\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "237b50c1",
   "metadata": {},
   "source": [
    "## Step 6: Answer the Exponential questions\n",
    "\n",
    "We use the fitted Exponential distribution to answer:\n",
    "\n",
    "3. **What is the probability that the next request arrives within 5 minutes?**\n",
    "4. **What is the probability that we wait more than 10 minutes for the next request?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "227e9c8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "q3_exp = expon.cdf(5, scale=1/lam)\n",
    "q4_exp = 1-expon.cdf(10, scale=1/lam)\n",
    "\n",
    "print(\"Question 3: P(Y <= 5) =\", q3_exp)\n",
    "print(\"Question 4: P(Y > 10) =\", q4_exp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a8503b5",
   "metadata": {},
   "source": [
    "## Step 7: Erlang Distribution for waiting until 4 requests arrive\n",
    "\n",
    "Now let\n",
    "\n",
    "$$\n",
    "Z = \\text{waiting time (in minutes) until 4 requests arrive}\n",
    "$$\n",
    "\n",
    "If request arrivals follow an Exponential waiting-time model with rate $\\lambda$,\n",
    "then the waiting time until the **4th** request follows an **Erlang** distribution with:\n",
    "\n",
    "$$\n",
    "k = 4\n",
    "$$\n",
    "\n",
    "and the same rate parameter $\\lambda$.\n",
    "\n",
    "In SciPy, we use:\n",
    "\n",
    "- shape parameter `a = 4`\n",
    "- scale parameter `1 / lambda`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24c2fa19",
   "metadata": {},
   "outputs": [],
   "source": [
    "k = 4\n",
    "erlang_scale = 1 / lam\n",
    "\n",
    "print(\"Shape parameter k =\", k)\n",
    "print(\"Scale parameter =\", erlang_scale)\n",
    "print(\"Theoretical mean for Erlang =\", k / lam, \"minutes\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a57f1ed1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot histogram of waiting time until 4 requests with fitted Erlang PDF\n",
    "x = np.linspace(0, erlang_data.max(), 500)\n",
    "pdf_erlang = erlang.pdf(x, a=k, scale=erlang_scale)\n",
    "\n",
    "plt.figure(figsize=(9, 5))\n",
    "plt.hist(erlang_data, bins=150, density=True, edgecolor=\"black\", alpha=0.7)\n",
    "plt.plot(x, pdf_erlang, linewidth=2)\n",
    "plt.xlabel(\"Minutes until 4 requests\")\n",
    "plt.ylabel(\"Density\")\n",
    "plt.title(\"Waiting Time Until 4 Requests with Fitted Erlang Curve\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a037a39",
   "metadata": {},
   "source": [
    "## Step 8: Answer the Erlang questions\n",
    "\n",
    "We use the fitted Erlang model to answer:\n",
    "\n",
    "5. **What is the probability that 4 requests arrive within 30 minutes?**\n",
    "6. **What is the probability that waiting for 4 requests takes more than 60 minutes?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca07f80e",
   "metadata": {},
   "outputs": [],
   "source": [
    "q5_erlang = erlang.cdf(30, a=k, scale=erlang_scale)\n",
    "q6_erlang = 1-erlang.cdf(60, a=k, scale=erlang_scale)\n",
    "\n",
    "print(\"Question 5: P(Z <= 30) =\", q5_erlang)\n",
    "print(\"Question 6: P(Z > 60) =\", q6_erlang)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcc82c08",
   "metadata": {},
   "source": [
    "## Task\n",
    "\n",
    "Using the same notebook structure:\n",
    "\n",
    "1. For the **Normal** part, change the first question to compute  \n",
    "   **the probability that memory utilization is greater than 91%**.\n",
    "2. For the **Exponential** part, compute  \n",
    "   **the probability that the next request arrives within 3 minutes**.\n",
    "3. For the **Erlang** part, compute  \n",
    "   **the probability that waiting for 4 requests takes more than 30 minutes**.\n",
    "4. Change the Erlang part from **4 requests** to **3 requests** and repeat the calculations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da313651",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this session, Python was used to apply Unit 4 to real computing datasets:\n",
    "\n",
    "- **Normal distribution** was used to model **memory utilization** from Alibaba machine usage data,\n",
    "- **Exponential distribution** was used to model waiting time until the next request,\n",
    "- **Erlang distribution** was used to model waiting time until the 4th request,\n",
    "- **SciPy** made it possible to compute probabilities directly from fitted continuous distributions,\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}