{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fa6753da",
   "metadata": {},
   "source": [
    "# Application Session 9 (Unit 6)\n",
    "\n",
    "# Estimating Mobile Internet Speed in Saudi Arabia using Sampling Distributions and the Central Limit Theorem\n",
    "\n",
    "In this application session, we use a real filtered dataset based on **Ookla mobile-network speed tiles in Saudi Arabia**.\n",
    "\n",
    "Each row represents **one small geographic tile/location in Saudi Arabia**. For each tile, the dataset gives the average mobile download speed, upload speed, latency, and number of tests observed in that tile.\n",
    "\n",
    "The main goal is to study how a **random sample of locations** can be used to estimate a population parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d02f716d",
   "metadata": {},
   "source": [
    "## Practical Questions for This Session\n",
    "\n",
    "By the end of this notebook, we will answer these questions:\n",
    "\n",
    "1. **Can a random sample of Saudi mobile-network locations estimate the average mobile download speed across Saudi Arabia?**\n",
    "2. **How much does the sample mean change from one random sample to another?**\n",
    "3. **What is the sampling distribution of the sample mean?**\n",
    "4. **How does increasing the sample size reduce the standard error?**\n",
    "5. **How does the Central Limit Theorem help us approximate probabilities involving the sample mean?**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94b56ba1",
   "metadata": {},
   "source": [
    "## Session Goals\n",
    "\n",
    "This session aims to help students:\n",
    "\n",
    "- work with a real computing/network-performance dataset,\n",
    "- distinguish between a population parameter and a sample statistic,\n",
    "- use the sample mean as a point estimator of the population mean,\n",
    "- build a sampling distribution using repeated random samples,\n",
    "- compute and interpret the standard error,\n",
    "- use the Central Limit Theorem to approximate probabilities involving sample means,\n",
    "- understand why sampling is useful when the population is large."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26dfc0b3",
   "metadata": {},
   "source": [
    "## Step 1: Import libraries and load the dataset\n",
    "\n",
    "We use:\n",
    "\n",
    "- **pandas** to load and organize the dataset,\n",
    "- **NumPy** for numerical operations,\n",
    "- **Matplotlib** for visualization,\n",
    "- **SciPy** for normal-distribution probability calculations.\n",
    "\n",
    "Make sure the file **`Session_9_Saudi_Mobile_Speeds.csv`** is uploaded to the Colab session storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78da8805",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import norm\n",
    "from pathlib import Path\n",
    "\n",
    "file_name = \"Session_9_saudi_mobile_speeds.csv\"\n",
    "\n",
    "speed_df = pd.read_csv(file_name)\n",
    "\n",
    "speed_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71456bae",
   "metadata": {},
   "source": [
    "## Step 2: Inspect the dataset structure\n",
    "\n",
    "The filtered dataset contains four columns:\n",
    "\n",
    "| Column | Meaning |\n",
    "|---|---|\n",
    "| `download_mbps` | average mobile download speed in one Saudi geographic tile/location |\n",
    "| `upload_mbps` | average mobile upload speed in one Saudi geographic tile/location |\n",
    "| `avg_lat_ms` | average latency in milliseconds in one tile/location |\n",
    "| `tests` | number of Speedtest measurements contributing to that location average |\n",
    "\n",
    "Important: one row is **one geographic tile/location**, not one device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e47424d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Dataset shape:\", speed_df.shape)\n",
    "print(\"Columns:\")\n",
    "print(speed_df.columns)\n",
    "\n",
    "speed_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0813d21",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert columns to numeric in case any values were read as text\n",
    "numeric_columns = [\"download_mbps\", \"upload_mbps\", \"avg_lat_ms\", \"tests\"]\n",
    "\n",
    "for col in numeric_columns:\n",
    "    speed_df[col] = pd.to_numeric(speed_df[col], errors=\"coerce\")\n",
    "\n",
    "# Remove missing or invalid values\n",
    "speed_df = speed_df.dropna(subset=numeric_columns)\n",
    "speed_df = speed_df[\n",
    "    (speed_df[\"download_mbps\"] > 0) &\n",
    "    (speed_df[\"upload_mbps\"] > 0) &\n",
    "    (speed_df[\"avg_lat_ms\"] > 0) &\n",
    "    (speed_df[\"tests\"] > 0)\n",
    "].copy()\n",
    "\n",
    "print(\"Cleaned dataset shape:\", speed_df.shape)\n",
    "speed_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33d18d0b",
   "metadata": {},
   "source": [
    "## Step 3: Define the statistical problem\n",
    "\n",
    "In this session, we focus on:\n",
    "\n",
    "$$\n",
    "X = \\text{average download speed in one Saudi geographic location}\n",
    "$$\n",
    "\n",
    "The population is:\n",
    "\n",
    "$$\n",
    "\\text{All Saudi mobile-network geographic locations in the dataset}\n",
    "$$\n",
    "\n",
    "The population parameter is:\n",
    "\n",
    "$$\n",
    "\\mu = \\text{true mean download speed across all Saudi locations}\n",
    "$$\n",
    "\n",
    "A random sample is:\n",
    "\n",
    "$$\n",
    "\\text{a randomly selected group of Saudi locations}\n",
    "$$\n",
    "\n",
    "The sample statistic is:\n",
    "\n",
    "$$\n",
    "\\bar{x} = \\text{mean download speed of the selected sample of locations}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1084c8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Main variable for Unit 6\n",
    "X = speed_df[\"download_mbps\"]\n",
    "\n",
    "# Population parameters for the full dataset\n",
    "mu = X.mean()\n",
    "sigma = X.std(ddof=0)  # population standard deviation\n",
    "median = X.median()\n",
    "\n",
    "print(f\"Population size N = {len(X):,} tiles\")\n",
    "print(f\"Population mean μ = {mu:.2f} Mbps\")\n",
    "print(f\"Population median = {median:.2f} Mbps\")\n",
    "print(f\"Population standard deviation σ = {sigma:.2f} Mbps\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4570383d",
   "metadata": {},
   "source": [
    "## Step 4: Visualize the population distribution\n",
    "\n",
    "The original distribution of `download_mbps` is usually not normal. Network-speed data often contains many moderate values and some very high values.\n",
    "\n",
    "This is useful for Unit 6 because the Central Limit Theorem is about the distribution of **sample means**, not the original distribution of individual observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "633851d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 5))\n",
    "plt.hist(X, bins=100, edgecolor=\"black\")\n",
    "plt.xlabel(\"Download speed (Mbps)\")\n",
    "plt.ylabel(\"Frequency\")\n",
    "plt.title(\"Population Distribution of Saudi Mobile Download Speed by Tile\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7caf02bb",
   "metadata": {},
   "source": [
    "## Step 5: Take one random sample\n",
    "\n",
    "Suppose one student team collects a random sample of Saudi mobile-network locations.\n",
    "\n",
    "The sample mean is a **point estimate** of the population mean:\n",
    "\n",
    "$$\n",
    "\\bar{x} \\approx \\mu\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34a56afe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose one sample size\n",
    "n = 1000\n",
    "\n",
    "# Take one random sample of n locations/tiles\n",
    "sample = X.sample(n, replace=False)\n",
    "\n",
    "sample_mean = sample.mean()\n",
    "sampling_error = sample_mean - mu\n",
    "\n",
    "print(f\"Sample size n = {n}\")\n",
    "print(f\"Sample mean x̄ = {sample_mean:.2f} Mbps\")\n",
    "print(f\"Population mean μ = {mu:.2f} Mbps\")\n",
    "print(f\"Sampling error x̄ - μ = {sampling_error:.2f} Mbps\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b5d6ca4",
   "metadata": {},
   "source": [
    "## Step 6: Sampling Distributions — How Sample Means Vary Across Samples\n",
    "\n",
    "Imagine many teams collecting samples in parallel:\n",
    "\n",
    "- Team 1 samples 50 locations.\n",
    "- Team 2 samples 50 locations.\n",
    "- Team 3 samples 50 locations.\n",
    "- and so on.\n",
    "\n",
    "Each team gets a different sample mean. This variation among sample means is exactly why we study the **sampling distribution**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d18d4c4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_samples(n, sample_count):\n",
    "    sample_means = []\n",
    "    for sample in range(sample_count):\n",
    "        sample_data = X.sample(n, replace=False)\n",
    "        sample_means.append(sample_data.mean())\n",
    "    return np.array(sample_means)\n",
    "\n",
    "\n",
    "sample_count = 300\n",
    "n = 100\n",
    "\n",
    "sample_means = generate_samples(n, sample_count)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35c3f564",
   "metadata": {},
   "source": [
    "We now compare sample means from different samples to show how the sample mean varies from one random sample to another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "id": "c454a0c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 4))\n",
    "plt.bar(range(16), sample_means[0:16], edgecolor=\"black\")\n",
    "plt.axhline(mu, linestyle=\"--\", label=\"Population mean μ\")\n",
    "plt.xlabel(\"Sample\")\n",
    "plt.ylabel(\"Sample mean download speed (Mbps)\")\n",
    "plt.title(\"Sample Means from 16 Parallel Student Samples\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6523ae15",
   "metadata": {},
   "source": [
    "A sampling distribution is created by plotting the distribution of the sample means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7562f70e",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "sample_dist_mean= sample_means.mean()\n",
    "sample_dist_sd= sample_means.std(ddof=1)\n",
    "se = sigma / np.sqrt(n)\n",
    "\n",
    "print(f\"Mean of sample means = {sample_dist_mean:.2f} Mbps\")\n",
    "print(f\"Population mean μ = {mu:.2f} Mbps\")\n",
    "print(f\"Standard deviation of sample means = {sample_dist_sd:.2f} Mbps\")\n",
    "print(f\"Theoretical standard error σ/sqrt(n) = {se:.2f} Mbps\")\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "plt.hist(sample_means, bins=40, density=True, edgecolor=\"black\")\n",
    "plt.axvline(mu, linestyle=\"--\", label=\"Population mean μ\")\n",
    "\n",
    "x_values = np.linspace(sample_means.min(), sample_means.max(), 300)\n",
    "normal_curve = norm.pdf(x_values, loc=sample_dist_mean, scale=se)\n",
    "plt.plot(x_values, normal_curve, linewidth=2, label=\"Normal approximation\")\n",
    "plt.xlabel(\"Sample mean download speed (Mbps)\")\n",
    "plt.ylabel(\"PDF\")\n",
    "plt.title(\"Sampling Distribution of the Sample Mean\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4110f8c7",
   "metadata": {},
   "source": [
    "## Step 7: Effect of sample size on standard error\n",
    "\n",
    "For the sample mean:\n",
    "\n",
    "$$\n",
    "E(\\bar{X}) = \\mu\n",
    "$$\n",
    "\n",
    "and\n",
    "\n",
    "$$\n",
    "SE(\\bar{X}) = \\frac{\\sigma}{\\sqrt{n}}\n",
    "$$\n",
    "\n",
    "As the sample size \\( n \\) increases, the standard error decreases.\n",
    "\n",
    "This means larger samples usually give more stable estimates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "409e8b91",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_sizes = [10, 30, 50, 100, 300]\n",
    "\n",
    "comparison_rows = []\n",
    "\n",
    "for n in sample_sizes:\n",
    "    sample_means = generate_samples(n, sample_count)\n",
    "    simulated_se = sample_means.std(ddof=1)\n",
    "    theoretical_se = sigma / np.sqrt(n)\n",
    "\n",
    "    comparison_rows.append({\n",
    "        \"Sample size n\": n,\n",
    "        \"Mean of sample means\": sample_means.mean(),\n",
    "        \"Simulated SE\": simulated_se,\n",
    "        \"Theoretical SE = σ/sqrt(n)\": theoretical_se\n",
    "    })\n",
    "\n",
    "se_comparison = pd.DataFrame(comparison_rows)\n",
    "se_comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99c89197",
   "metadata": {},
   "source": [
    "## Step 8: Compare sampling distributions for different sample sizes\n",
    "\n",
    "The original download-speed data is skewed. However, according to the Central Limit Theorem, the sample mean becomes more stable and more approximately normal as \\( n \\) increases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4616231",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "sample_count=200\n",
    "for n in [5,10, 30, 100, 300]:\n",
    "    sample_means = generate_samples(n, sample_count)\n",
    "    se = sigma / np.sqrt(n)\n",
    "    sample_means_mean = sample_means.mean()\n",
    "\n",
    "    plt.figure(figsize=(10, 5))\n",
    "    plt.hist(sample_means, bins=40, density=True, edgecolor=\"black\")\n",
    "    plt.axvline(mu, linestyle=\"--\", label=\"Population mean μ\")\n",
    "\n",
    "    x_values = np.linspace(sample_means.min(), sample_means.max(), 300)\n",
    "    normal_curve = norm.pdf(x_values, loc=sample_means_mean, scale=se)\n",
    "    plt.plot(x_values, normal_curve, linewidth=2, label=\"Normal approximation\")\n",
    "\n",
    "    plt.xlabel(\"Sample mean download speed (Mbps)\")\n",
    "    plt.ylabel(\"Density\")\n",
    "    plt.title(f\"Sampling Distribution of x̄ for n = {n}\")\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8bff4dd",
   "metadata": {},
   "source": [
    "## Step 9: Use the CLT to answer a probability question\n",
    "\n",
    "Suppose a student team samples \\( n = 100 \\) Saudi locations.\n",
    "\n",
    "We want to estimate:\n",
    "\n",
    "$$\n",
    "P(\\bar{X} > 150)\n",
    "$$\n",
    "\n",
    "Here, 150 Mbps is used as a selected value to practice probability calculations involving the sample mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33089719",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 100\n",
    "se = sigma / np.sqrt(n)\n",
    "\n",
    "z = (150 - mu) / se\n",
    "probability = 1 - norm.cdf(z)\n",
    "\n",
    "print(f\"Sample size n = {n}\")\n",
    "print(f\"Population mean μ = {mu:.2f} Mbps\")\n",
    "print(f\"Standard error = {se:.2f} Mbps\")\n",
    "print(f\"P(x̄ > 150) = {probability:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "898fffd7",
   "metadata": {},
   "source": [
    "## Step 10: Compare CLT probability with simulation\n",
    "\n",
    "Now we estimate the same probability using repeated random samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79366ccc",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 100\n",
    "simulated_means = generate_samples(n, sample_count=1000)\n",
    "simulated_probability = np.mean(simulated_means > 150)\n",
    "\n",
    "print(f\"CLT probability = {probability:.4f}\")\n",
    "print(f\"Simulated probability = {simulated_probability:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b69af338",
   "metadata": {},
   "source": [
    "## Student Task\n",
    "\n",
    "Complete the following tasks:\n",
    "\n",
    "1. Repeat the analysis using `upload_mbps` instead of `download_mbps`. Does the sampling distribution show the same behavior?\n",
    "3. For $(n = 30)$, estimate $P(\\bar{X} > 100)$ using the simulated sampling distribution and using the theoretical normal approximation from the Central Limit Theorem. Compare the two results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80cc4839",
   "metadata": {},
   "source": [
    "\n",
    "## Summary: Points Learned\n",
    "\n",
    "- Understand population, sample, parameter, and statistic.\n",
    "- Use the sample mean $\\bar{x}$ to estimate the population mean $\\mu$.\n",
    "- Recognize that different samples give different values of $\\bar{x}$.\n",
    "- Understand sampling variability.\n",
    "- Build and interpret the sampling distribution of $\\bar{X}$.\n",
    "- Compute and interpret the standard error $SE(\\bar{X})$.\n",
    "- See how larger samples produce smaller standard error.\n",
    "- Apply the Central Limit Theorem to $\\bar{X}$.\n",
    "- Use the normal model to estimate probabilities involving $\\bar{X}$."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}