{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Archival Methods: CSV Workbench\n",
    "Use this notebook for a careful first inspection of a historical dataset. It does not alter the source file. Add the specific research question and variables at the top of your analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import re\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "pd.set_option('display.max_columns', 100)\n",
    "pd.set_option('display.max_colwidth', 120)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load a CSV\n",
    "In Google Colab, the cell opens a file picker. In local Jupyter, enter a path to the CSV when prompted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from google.colab import files\n",
    "    uploaded = files.upload()\n",
    "    csv_path = Path(next(iter(uploaded)))\n",
    "except ImportError:\n",
    "    csv_path = Path(input('Path to CSV: ').strip()).expanduser()\n",
    "\n",
    "if not csv_path.exists():\n",
    "    raise FileNotFoundError(csv_path)\n",
    "\n",
    "df_raw = pd.read_csv(csv_path, low_memory=False)\n",
    "print(f'{csv_path.name}: {df_raw.shape[0]:,} rows × {df_raw.shape[1]:,} columns')\n",
    "display(df_raw.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Inventory before cleaning\n",
    "Preserve the original column labels in `df_raw`. The summary below surfaces types, missingness, and examples before any transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inventory = pd.DataFrame({\n",
    "    'dtype': df_raw.dtypes.astype(str),\n",
    "    'missing_n': df_raw.isna().sum(),\n",
    "    'missing_pct': (df_raw.isna().mean() * 100).round(1),\n",
    "    'unique_n': df_raw.nunique(dropna=True),\n",
    "    'example': [s.dropna().iloc[0] if s.notna().any() else None for _, s in df_raw.items()],\n",
    "})\n",
    "display(inventory)\n",
    "print('Duplicate rows:', int(df_raw.duplicated().sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Make a working copy\n",
    "Column names are normalized only in `df`. Keep a written record of every substantive change you make next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_column(name):\n",
    "    name = re.sub(r'[^0-9a-zA-Z]+', '_', str(name).strip().lower())\n",
    "    return name.strip('_')\n",
    "\n",
    "df = df_raw.copy()\n",
    "df.columns = [clean_column(column) for column in df.columns]\n",
    "display(pd.DataFrame({'source_column': df_raw.columns, 'working_column': df.columns}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Explore numeric patterns\n",
    "These plots are descriptive, not explanatory. Check units, denominators, boundaries, collection practices, and missingness before making a historical claim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric = df.select_dtypes(include='number')\n",
    "display(numeric.describe().T)\n",
    "\n",
    "if not numeric.empty:\n",
    "    numeric.hist(figsize=(14, max(4, 2.6 * ((len(numeric.columns) + 3) // 4))), bins=24)\n",
    "    plt.tight_layout()\n",
    "else:\n",
    "    print('No numeric columns were inferred. Inspect the source formatting before coercing values.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Research log\n",
    "- Historical question:\n",
    "- Unit of observation:\n",
    "- Geographic and temporal coverage:\n",
    "- Source and provenance:\n",
    "- Known exclusions or missing records:\n",
    "- Transformations made:\n",
    "- Next test or visualization:"
   ]
  }
 ],
 "metadata": {
  "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" },
  "language_info": { "name": "python", "version": "3" }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}