Resources

Check each source yourself: some need a free account or key, and they update at different speeds.

Real-time and benchmark data by track

Climate and heat-health

- Real-time: Open-Meteo, Meteostat, NASA POWER

- Benchmark: ERA5 (Copernicus), MODIS land surface temperature

Air quality

- Real-time: OpenAQ (free key), Copernicus CAMS

- Benchmark: Beijing Multi-Site Air Quality (UCI; Zhang et al., 2017)

Energy demand

- Real-time: ENTSO-E Transparency Platform, Open-Meteo

- Benchmark: UCI Electricity Load Diagrams 2011-2014; ETT (Zhou et al., AAAI 2021)

Public health

- Real-time: Weather APIs, health department case reports

- Benchmark: OpenDengue (Clarke et al., Scientific Data, 2024)

Food security

- Real-time: WFP food prices on HDX

- Benchmark: FAOSTAT price and production series

Education

- Real-time: Public education statistics

- Benchmark: OULAD (Kuzilek et al., 2017); UCI Student Dropout (Realinho et al., 2022)

Information integrity

- Real-time: GDELT, news RSS

- Benchmark: LIAR (Wang, ACL 2017); FakeNewsNet (Shu et al., 2020); Urdu fake news (Amjad et al., 2020)

Urban mobility

- Real-time: GTFS-Realtime feeds

- Benchmark: METR-LA, PEMS-BAY (Li et al., ICLR 2018)

Pakistani sources

Pakistan Bureau of Statistics, Pakistan Meteorological Department, NDMA and provincial PDMAs, State Bank of Pakistan, NEPRA, provincial health, education and environment departments, and Pakistan datasets on the Humanitarian Data Exchange (HDX). Check coverage and licence first.

Suggested tools

- Gathering: requests, httpx, pdfplumber, Tesseract OCR

- Data quality: pandera, Great Expectations

- Warehouse: DuckDB or SQLite

- Modelling: scikit-learn, gradient boosting, Prophet, transformers

- RAG: Chroma or FAISS, multilingual embeddings

- Agents: LangGraph, CrewAI, AutoGen, OpenAI Agents SDK, or plain Python tool calling

- Orchestration: Prefect (recommended), Airflow, Dagster

- Dashboard: Gradio or Streamlit on Hugging Face Spaces

- CI: GitHub Actions, pytest