data-fetcher-pipeline

License Python Version Data Integrity Architecture Config Security

A professional-grade, automated retrieval pipeline designed for Data Science, Machine Learning, and Business Intelligence workflows. This module ensures secure, rate-limited extraction from global data sources prior to downstream analysis.

1. High-Level Architecture & Value Proposition

The pipeline is architected to optimize developer velocity while maintaining strict data integrity and operational security.

  • Zero-Imputation Data Purity: Data is preserved precisely as served by the upstream source. The pipeline acts strictly as a retrieval and normalization layer, enforcing raw extraction without unauthorized data cleaning or unapproved imputation.
  • Dynamic Format Normalization & Automated Dictionaries: The extraction engine natively resolves data payloads and standardizes the output schema. Upon successful extraction, a comprehensive metadata dictionary (dataset_description.txt) is automatically generated, outlining null-density distributions and schema constraints.
  • Secure Configuration (Lazy-Loaded State): Eliminating insecure shell-based .env population, the pipeline now manages credentials via an isolated, OS-standard JSON configuration file (~/.config/data-fetcher-pipeline/config.json). This mitigates vulnerability flags generated by static application security testing (SAST) tools like Snyk and Socket. Credentials are fundamentally lazy-loaded and only evaluated at execution time for specific target endpoints.
  • Operational Efficiency: Substantial reduction in manual boilerplate code, payload parsing, and repetitive directory configuration.

2. Workspace Visualization & File Management

The pipeline dynamically provisions directory structures to prevent filesystem pollution. Data partitions are generated sequentially based on the target source, format, and topic schema.

./data_raw/datasets_of_data-fetcher-pipeline/
 ├── kaggle/
 │ └── CSV/
 │   └── retail_sales_data_2024/
 │     ├── _raw.csv
 │     └── dataset_description.txt
 └── openml_org/
   └── JSON/
     └── healthcare_metrics/
       ├── _raw.json
       └── dataset_description.txt

3. Deep Technical Guide

Hybrid Installation Strategy

The pipeline now supports dual installation vectors to maximize OS compatibility and user preference, while retaining full compatibility with AI agent orchestration.

Method 1: Global CLI (Recommended / Windows-Friendly) Deploy the Python package globally. This automatically resolves dependencies and exposes the data-fetcher executable to your system path.

pip install .

You can now execute the engine directly from any directory:

data-fetcher --source openml --query "finance"

Method 2: Local Python Execution Run the module directly without installing globally:

python -m data_fetcher.cli

AI Agent Deployment Context

Browse the package registry:

npx skills add zyadmad56-spec/data-fetcher-pipeline --list

Deploy the package via the Skills CLI framework:

npx skills add zyadmad56-spec/data-fetcher-pipeline

Agent-specific installations:

npx skills add zyadmad56-spec/data-fetcher-pipeline --agent codex
npx skills add zyadmad56-spec/data-fetcher-pipeline --agent claude-code
npx skills add zyadmad56-spec/data-fetcher-pipeline --agent cursor

Global provisioning:

npx skills add zyadmad56-spec/data-fetcher-pipeline --global

Works dynamically with Claude Code, Codex, Cursor, OpenCode, and other supported automated agents via the Skills CLI.

Engine Execution

Execute the pipeline via natural language execution through your agent interface. The pipeline handles configuration generation and execution handoff securely.

Use the data-fetcher-pipeline to get the latest COVID-19 dataset from WHO.
Use the data-fetcher-pipeline to download the SEC EDGAR 10-K filings for AAPL.
Fetch the housing prices dataset from Kaggle using the data-fetcher-pipeline.

OpenML Interactive Strategy

The pipeline integrates openml to allow CLI-based intelligent lookup. By triggering the data-fetcher executable, the OpenMLFetcher subclass resolves dynamic text queries (e.g., "finance") against the OpenML global index, applying secondary sorting algorithms (descending by NumberOfInstances) to extract and reconstruct the most optimal dataset payload.

Supported Data Sources

Source Description Typical use case
OpenML An inclusive, open-source machine learning platform for dynamically searching and retrieving rich datasets and experiments. Data Scientists and ML Engineers querying global index for top-ranked ML datasets.
Kaggle The premier platform for data science competitions and massive, diverse machine learning datasets. Data Scientists and ML Engineers training machine learning models or testing predictive algorithms.
SEC (EDGAR) The US Securities and Exchange Commission database, essential for fetching raw corporate financial filings (10-K, 10-Q) and deep market analysis data. Data Analysts and Business Analysts performing financial modeling or market analysis.
FRED Federal Reserve Economic Data, the ultimate source for macroeconomic time-series data, socioeconomic metrics, and financial health indicators. Data Engineers building macro-level data warehouses and researchers running global analyses.

The pipeline enforces sequential fetching and polite-request delays across all these sources to comply with server limitations and ensure stable, continuous extraction.

Internal Code Architecture

data-fetcher-pipeline/
├── README.md
├── SKILL.md
├── LICENSE
├── requirements.txt
├── setup.py
├── config_template.json
├── references/
│   └── source-constraints.md
├── src/
│   └── data_fetcher/
│       ├── __init__.py
│       ├── base.py
│       ├── cli.py
│       ├── config.py
│       ├── factory.py
│       ├── wizard.py
│       ├── format_alchemy.py
│       └── fetchers/
│           ├── __init__.py
│           ├── airbnb.py
│           ├── fred.py
│           ├── generic.py
│           ├── kaggle.py
│           ├── openml.py
│           ├── sec.py
│           └── yahoo.py
└── tests/
    ├── test_base_fetcher.py
    ├── test_cli.py
    ├── test_config.py
    ├── test_fetcher_engine.py
    ├── test_fetchers.py
    └── test_format_alchemy.py
  • cli.py: Entry point. Parses CLI arguments, orchestrates the setup wizard, and dispatches to the factory.
  • base.py: Abstract Base Class enforcing the scout → pre-flight → extract → validate → save lifecycle.
  • factory.py: Strategy Factory. Lazy-loads and routes source names to their concrete BaseFetcher subclass.
  • config.py: Credential management. Secure JSON config at ~/.config/data-fetcher-pipeline/config.json.
  • wizard.py: Interactive zero-args onboarding flow.
  • fetchers/: One module per data source. Each implements scout() and extract().
  • format_alchemy.py: Post-extraction CSV → SQLite → Excel transformation pipeline. without shell dependencies.

Antigravity's Architectural Assessment

An objective architectural evaluation of the data-fetcher-pipeline.

Hybrid Installation Viability: Deploying as a native Python package (pip install .) maximizes cross-platform accessibility. Python-native deployments empower robust CLI toolchains (data-fetcher) and gracefully handle rapid interactive wizarding.

Data Source Efficacy: The selected data architectures are fundamentally robust and deeply relevant for modern data engineering.

  • OpenML and Kaggle successfully accommodate highly specific, heavily dimensional payloads required for advanced predictive modeling.
  • SEC (EDGAR) provides unfiltered statutory records, requiring high parsing complexity but yielding unmatched institutional alpha.
  • FRED supplies rigorous, standardized macroeconomic telemetry. Collectively, these pipelines construct a formidable, institutionally viable data lake generation tool. Enforcing rigid "Zero-Imputation" rules across these endpoints proves that this repository is built strictly for enterprise-level manipulation, offloading data cleansing duties accurately back to the analytics layer.

4. Recent Updates

  • Kaggle & SEC EDGAR Architecture Integration: Fully implemented native extraction classes KaggleFetcher and SECFetcher. Configuration handling has been securely routed through the global state validation layer to prevent crashes.
  • Cross-Platform Path Unification: Removed all hardcoded absolute paths and shell dependencies. Directory provisioning is now 100% native Python, ensuring absolute compatibility across Windows, macOS, and Linux without fragile POSIX shell fallbacks.
  • Format Alchemy ETL Module: Deployed an isolated post-processing transformation engine (src/data_fetcher/format_alchemy.py). Upon successful raw extraction, the pipeline prompts the user (y/n) to automatically infer datatypes and convert the raw CSV payload into a local SQLite database and an auto-formatted Excel workbook without violating the core "Pure Fetcher Philosophy".

5. How to Install

For new engineers cloning the repository locally, execute the following protocol:

A Quick Note on the Package Footprint: You might notice the tests/ folder is visible right here on GitHub. We keep it tracked because it acts as essential, living documentation for developers and powers our CI/CD infrastructure. But don't worry about bloat! When you run a local pip install, Python's setup.py steps in and automatically filters out the test files. This guarantees that the package installed on your machine remains incredibly lightweight, clean, and free of any unnecessary testing overhead.

  1. Clone the Repository & Navigate: 
    git clone <repository_url>
cd data-fetcher-pipeline
  2. Install Dependencies: Ensure your virtual environment is active, then install all strict dependencies outlined in the pipeline: 
    pip install -r requirements.txt
  3. Trigger First-Run Wizard: Execute the engine without runtime arguments. The architecture will intercept the empty payload and automatically guide you through a secure setup sequence to instantiate your API keys. 
    data-fetcher