Configuration

All pipeline parameters are centralized in config.py at the project root.

config.py

# --- Active Tickers Check Job ---
ACTIVE_TICKERS_BATCH_SIZE = 100
ACTIVE_TICKERS_THREADS = 30

# --- Financial Data ETL Jobs ---
ETL_BATCH_SIZE = 50
ETL_THREADS = 10

# --- General ---
SCHEMA = "finance"
LOG_LEVEL = 20  # INFO

# Table name -> stockdex method mapping
FINANCIAL_TABLES = {
    "income_stmt": "yahoo_api_income_statement",
    "cash_flow": "yahoo_api_cash_flow",
    "balance_sheet": "yahoo_api_balance_sheet",
    "financials": "yahoo_api_financials",
}

Environment Variables

Variable	Required	Description
`PG_NEON_FINANCE_URL`	Yes (in CI)	PostgreSQL connection string for Neon database

The connection string is stored as a GitHub Secret and injected into workflows. For local development, it falls back to a hardcoded default in postgres_interface.py.

Dependencies

All Python dependencies are listed in finance/requirements.txt:

pandas — Data manipulation and melting
openpyxl — Reading Excel ticker files
python-dotenv — Environment variable loading
SQLAlchemy — Database ORM and connection management
psycopg2-binary — PostgreSQL driver
stockdex — Yahoo Finance API wrapper