Operations

Use this page for local CLI operations and docs validation while working on OpenOpps v0.1. There is no hosted runtime to initialize; durable state lives in local SQLite files controlled by OPENOPPS_ settings.

Command Modes

Use uv run openopps ... inside the checkout. After uv tool install -e . from the repo root, the same examples can be run as openopps ... directly.

Public docs surfaces

The docs app exposes three v1 URL surfaces plus exact legacy redirects for moved routes.

Bare /jobs redirects to /, and /docs/explorer redirects to /explorer. Trailing-slash variants normalize through the same canonical destinations. Job detail routes and /jobs/sitemap/*.xml remain available under the /jobs/ prefix; do not add broad /jobs/:path redirects.

Storage

SQLite is the default DB-backed storage mode:

uv run openopps admin db init
uv run openopps admin db status
uv run openopps sync a16z --metrics-json
uv run openopps sources sync a16z --metrics-json
uv run openopps boards sync --source a16z --provider any --metrics-json
uv run openopps jobs sync --provider any --metrics-json

The default database path is controlled by OPENOPPS_DB_URL and resolves to openoppsdb.sqlite for sqlite:///openoppsdb.sqlite.

All sync commands in this section write local SQLite records unless they explicitly use --no-db or are documented as dry-run diagnostics.

admin db init applies the current v0.1 Alembic schema head for the durable app SQLite database. HTTP response cache rows live in that same SQLite database and are managed by cache.py, not by the Alembic schema history. If a pre-release local SQLite database was stamped before the v0.1 schema was finalized, OpenOpps reports the missing schema columns and asks you to reset that local database or point OPENOPPS_DB_URL at a new SQLite file.

Job storage separates stable posting identity from changing content. The jobs table tracks lifecycle fields such as status, first_seen_at, last_seen_at, closed_at, and current hashes. job_versions stores one normalized content state per posting, with child tables for locations, skills, skill keywords, and responsibility or qualification bullets. job_payload_snapshots stores distinct raw listing/detail payloads so raw upstream JSON is never discarded.

Every successful provider-route job sync writes a job_sync_runs row and job_sync_observations rows for new, unchanged, changed, reopened, and closed outcomes. Raw payload drift means provider JSON changed but normalized content stayed the same, such as an ignored metadata key, key order change, or provider-internal token change. Raw-only drift updates payload snapshots and observations but does not create a new job_versions row.

Use status or doctor to inspect the full local runtime state:

uv run openopps status
uv run openopps doctor --json

Cache

The shared request cache stores successful JSON request payloads in the configured SQLite application database for source adapters and job providers. It records deterministic request identity, payload hash, selected headers, freshness timestamps, ETag/Last-Modified validators, and stale-on-error eligibility.

uv run openopps cache status
uv run openopps admin cache purge --namespace http-json --json
uv run openopps sources sync a16z --refresh-cache --metrics-json
uv run openopps jobs sync --provider any --refresh-cache --metrics-json

--refresh-cache bypasses cache reads for that operation and lets successful responses update cache state.

admin cache purge deletes cache rows. Scope it with --namespace when you only need to invalidate one cache family.

Examples

Seed deterministic synthetic records for local demos and smoke tests without upstream network access:

uv run openopps examples seed --seed 42 --boards 4 --jobs-per-board 2 --json
uv run openopps jobs list --source example --json
uv run openopps providers coverage --json
uv run openopps admin sources yield --json

For a fully isolated smoke test, point OPENOPPS_DB_URL at a temporary SQLite file before seeding.

No-DB Source Sync

For one-off source extraction without persistence, use explicit JSONL output:

uv run openopps sources sync a16z --no-db --output /tmp/a16z-boards.jsonl

Human sync commands use a brief dynamic progress display by default. The top-level sync command reports source, board, and job stages in order; individual sources sync, boards sync, and jobs sync commands report the stage they own. Add --verbose when you need detailed provider warnings on stderr. Machine-readable modes such as --metrics-json keep stdout parseable.

Exports

uv run openopps boards export --format jsonl --output /tmp/openopps-boards.jsonl
uv run openopps boards export --provider ashbyhq --has-jobs --format jsonl --output /tmp/ashby-boards.jsonl
uv run openopps jobs export --remote Full --skill Python --format csv --output /tmp/openopps-jobs.csv
uv run openopps jobs export --status all --format jsonl --output /tmp/openopps-all-jobs.jsonl
uv run openopps jobs export --source a16z --salary-min 150000 --format parquet --output /tmp/openopps-jobs.parquet
uv run openopps jobs export --status all --format sqlite --output /tmp/openopps-jobs.sqlite

Use jsonl for auditability, csv for spreadsheet inspection, parquet for analytics workflows, and sqlite for portable relational handoff. JSONL exports stream records as they are encoded. Empty JSONL and CSV exports produce empty files; empty Parquet exports produce a readable empty Parquet table. CSV exports neutralize spreadsheet formula-leading strings by prefixing a single quote; JSONL and Parquet preserve values as-is for machine processing. SQLite exports should preserve the flattened export contract while storing nested values as stable JSON strings.

boards export accepts the same board filters as boards list, including --source, --provider, --market, --location, --domain, --has-jobs, --min-staff, --max-staff, and --limit.

jobs export accepts the same job filters as jobs list, including --source, --board, --provider, --location, --department, --team, --workplace-type, --remote, --employment-type/--type, --salary-min, --salary-max, --skill, --query, --posted-after, --posted-before, --status, and --limit. These filters use normalized current-version job fields; raw upstream payloads remain exported for auditability but are not part of primary filtering semantics. Job list and export commands default to --status open; pass --status all when you intentionally need closed jobs in an audit extract.

Route Probing

The stable everyday route resolution command is boards sync, which enriches boards and applies successful missing route probes:

uv run openopps boards sync --source a16z --provider all --limit 25 --metrics-json

The admin route-probing command remains dry-run-first for diagnostics:

uv run openopps admin providers probe-routes --source a16z --provider all --limit 25 --json

Use --apply only after inspecting matched routes.

Source Yield

Use source-yield reporting after source syncs, route probing, and job syncs when you need to compare which source families are turning into active job routes:

uv run openopps admin sources yield --json
uv run openopps admin sources yield --source sec-company-tickers

The report is offline and reads persisted SQLite records only. It does not fetch sources, probe routes, or sync jobs. Treat yieldScore as a local snapshot metric rather than a global benchmark unless the database was refreshed from a representative source set.

Validation

Use just from the repository root for the contributor validation graph:

just quick
just ci
just lock-check
just openspec-validate-all
just docs-check
just docs-test
just kaggle-meta
just kaggle-bundle-check kaggle/openoppsdb.sqlite

Run Python tests from the repository root:

uv run pytest
uv run pytest --cov=openopps --cov-report=term-missing
uv lock --check

Run docs checks from the docs app:

cd docs
pnpm data:generate
pnpm data:generate:search
pnpm types:check
pnpm build
pnpm lint
pnpm test

pnpm data:generate refreshes package-derived provider/source metadata. pnpm types:check also regenerates that metadata before generating Fumadocs MDX artifacts, generating Next.js route types, and running TypeScript without emitting files. pnpm lint runs the Next ESLint surface. pnpm test runs the docs Vitest suite. just docs-build runs the production docs build and the function trace guard that keeps committed search artifacts out of API route bundles. Use just docs-rtk-lint from the repository root for the optional maintainer rtk lint surface; it requires rtk locally and is not silently skipped inside just ci. pnpm data:generate:search refreshes the committed static search index used by Jobs and Explorer from kaggle/openoppsdb.sqlite; run it after updating the local OpenOppsDB snapshot. From the repository root you can also use just docs-search-index. The generated docs/public/data/openopps-search/ tree is intentionally committed so the docs host can serve search, filters, dashboard aggregates, and job preview/detail shards without a live backend. Preview descriptions are limited to the normalized description fields present in that snapshot; when those fields are empty, the jobs surface shows metadata and links back to the source posting.

just docs-search-index-check requires the ignored local kaggle/openoppsdb.sqlite snapshot and fails clearly when that file is unavailable. Do not remove committed search-index compatibility for the current artifact version unless the v4 generator has been run from that snapshot and docs/public/data/openopps-search/ has been refreshed.

Run OpenSpec checks from the repository root:

rtk npx -y @fission-ai/openspec@latest list --json
rtk npx -y @fission-ai/openspec@latest status --change prepare-v0-1-release --json
rtk npx -y @fission-ai/openspec@latest validate --all --strict

OpenSpec-backed changes should keep proposal, design, specs, tasks, docs, nested AGENTS.md, CI, and just recipes synchronized. Use instructions --json when handing an OpenSpec task graph to another agent.

Dependency And Secret Hygiene

uv.lock and docs/pnpm-lock.yaml are committed reproducibility artifacts. Local validation and CI both run uv lock --check, and docs installs use pnpm install --frozen-lockfile. Renovate is configured at the repo root for Python pyproject.toml/uv.lock and docs npm/pnpm dependency maintenance.

Keep credentials local. .env, .env.*, .envrc, Kaggle kaggle.json, local package-registry credential files, .netrc, key bundles, and token or credential JSON files are ignored; .env.example remains the tracked non-secret template. Do not print secrets in logs, docs, CI output, or generated artifacts.

Outbound fetch threat model

OpenOpps source and job providers fetch public HTTPS endpoints from the local CLI. openopps.http.assert_public_fetch_url is a defense-in-depth guard for that local threat model:

• URLs must be public https:// endpoints with a host.
• DNS is resolved once before the request; any address that resolves to a non-global-routable IP is rejected.
• Redirect hops are origin-checked, but each hop performs its own DNS lookup.

This guard is not rebinding-proof. httpx resolves DNS again at connect time, so a hostile or compromised resolver could return a public address during the pre-check and a private/metadata address when the socket connects (classic DNS TOCTOU / rebinding). Redirect chains widen the window because every hop repeats the lookup.

v0.1 intentionally stops at lightweight validation rather than connect-time IP pinning or a trusted resolver policy. Treat the check as reducing accidental SSRF against RFC1918/link-local targets, not as a network sandbox. If you need stronger outbound controls later, pin the vetted addresses into the client connect path or run ingestion behind an egress policy you control.

Telemetry Operations

The docs app telemetry source of truth should be a first-party event lake. The default local mode is no-op; production-like deployments should write append-only events to persistent storage, compact them into Parquet with a separate maintenance job, and query them with DuckDB or another local analytical engine.

The current collector accepts only noop and local-event-lake as OPENOPPS_TELEMETRY_SINK values. Cloudflare, Umami, or BI dashboards should be treated as downstream mirrors fed from the event lake or from sanitized aggregates, not as the canonical raw telemetry store.

PostHog can be enabled as a free hosted product-analytics mirror by setting OPENOPPS_POSTHOG_PROJECT_API_KEY. The route forwards only allowlisted, sanitized events after OpenOpps redaction and never forwards raw posting bodies, request headers, or raw IPs. Forwarding is bounded by OPENOPPS_POSTHOG_TIMEOUT_MS and runs after successful local-event-lake writes, so a slow hosted mirror cannot block the canonical sink. Browser session replay is a separate public-key opt-in through NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY; the browser SDK masks all inputs and text, blocks elements marked data-openopps-private, data-telemetry-private, or data-sensitive, disables network body/header capture, leaves sampling plus URL/event/linked-flag controls to PostHog project configuration, and keeps PostHog automatic pageview/autocapture events off so OpenOpps remains the event schema source of truth. Use PostHog for dashboards and masked replay while keeping local-event-lake as the canonical raw sink when persistent storage is available.

Telemetry operations should preserve the event lake, cap payload sizes, record redaction/truncation evidence, and avoid raw IP retention. Keep event names and property keys allowlisted so exploratory UI instrumentation does not accidentally persist secrets from query strings, stack traces, or copied provider payloads. Page-engagement stories should sum visibleDurationDeltaMs and durationDeltaMs; cumulative visibleDurationMs and durationMs are retained for last-known-state debugging.

OpenOppsDB Kaggle

The generated Kaggle bundle lives in kaggle/. It contains dataset-metadata.json, kernel-metadata.json, the connected openoppsdb-manager.ipynb notebook, public example notebooks under kaggle/starter/ and kaggle/examples/, and generated SQLite/CSV/Parquet artifacts when a data DB is supplied. dataset-metadata.json is the Kaggle UI source of truth for public file descriptions and field-level descriptors for every CSV and Parquet export. openoppsdb.sqlite remains directly readable and carries openopps_tables/openopps_columns metadata for SQLite clients; if Kaggle does not expose nested SQLite table previews for a fresh upload, use the mirrored CSV/Parquet exports for Kaggle-rendered table previews and field metadata. Full snapshot runs write private sync_metrics.json, status.json, coverage.json, and snapshot-quality.json evidence while validating the bundle, then prune those files before publishing. The live dataset recipes stage a temporary upload directory containing only Kaggle dataset control files plus openoppsdb.sqlite, exports/csv/*.csv, and exports/parquet/*.parquet; notebooks are pushed as separate Kaggle kernels.

Run the local non-live generation and validation path from the repository root:

just kaggle-meta
PYTHONPATH=scripts uv run python -m openopps_kaggle --data-db kaggle/openoppsdb.sqlite
just kaggle-bundle-check kaggle/openoppsdb.sqlite

For a fresh local SQLite bundle seed, initialize the DB and run top-level source discovery before bundle generation:

OPENOPPS_DB_URL="sqlite:///$PWD/.tmp/openoppsdb-operational.sqlite" uv run openopps admin db init
OPENOPPS_DB_URL="sqlite:///$PWD/.tmp/openoppsdb-operational.sqlite" uv run openopps sync --metrics-json --refresh-cache
PYTHONPATH=scripts uv run python -m openopps_kaggle --data-db .tmp/openoppsdb-operational.sqlite

The manager notebook should be scheduled in Kaggle as one daily full snapshot, for example 0 6 * * *. Each run installs OpenOpps from git+https://github.com/wyattowalsh/openopps.git@main by default, reads public snapshot input from wyattowalsh/openoppsdb, copies the verified openopps_kaggle runtime package from the private wyattowalsh/openoppsdb-manager-runtime input, copies the newest /kaggle/input/**/openoppsdb.sqlite ledger into /kaggle/working/openoppsdb/openoppsdb.sqlite, may restore large columns from prior Parquet exports when upgrading legacy thin snapshots, rehydrates the public SQLite snapshot into a fresh operational Alembic schema when needed, and runs bounded openopps jobs sync --metrics-json --freshness-seconds --limit. It writes private status and coverage evidence, then runs python -m openopps_kaggle to backfill derived tables, export parquet-first artifacts, regenerate metadata, prune private evidence, stage the public upload directory, and attempt best-effort live Kaggle file metadata repair. Run just kaggle-live-file-metadata from a browser-authenticated local maintainer environment after live publishes when the Kaggle DataBundle checklist or column-description score must be repaired authoritatively. Refresh the private runtime dataset with the current openopps_kaggle package before pushing the manager. The notebook seeds bounded Kaggle runtime defaults for source freshness, job-route freshness, route limits, concurrency, connection limits, timeouts, and retries while allowing OPENOPPS_ environment overrides.

The scheduled Kaggle notebook environment must expose Kaggle API credentials before the manager starts. Configure KAGGLE_USERNAME and KAGGLE_KEY, or KAGGLE_API_TOKEN, as Kaggle notebook secrets/environment variables; otherwise the manager fails fast before running the expensive sync.

Live deployment remains a local maintainer action with Kaggle CLI credentials:

just kaggle-dataset-create
just kaggle-dataset-version "OpenOppsDB daily snapshot"
just kaggle-runtime-generator-version "OpenOppsDB manager runtime generator"
just kaggle-notebook-push
just kaggle-example-notebooks-push
just kaggle-example-notebooks-status
just kaggle-example-notebooks-pull-check
just kaggle-live-verify

The public example notebooks are generated from the repo source of truth and pushed as separate Kaggle kernels: wyattowalsh/openoppsdb-starter-notebook, wyattowalsh/openoppsdb-advanced-usage, wyattowalsh/openoppsdb-hiring-market-map, and wyattowalsh/openoppsdb-skills-radar. They are read-only, internet-disabled, credential-free, and attached only to wyattowalsh/openoppsdb. After pushing, run just kaggle-example-notebooks-pull-check to pull and verify the live source bundles; just kaggle-example-notebooks-files page_size=200 lists output files emitted by those notebook runs.

Use just kaggle-dataset-create only for the first public upload; use just kaggle-dataset-version after the dataset exists. Use just kaggle-runtime-generator-create only for the first private manager-runtime upload; use just kaggle-runtime-generator-version before pushing the manager notebook after generator changes. The live write recipes stage temporary upload directories and run through the Kaggle CLI, so run kaggle auth login before publishing and retry with kaggle auth login --force if Kaggle write endpoints return 401. They fail clearly when Kaggle credentials are unavailable and are intentionally outside CI. just kaggle-notebook-push and just kaggle-example-notebooks-push pass a one-hour Kaggle kernel timeout by default; override their timeout argument only for a deliberate longer maintenance run.

CI/CD

GitHub Actions mirrors local validation instead of inventing a second build path. CI should stay least-privilege, cache-aware, and split into independently debuggable lanes:

• Python package tests and coverage through uv.
• Lock-file consistency through uv lock --check.
• OpenSpec strict validation for every active change.
• Docs generated data, MDX/type-check, build, tests, and lint through pnpm.
• Kaggle metadata generation for exported schema changes.
• Workflow lint and whitespace checks.

The broader rtk lint docs checklist remains an explicit maintainer recipe (just docs-rtk-lint) rather than a conditional CI step.

When CI changes, update Justfile, this operations page, root README, and repo instructions in the same logical change.

Troubleshooting