# Operations (/docs/operations) Use this page for local CLI operations and docs validation while working on OpenOpps v0.1. There is no hosted runtime to initialize; durable state lives in local SQLite files controlled by `OPENOPPS_` settings.
Local state Sync and seed commands write to the database selected by OPENOPPS\_DB\_URL.
Dry-run diagnostics Route probing and health checks report first and persist only with --apply.
Export boundary JSONL, CSV, Parquet, and SQLite exports preserve normalized records for audit or analysis.
## Command Modes [#command-modes] Use `uv run openopps ...` inside the checkout. After `uv tool install -e .` from the repo root, the same examples can be run as `openopps ...` directly. ## Public docs surfaces [#public-docs-surfaces] The docs app exposes three v1 URL surfaces plus exact legacy redirects for moved routes. | URL | Purpose | | ----------- | ----------------------------------------------------------------- | | `/` | Interactive jobs board (search, filters, inline preview) | | `/explorer` | Dataset explorer across jobs, boards, and providers | | `/jobs/:id` | Static job detail pages for shareable permalinks and SEO metadata | | `/?job=:id` | In-app deep link that opens a job preview on the board | Bare `/jobs` redirects to `/`, and `/docs/explorer` redirects to `/explorer`. Trailing-slash variants normalize through the same canonical destinations. Job detail routes and `/jobs/sitemap/*.xml` remain available under the `/jobs/` prefix; do not add broad `/jobs/:path` redirects. ## Storage [#storage] SQLite is the default DB-backed storage mode: ```bash uv run openopps admin db init uv run openopps admin db status uv run openopps sync a16z --metrics-json uv run openopps sources sync a16z --metrics-json uv run openopps boards sync --source a16z --provider any --metrics-json uv run openopps jobs sync --provider any --metrics-json ``` The default database path is controlled by `OPENOPPS_DB_URL` and resolves to `openoppsdb.sqlite` for `sqlite:///openoppsdb.sqlite`. All sync commands in this section write local SQLite records unless they explicitly use `--no-db` or are documented as dry-run diagnostics. `admin db init` applies the current v0.1 Alembic schema. Live `uv run alembic heads` is `0006_url_pull_runs` only. `0005_update_snapshot_ledger` is the landed update-snapshot ledger; `0006` adds `url_pull_runs` and membership-scoped URL-pull persistence. `0005` is no longer blocked. HTTP response cache rows live in that same SQLite database and are managed by `cache.py`, not by the Alembic schema history. Unsupported or partial pre-release schemas fail closed; reset that local database or point `OPENOPPS_DB_URL` at a new SQLite file instead of relying on silent repair. URL pull is opt-in persist: `jobs pull ` defaults to `--save` False; `--no-save` is the ephemeral alias; HTTP cache is independent; persist failure exits 9. Saved rows use reserved `url-pull` digest identity and are not catalog sources. Unscoped `jobs sync` excludes those routes. This page does not authorize Workers upload, Kaggle mutation, hosted-alpha, v7 7.6, or source-policy 1780 grants. Job storage separates stable posting identity from changing content. The `jobs` table tracks lifecycle fields such as `status`, `first_seen_at`, `last_seen_at`, `closed_at`, and current hashes. `job_versions` stores one normalized content state per posting, with child tables for locations, skills, skill keywords, and responsibility or qualification bullets. `job_payload_snapshots` stores distinct raw listing/detail payloads so raw upstream JSON is never discarded. Every provider-route job sync creates a pending `job_sync_runs` row before network access and finishes it as succeeded or failed with bounded error classification, authority, timing, and committed-batch metadata. A route may close unseen jobs only after its provider returns a complete authoritative snapshot. Truncated pagination, count mismatch, continuation loops, access/schema failures, and ambiguous empty results preserve existing job state and fail the run. Successful runs write `job_sync_observations` rows for `new`, `unchanged`, `changed`, `reopened`, and `closed` outcomes. Raw payload drift means provider JSON changed but normalized content stayed the same; raw-only drift updates payload snapshots and observations but does not create a new `job_versions` row. Use `status` or `doctor` to inspect the full local runtime state: ```bash uv run openopps status uv run openopps doctor --json ``` ## Cache [#cache] The shared request cache stores successful JSON response payloads in the configured SQLite application database for source adapters and job providers. Cache schema version 2 stores a SHA-256 request identity and redacted canonical location, never a raw request body, URL user information, or credential-bearing query value. It records the response payload hash, an allowlist of response headers, freshness timestamps, ETag/Last-Modified validators, and stale-on-error eligibility. Legacy cache rows are invalidated. ```bash uv run openopps cache status uv run openopps admin cache purge --namespace http-json --json uv run openopps sources sync a16z --refresh-cache --metrics-json uv run openopps jobs sync --provider any --refresh-cache --metrics-json ``` `--refresh-cache` bypasses cache reads for that operation and lets successful responses update cache state. `admin cache purge` deletes cache rows. Scope it with `--namespace` when you only need to invalidate one cache family. ## Examples [#examples] Seed deterministic synthetic records for local demos and smoke tests without upstream network access: ```bash uv run openopps examples seed --seed 42 --boards 4 --jobs-per-board 2 --json uv run openopps jobs list --source example --json uv run openopps providers coverage --json uv run openopps admin sources yield --json ``` For a fully isolated smoke test, point `OPENOPPS_DB_URL` at a temporary SQLite file before seeding. ## No-DB Source Sync [#no-db-source-sync] For one-off source extraction without persistence, use explicit JSONL output: ```bash uv run openopps sources sync a16z --no-db --output /tmp/a16z-boards.jsonl ``` Human sync commands use a brief dynamic progress display by default. The top-level `sync` command reports source, board, and job stages in order; individual `sources sync`, `boards sync`, and `jobs sync` commands report the stage they own. Add `--verbose` when you need detailed provider warnings on stderr. Machine-readable modes such as `--metrics-json` keep stdout parseable. `jobSyncAttempts` counts durable route attempts, while `jobSyncRuns` counts only authoritative successes; failed, invalid, and partial snapshots therefore cannot masquerade as release-quality success evidence. ## Exports [#exports] ```bash uv run openopps boards export --format jsonl --output /tmp/openopps-boards.jsonl uv run openopps boards export --provider ashbyhq --has-jobs --format jsonl --output /tmp/ashby-boards.jsonl uv run openopps jobs export --remote Full --skill Python --format csv --output /tmp/openopps-jobs.csv uv run openopps jobs export --status all --format jsonl --output /tmp/openopps-all-jobs.jsonl uv run openopps jobs export --source a16z --salary-min 150000 --format parquet --output /tmp/openopps-jobs.parquet uv run openopps jobs export --status all --format sqlite --output /tmp/openopps-jobs.sqlite ``` Use `jsonl` for auditability, `csv` for spreadsheet inspection, `parquet` for analytics workflows, and `sqlite` for portable relational handoff. JSONL exports stream records as they are encoded. Empty JSONL and CSV exports produce empty files; empty Parquet exports produce a readable empty Parquet table. CSV exports neutralize spreadsheet formula-leading strings by prefixing a single quote; JSONL and Parquet preserve values as-is for machine processing. SQLite exports should preserve the flattened export contract while storing nested values as stable JSON strings. `boards export` accepts the same board filters as `boards list`, including `--source`, `--provider`, `--market`, `--location`, `--domain`, `--has-jobs`, `--min-staff`, `--max-staff`, and `--limit`. `jobs export` accepts the same job filters as `jobs list`, including `--source`, `--board`, `--provider`, `--location`, `--department`, `--team`, `--workplace-type`, `--remote`, `--employment-type`/`--type`, `--salary-min`, `--salary-max`, `--skill`, `--query`, `--posted-after`, `--posted-before`, `--status`, and `--limit`. JSON/JSONL also accepts `--profile core|search|full|raw` (default `full`). These filters use normalized current-version job fields; raw upstream payloads remain exported for auditability but are not part of primary filtering semantics. Job list and export commands default to `--status open`; pass `--status all` when you intentionally need closed jobs in an audit extract. ## Route Probing [#route-probing] The stable everyday route resolution command is `boards sync`, which enriches boards and applies successful missing route probes: ```bash uv run openopps boards sync --source a16z --provider all --limit 25 --metrics-json ``` The admin route-probing command remains dry-run-first for diagnostics: ```bash uv run openopps admin providers probe-routes --source a16z --provider all --limit 25 --json ``` Use `--apply` only after inspecting matched routes. ## Source Yield [#source-yield] Use source-yield reporting after source syncs, route probing, and job syncs when you need to compare which source families are turning into active job routes: ```bash uv run openopps admin sources yield --json uv run openopps admin sources yield --source sec-company-tickers ``` The report is offline and reads persisted SQLite records only. It does not fetch sources, probe routes, or sync jobs. Treat `yieldScore` as a local snapshot metric rather than a global benchmark unless the database was refreshed from a representative source set. `openopps providers coverage --json` also exposes `gaps.sourceScope` for v0.1 startup-board exclusions (WorkAtAStartup), Wellfound/Angel unsupported rationales, and the Editorial label audit decision. ## Quarantined source discovery [#quarantined-source-discovery] `openopps discovery` is advanced admin, not an everyday sync stage. It is not same-run with `openopps sync`, `sources sync`, `boards sync`, or `jobs sync`. Default unscoped `openopps sync` remains the snapshot writer and keeps using the last reviewed approved catalog. Scout output is an explicit quarantine directory, not a snapshot and not live publication evidence. ```bash uv run openopps discovery scout --output /absolute/quarantine-root --json uv run openopps discovery verify-scout /absolute/quarantine-root --json uv run openopps discovery preview-promotion --json just ci-discovery just source-policy-check just source-policy-audit ``` The same callbacks are aliased as `openopps admin sources scout`, `verify-scout`, and `preview-promotion`. None accept `--apply`. Scout writes only the required `--output` directory. It does not mutate operational SQLite, Git, the packaged catalog, Kaggle, or Cloudflare, and it cannot activate candidates in the same invocation. The current CLI scout publishes an evaluation bundle from an empty occurrence set against read-only v7 policy digests. Channel enumerators stay replay-library surfaces. Command strings are on [CLI](/docs/cli#quarantined-source-discovery). Isolated `OPENOPPS_DISCOVERY_*` limits are on [Configuration](/docs/configuration#isolated-discovery-scout). Taxonomy fields are on [Providers](/docs/providers#required-discovery-taxonomy). Skill contributor gates are on [Contributing](/docs/contributing#quarantined-discovery-and-the-source-scout-skill). ### Policy declaration versus independent verification [#policy-declaration-versus-independent-verification] Declared catalog status is not verified live rights. v7 source-policy is a deny-only overlay. `just source-policy-check` runs `scripts/source_policy_review.py validate`: canonical evidence bytes, schema, and the exact committed-v6 corpus identity. That structural gate is part of `just ci` via `ci-artifacts`. It is not permission to publish. `just source-policy-audit` runs the same validator, prints the eligibility summary, and exits 2 while any source is blocked. The audit type fixes `independentlyVerifiedAllowedCount` at `0` and labels allowed keys `repository_catalog_declarations_not_independent_legal_review`. Current committed-v6 counts: 1787 sources, 7 catalog-declared allowed, 0 independently verified, 1780 blocked. The 1780 stay blocked pending written Getro/Consider grants (`getro-terms-v3-1`, `consider-terms-observed-2026-08-13`) plus snapshot keys without independent review (`historical-snapshot-sources-unreviewed`). Do not treat a green structural check as eligibility, and do not render a selector or publish while the audit is red. Discovery promotion binds those read-only v7 digests in `openopps.discovery.policy` and requires independent positive closure of five operations: `access`, `license`, `redistribution`, `sync`, and `publication`. Deny-overlay strings such as `allowed`, `public`, or `granted` stay `unresolved`. Untrusted observations and `sourceAttribution` never grant permission; attribution is a requirement only. A `DiscoveryPromotionPolicyDecision` plus the evidence-only receipt still set `grantsAuthority=false`. Scout, verify, preview, CI, and scheduled workflows cannot mint a positive decision or invoke apply. ### Complete versus degraded accounting [#complete-versus-degraded-accounting] Conservation of a planned set is not a complete attestation. Candidate, source, and route denominators are exact and mutually exclusive. Scout candidate accounting (`ScoutCandidateAccounting` in `src/openopps/discovery/models.py`) conserves: ```text observedCandidateOccurrences = invalidOccurrences + normalizedOccurrences normalizedOccurrences = duplicateOccurrences + uniqueCandidates uniqueCandidates = alreadyApproved + quarantinedCandidates quarantinedCandidates = promotable + blocked + unsupported + inconclusive ``` Source terminals (`SourceAccounting` in `src/openopps/discovery/accounting.py`): ```text planned = succeeded + failed + timedOut + freshSkipped + policyBlocked + rateLimited + cancelled + unstarted ``` `complete` is true only when the run is terminal `succeeded`, every planned id is accounted, none of `failed` / `timedOut` / `policyBlocked` / `rateLimited` / `cancelled` / `unstarted` are present, and every `succeeded` or `freshSkipped` outcome is authoritative with a freshness-context digest that matches the pin. Fresh skips are not a completeness exemption without that digest. Route terminals (`RouteAccounting`) add `deferred`, `duplicateSkipped`, and `missingMetadata`. `complete` additionally forbids deferred, missing-metadata, and the failed classes above. Each `duplicateSkipped` route must name exactly one canonical representative that is an authoritative `succeeded` or `freshSkipped` bound to the same freshness context. A duplicate group of only skips fails closed. `classify_typed_degraded` returns `complete` with no class only when source and route are both complete, every operation terminal is `succeeded`, the operation channel state is `complete`, and `run_state` is `succeeded`. Otherwise it returns `degraded` plus one of `TYPED_DEGRADED_CLASSES`: `failed`, `timed_out`, `policy_blocked`, `rate_limited`, `cancelled`, `unstarted`, `missing_metadata`, `deferred`, `partial`, `nonterminal`. Multiple count classes collapse to `partial`, or to `nonterminal` when that class is present. Complete source or route accounting cannot pair with a non-succeeded run, and a complete attestation cannot carry a degraded class. Selector-bound `openopps sync --metrics-json` attaches `attestation` and `degradedClass` from that classifier. `jobSyncAttempts` versus `jobSyncRuns` still counts durable route attempts versus authoritative successes; those counters are not a complete attestation. `just source-discovery-accounting-check` conserves the pinned envelope as unstarted terminals and does not ingest. ### Bundle freshness, replay, rollback, and revocation [#bundle-freshness-replay-rollback-and-revocation] `discovery verify-scout` calls `verify_bundle` with `evaluation_bundle_verification_policy`: timezone-aware `observedAt`, a supported schema and profile, and `max_evidence_age` of 48 hours. Future-dated or stale evidence fails closed. That window is not the v7 public-data snapshot freshness gate. `OPENOPPS_DISCOVERY_EVIDENCE_RETENTION_SECONDS` (default `86400`) is a separate scout setting: the maximum age for exact-verified quarantine evidence to be reusable by a later isolated scout. Conditional reuse still requires those prior exact-verified bytes; verify-scout does not refetch. Replay here is offline fixture replay, not a live crawl. Channel enumerators (`enumerate_official_channel`, `enumerate_public_code_channel`, `enumerate_search_channel`, `enumerate_targeted_ats_channel`) consume maintainer-owned seeds plus captured observations and emit a closed `ChannelReplayReceipt`. They do not open sockets. Public CI and `just ci-discovery` set `OPENOPPS_DISCOVERY_NETWORK=disabled` and replay committed sanitized fixtures. Promotion-intent replay is a different, forbidden operation. Durable state is the hash-chained JSONL ledger at `src/openopps/discovery/data/promotion_decision_ledger.jsonl`. The only states are `reserved`, `applied`, and `revoked`; transitions are append-only (`reserved` → `applied` or `revoked`, `applied` → `revoked`, `revoked` terminal). Global replay keys are `decisionId` and `promotionIntentDigest` only. A semantic manifest digest is reusable content identity, not a global blacklist. `BundleVerificationPolicy.replayed_manifest_ids` / `revoked_manifest_ids` are legacy fields; CLI verify passes empty sets. There is no `openopps discovery rollback` command and no `--apply` flag. `preview-promotion` is a digest-bound dry-run (`applied=false`, `grantsAuthority=false`) and writes no repository files. The committed evidence-only receipt at `src/openopps/discovery/data/evidence_only_decision_receipt.json` also has `grantsAuthority=false`. Reserve, apply, recover, and revoke require `invocation_mode="maintainer"` in `openopps.discovery.promotion_runtime`. Recovery uses `choose_recovery_action`: `finalize` only when every owned path matches the journal `after` bytes; otherwise `restore_and_revoke` restores each journal `before` image and appends `revoked`. `revoke_promotion` appends `revoked` under the repository lock. That is a forward compensating ledger operation plus restored bytes, not a Git history rewrite. Reverted promotion digests stay revoked and need a fresh bundle and review. Candidate manifests cannot carry approval, reviewer, signature, receipt, or revocation fields. Kaggle `allow_no_rollback=1` on this page is a first-create dataset acknowledgement. It is not discovery ledger rollback. ### Skill non-authority and isolated validator [#skill-non-authority-and-isolated-validator] The portable skill at `agent-plugins/openopps.dev/skills/openopps-source-scout/` is advisory. Skill prose does not confine tools already authorized in a parent Codex, Cursor, or Grok Build harness. A suggestion is never approval, policy permission, review, promotion, or runtime activation. The parent harness is outside OpenOpps enforcement. Agent Plugin packaging, local client paths, and MCP filters are on [Agent Plugins](/docs/agent-plugins). Acceptance is only through the deterministic isolated validator `openopps.discovery.isolation.launch_isolated_scout` (committed fixtures via `agent-plugins/openopps.dev/skills/openopps-source-scout/scripts/validate_fixture.py`). The worker is a credential-free, allowlisted, bounded process that writes only a parent-owned new quarantine file; that is an application contract, not an OS sandbox claim. Do not live-install harness projections under `.agents/skills/` or `.cursor/skills/`. Selected Codex/Cursor copies must remain absent; Grok has no repository projection. Do not run `wagents --apply` or a live skill install from this workflow. `just source-discovery-skill-eval-check` runs the read-only evals, frontmatter, dry-run projection, and docs-steward skip gates. `resolve_docs_steward.py` searches with `uv run wagents skills search docs-steward --json` and skips when `wagents` is absent. ## Validation [#validation] Use `just` from the repository root for the contributor validation graph: ```bash just quick just ci just ci-discovery just agent-plugins-check just source-policy-check just lock-check just openspec-validate-all just web-check just web-build just web-test just web-e2e just web-a11y just web-lint just kaggle-meta just kaggle-bundle-check kaggle/openoppsdb.sqlite ``` `just ci` composes `ci-python`, `ci-openspec`, `ci-discovery`, `ci-web`, and `ci-artifacts`. These lanes cover the Python release gate (including `just agent-plugins-check`), strict OpenSpec validation, offline source-discovery gates with `OPENOPPS_DISCOVERY_NETWORK=disabled`, web type/build/unit/browser/accessibility/lint/search-artifact checks, Kaggle metadata and clean-bundle smoke, and repository drift. `ci-artifacts` runs `source-policy-check` and does not run `source-policy-audit`. `just ci-full` runs `just ci` plus the network-dependent `security-audit` (Python `pip-audit` and the full web dependency graph via `pnpm audit --audit-level high`) and `test-lowest-direct`. Use `ci` for day-to-day PR confidence; use `ci-full` before release or when dependency/security surfaces change. ### Jobs search public data release [#jobs-search-public-data-release] The committed v6 tree is a transition fallback when no channel is configured. For v7, set matching server and browser origins/channels as described in [Configuration](/docs/configuration#web-app-public-data-release). The shared snapshot client resolves the mutable channel once, pins the immutable release, and verifies manifest membership, byte length, and SHA-256 before serving search, details, metadata, or sitemap reads. Jobs search runs in a dedicated browser Web Worker using release-pinned chunks. The `/api/jobs/search` route is retained only as a fail-closed stale-client boundary and returns HTTP `410` with `browser_worker_required`; it never loads or scans the full production corpus. Saved-search counts use the same worker/session snapshot. Local saved jobs, searches, and retained details commit to IndexedDB before visible state advances; validated replacement imports are transactional and retain at most three bounded pre-import backups. The default-off offline-search installer is implemented for v7. An explicit opt-in performs a two-times-size quota preflight, downloads and verifies a bounded search/metadata projection, pins one immutable release, preserves the previous verified release until replacement succeeds, and removes only OpenOpps-owned caches on opt-out. Unit tests cover those local invariants. Deployed offline readiness is still unproven: the release gate remains open until a real deployed v7 release passes install, disconnect/readback, update/rollback, and complete Chromium/Firefox/WebKit journeys. See [Public Data Releases](/docs/public-data-releases) and `deployment/openopps-data/README.md` for v7 generation, rights/freshness gates, assets-only staging, exact remote readback, promote/rollback/re-promote barriers, archive recovery, and v6 exit criteria. Those tools prepare or verify a rollout; they do not prove that a live rollout occurred. Run Python tests from the repository root: ```bash uv run pytest uv run pytest --cov=openopps --cov-report=term-missing uv lock --check ``` Run web checks from the web app: Use Node 24.20.0 from `.node-version` and pnpm 11.24.0 from `web/package.json` exactly. Activate them with a version manager and Corepack before running the commands; with mise, prefix the command with `mise x node@24.20.0 pnpm@11.24.0 --`. ```bash cd web pnpm data:generate pnpm data:generate:search pnpm types:check pnpm build pnpm lint pnpm test ``` `pnpm data:generate` refreshes package-derived provider/source metadata. `pnpm types:check` also regenerates that metadata before generating Fumadocs MDX artifacts, generating Next.js route types, and running TypeScript without emitting files. `pnpm lint` runs the Next ESLint surface. `pnpm test` runs the docs Vitest suite. `just web-build` runs the production docs build and the function trace guard that keeps committed search artifacts out of API route bundles. Use `just web-rtk-lint` from the repository root for the optional maintainer `rtk lint` surface; it requires `rtk` locally and is not silently skipped inside `just ci`. `pnpm data:generate:search` refreshes the committed v6 transition index used by [Jobs](/) and [Explorer](/explorer) when no v7 channel is configured. Run it after updating the local OpenOppsDB snapshot; from the repository root you can also use `just web-search-index`. Preview descriptions are limited to the normalized description fields present in that snapshot; when those fields are empty, the jobs surface shows metadata and links back to the source posting. `just web-search-index-check` requires the ignored local `kaggle/openoppsdb.sqlite` snapshot and fails clearly when that file is unavailable. Do not remove the v6 tree until every v7 cutover exit criterion in [Public Data Releases](/docs/public-data-releases#version-6-transition-exit-criteria) has passed for one exact SHA. Run OpenSpec checks from the repository root. Pin `@fission-ai/openspec@1.6.0` in copy-paste commands for parity with CI; do not use a floating `@latest`. Override the `Justfile` default with `OPENOPPS_OPENSPEC='npx -y @fission-ai/openspec@1.6.0'` when `just openspec-*` should run a different invocation. ```bash rtk npx -y @fission-ai/openspec@1.6.0 list --json rtk npx -y @fission-ai/openspec@1.6.0 validate --all --strict ``` OpenSpec-backed changes should keep proposal, design, specs, tasks, docs, nested `AGENTS.md`, CI, and just recipes synchronized. Use `instructions --json` when handing an OpenSpec task graph to another agent. ## Dependency And Secret Hygiene [#dependency-and-secret-hygiene] `uv.lock` and `web/pnpm-lock.yaml` are committed reproducibility artifacts. Local validation and CI both run `uv lock --check`, and web installs use `pnpm install --frozen-lockfile`. Renovate is configured at the repo root for Python `pyproject.toml`/`uv.lock` and web npm/pnpm dependency maintenance. Dependabot owns GitHub Actions pin updates only. Keep credentials local. `.env`, `.env.*`, `.envrc`, Kaggle `kaggle.json`, local package-registry credential files, `.netrc`, key bundles, and token or credential JSON files are ignored; `.env.example` remains the tracked non-secret template. Do not print secrets in logs, docs, CI output, or generated artifacts. ## Outbound fetch threat model [#outbound-fetch-threat-model] OpenOpps source and job providers fetch public HTTPS endpoints from the local CLI. `openopps.http.assert_public_fetch_url` is a defense-in-depth guard for that local threat model: * URLs must be public `https://` endpoints with a host. * DNS is resolved once before the request; any address that resolves to a non-global-routable IP is rejected. * Redirect hops are origin-checked, but each hop performs its own DNS lookup. This guard is **not** rebinding-proof. httpx resolves DNS again at connect time, so a hostile or compromised resolver could return a public address during the pre-check and a private/metadata address when the socket connects (classic DNS TOCTOU / rebinding). Redirect chains widen the window because every hop repeats the lookup. v0.1 intentionally stops at lightweight validation rather than connect-time IP pinning or a trusted resolver policy. Treat the check as reducing accidental SSRF against RFC1918/link-local targets, not as a network sandbox. If you need stronger outbound controls later, pin the vetted addresses into the client connect path or run ingestion behind an egress policy you control. ## Telemetry Operations [#telemetry-operations] Telemetry in this section applies **only** to the Fumadocs/Next.js docs app (Jobs, Explorer, and `/docs/*`). The `openopps` CLI does not collect, store, or forward usage telemetry; CLI behavior is controlled solely by `OPENOPPS_*` settings in [Configuration](/docs/configuration). The docs app telemetry source of truth should be a first-party event lake. The default local mode is no-op; production-like deployments should write append-only events to persistent storage, compact them into Parquet with a separate maintenance job, and query them with DuckDB or another local analytical engine. | Sink | Operating model | Notes | | ------------------ | ---------------------------------------------------------- | ------------------------------------------------------- | | `noop` | Collects nothing. | Default for local development or hosts without storage. | | `local-event-lake` | Appends sanitized NDJSON events under UTC date partitions. | Canonical free sink when persistent disk is available. | The current collector accepts only `noop` and `local-event-lake` as `OPENOPPS_TELEMETRY_SINK` values. Cloudflare, Umami, or BI dashboards should be treated as downstream mirrors fed from the event lake or from sanitized aggregates, not as the canonical raw telemetry store. PostHog can be enabled as a free hosted product-analytics mirror by setting `OPENOPPS_POSTHOG_PROJECT_API_KEY`. The route forwards only allowlisted, sanitized events after OpenOpps redaction and never forwards raw posting bodies, request headers, or raw IPs. Forwarding is bounded by `OPENOPPS_POSTHOG_TIMEOUT_MS` and runs after successful `local-event-lake` writes, so a slow hosted mirror cannot block the canonical sink. Browser session replay is a separate public-key opt-in through `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY`; the browser SDK masks all inputs and text, blocks elements marked `data-openopps-private`, `data-telemetry-private`, or `data-sensitive`, disables network body/header capture, leaves sampling plus URL/event/linked-flag controls to PostHog project configuration, and keeps PostHog automatic pageview/autocapture events off so OpenOpps remains the event schema source of truth. Use PostHog for dashboards and masked replay while keeping `local-event-lake` as the canonical raw sink when persistent storage is available. Telemetry operations should preserve the event lake, cap payload sizes, record redaction/truncation evidence, and avoid raw IP retention. Keep event names and property keys allowlisted so exploratory UI instrumentation does not accidentally persist secrets from query strings, stack traces, or copied provider payloads. Page-engagement stories should sum `visibleDurationDeltaMs` and `durationDeltaMs`; cumulative `visibleDurationMs` and `durationMs` are retained for last-known-state debugging. | Variable | Default | Production rule | | ---------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NEXT_PUBLIC_OPENOPPS_TELEMETRY_ENABLED` | disabled | Enables the browser client only when explicitly set. | | `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY` | unset | Optional browser PostHog key for masked session replay; ignored unless browser telemetry is enabled. | | `NEXT_PUBLIC_OPENOPPS_POSTHOG_RECORDING` | disabled | Explicit replay opt-in. The PostHog client is initialized with session recording disabled unless this is `true`. | | `NEXT_PUBLIC_OPENOPPS_POSTHOG_HOST` | `https://us.i.posthog.com` | Optional browser PostHog ingestion host. | | `OPENOPPS_TELEMETRY_SINK` | `noop` | Use `local-event-lake` only when persistent storage is available. | | `OPENOPPS_TELEMETRY_DIR` | unset | Required for `local-event-lake`; events are partitioned by UTC date. | | `OPENOPPS_TELEMETRY_SALT` | test/dev fallback | Required before `local-event-lake` accepts writes; rotate through deployment config, not source. | | `OPENOPPS_TELEMETRY_IP_MODE` | `hash` | `raw` is rejected when `NODE_ENV=production`; use `hash` or `drop`. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_MAX` | `120` | Public POST limit per salted request fingerprint. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_WINDOW_MS` | `60000` | Rate-limit window in milliseconds. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_MAX_BUCKETS` | `4096` | Maximum in-process rate-limit fingerprints retained before oldest buckets are evicted. | | `OPENOPPS_TELEMETRY_TRUSTED_PROXY` | `none` | Valid values are `none`, `cloudflare`, `vercel`, and `forwarded`. Client IP headers are ignored unless a mode is selected; `cloudflare` trusts `cf-connecting-ip`, `vercel` trusts `x-vercel-forwarded-for`, and `forwarded` trusts the first `x-forwarded-for` hop only when your ingress strips client-supplied values. `x-real-ip` is not trusted by this route. | | `OPENOPPS_POSTHOG_PROJECT_API_KEY` | unset | Optional PostHog project key for sanitized hosted product analytics. | | `OPENOPPS_POSTHOG_HOST` | `https://us.i.posthog.com` | Optional PostHog ingestion host. | | `OPENOPPS_POSTHOG_TIMEOUT_MS` | `1500` | Best-effort PostHog forwarding timeout, clamped to 100-10000 ms. | ## OpenOppsDB Kaggle [#openoppsdb-kaggle] The generated Kaggle bundle lives in `kaggle/`. It contains `dataset-metadata.json`, `kernel-metadata.json`, the connected `openoppsdb-manager.ipynb` notebook, public example notebooks under `kaggle/starter/` and `kaggle/examples/`, and generated SQLite/CSV/Parquet artifacts when a data DB is supplied. `dataset-metadata.json` is the Kaggle UI source of truth for public file descriptions and field-level descriptors for every CSV and Parquet export. `openoppsdb.sqlite` remains directly readable and carries `openopps_tables`/`openopps_columns` metadata for SQLite clients; if Kaggle does not expose nested SQLite table previews for a fresh upload, use the mirrored CSV/Parquet exports for Kaggle-rendered table previews and field metadata. Full snapshot runs write private `sync_metrics.json`, `status.json`, `coverage.json`, and `snapshot-quality.json` evidence while validating the bundle, then prune those files before publishing. The live dataset recipes stage a temporary upload directory containing only Kaggle dataset control files plus `openoppsdb.sqlite`, `exports/csv/*.csv`, and `exports/parquet/*.parquet`; notebooks are pushed as separate Kaggle kernels. Run the local non-live generation and validation path from the repository root: ```bash just kaggle-meta PYTHONPATH=scripts uv run python -m openopps_kaggle --data-db kaggle/openoppsdb.sqlite just kaggle-bundle-check kaggle/openoppsdb.sqlite ``` For a fresh local SQLite bundle seed, initialize a **clean-schema** DB (never a legacy root ledger with `sources.enabled`) and run a bounded jobs sync before bundle generation. This is a faster local maintainer path, not the manager contract: the scheduled manager runs the full `openopps sync --metrics-json` pipeline under its notebook time budget. ```bash OPENOPPS_DB_URL="sqlite:///$PWD/.tmp/openoppsdb-operational.sqlite" uv run openopps admin db init OPENOPPS_DB_URL="sqlite:///$PWD/.tmp/openoppsdb-operational.sqlite" uv run openopps jobs sync --metrics-json --freshness-seconds 86400 --limit 120 PYTHONPATH=scripts uv run python -m openopps_kaggle --data-db .tmp/openoppsdb-operational.sqlite ``` The manager notebook should be scheduled in Kaggle as one daily full-snapshot attempt, for example `0 6 * * *`. Each run reads public snapshot input from `wyattowalsh/openoppsdb`, copies the verified `openopps_kaggle` runtime package from the private `wyattowalsh/openoppsdb-manager-runtime` input, copies the newest `/kaggle/input/**/openoppsdb.sqlite` ledger into `/kaggle/working/openoppsdb/openoppsdb.sqlite`, may restore large columns from prior Parquet exports when upgrading legacy thin snapshots, rehydrates the public SQLite snapshot into a fresh operational Alembic schema when needed, and then runs `openopps sync --metrics-json` (packaged catalog sources, boards, and jobs) under a 6000s budget. It writes private status and coverage evidence, runs `python -m openopps_kaggle` to backfill derived tables, export parquet-first artifacts, regenerate metadata, prune private evidence, and stage the exact public upload, then publishes and reads back one immutable version through the publication ledger. Timeout recovery is an explicit bounded partial-publication policy. The manager may continue only when the timed invocation completed at least one fresh, non-example, successful authoritative run with positive durable job evidence and matching observations. Reconstructed metrics retain provider failures and add `partial_sync_timeout`; without qualifying evidence, the timeout fails closed. Because the current quality policy can publish that degraded snapshot with warnings, `partial_sync_timeout` means incomplete route coverage and must not be interpreted as a full daily-crawl attestation. Keep the scheduled manager as the sole writer to its working SQLite copy: recovery uses an append-only rowid/time window, not a cross-process invocation identifier, and a concurrent sync against the same file can invalidate attribution. The manager has no mutable OpenOpps package default. Set `OPENOPPS_PACKAGE_SPEC` to `git+https://github.com/wyattowalsh/openopps.git@` as a Kaggle notebook secret or environment variable (or let `just kaggle-notebook-push execute=1` bake the current `HEAD` SHA into a temporary push copy); branches, tags, ranges, and unpinned packages fail before installation. The generated notebook also pins the canonical runtime-package digest through `OPENOPPS_RUNTIME_PACKAGE_SHA256` (currently `8444339907d9963a07ea2e3a5dec68fe9d1fab30dee184367c16e24c0b2638a4`), which must match the private runtime dataset's `runtime-manifest.json`. Regenerate the manager notebook and refresh the private runtime dataset together whenever that package changes. Scheduled rehydrate accepts a pre-`0004` public `job_sync_runs` table by inserting the lifecycle columns from `synced_at` / `success` / `error`. The snapshot quality gate does not hard-block empty derived skill tables when every `job_versions.skills` value is null or `[]`. The notebook seeds bounded Kaggle runtime defaults for source freshness, job-route freshness, concurrency, connection limits, timeouts, and retries while allowing `OPENOPPS_` environment overrides. Live file/column metadata repair is not part of dataset publication or immutable-version readback. After a verified live publish, run `just kaggle-live-file-metadata` separately from a browser-authenticated maintainer environment when the Kaggle DataBundle checklist or column-description score needs authoritative repair, and record its outcome independently. The scheduled Kaggle notebook environment must expose Kaggle API credentials before the manager starts. Configure `KAGGLE_USERNAME` and `KAGGLE_KEY`, or `KAGGLE_API_TOKEN`, as Kaggle notebook secrets/environment variables; otherwise the manager fails fast before running the expensive sync. Live deployment remains a local maintainer action with Kaggle CLI credentials. Every create, version, and kernel-push recipe is dry-run by default. Create/version recipes rebuild from `db=` before staging so a stale `kaggle/` tree cannot silently ship (`allow_stale=1` is a loud override only): ```bash # Review dry-run ledgers first. Replace 42/7 with the exact live versions. just kaggle-dataset-version message="OpenOppsDB daily snapshot" db=.tmp/openoppsdb-operational.sqlite expected_current_version=42 just kaggle-runtime-generator-version message="OpenOppsDB manager runtime generator" expected_current_version=7 just kaggle-notebook-push just kaggle-example-notebooks-push # Execute only after reviewing those plans. just kaggle-dataset-version message="OpenOppsDB daily snapshot" db=.tmp/openoppsdb-operational.sqlite expected_current_version=42 execute=1 just kaggle-runtime-generator-version message="OpenOppsDB manager runtime generator" expected_current_version=7 execute=1 just kaggle-notebook-push execute=1 just kaggle-example-notebooks-push execute=1 just kaggle-example-notebooks-status just kaggle-example-notebooks-pull-check just kaggle-live-verify ``` The public example notebooks are generated from the repo source of truth and pushed as separate Kaggle kernels: * `wyattowalsh/openoppsdb-starter-notebook` — front door (`%%sql` + Plotly) * `wyattowalsh/openoppsdb-explorer` — featured Gradio explorer (jobs, companies, skills, filters/plots) * `wyattowalsh/openoppsdb-advanced-usage` — joins, version history, company drill-down * `wyattowalsh/openoppsdb-sql-playground` — JupySQL / DuckDB studio * `wyattowalsh/openoppsdb-hiring-market-map` — company/provider/location charts * `wyattowalsh/openoppsdb-skills-radar` — skills and co-occurrence * `wyattowalsh/openoppsdb-snapshot-health` — coverage, freshness, observation mix They are read-only, credential-free, and attached only to `wyattowalsh/openoppsdb`. Kernels that need current JupySQL or Gradio enable internet for a pinned `pip install`; they still do not use credentials or publish CLIs. After pushing, run `just kaggle-example-notebooks-pull-check` to pull and verify the live source bundles; `just kaggle-example-notebooks-files page_size=200` lists output files emitted by those notebook runs. Use `just kaggle-dataset-create db=` only to prepare the first public upload and `just kaggle-runtime-generator-create` only to prepare the first private manager-runtime upload. After inspecting the dry-run ledger, a first create requires both `execute=1` and `allow_no_rollback=1`; this is the explicit acknowledgment that no immutable rollback target exists yet: ```bash just kaggle-dataset-create db=.tmp/openoppsdb-operational.sqlite just kaggle-dataset-create db=.tmp/openoppsdb-operational.sqlite execute=1 allow_no_rollback=1 just kaggle-runtime-generator-create just kaggle-runtime-generator-create execute=1 allow_no_rollback=1 ``` Subsequent version writes require `expected_current_version=` and `execute=1`; the live preflight must observe that exact version, and the ledger preserves it as the rollback target. The live write path stages a temporary upload directory, requires Kaggle CLI credentials, verifies the new immutable version by exact remote readback, and remains intentionally outside CI. `just kaggle-bundle-smoke` is the non-secret clean-DB stage smoke used for local/CI confidence. `just kaggle-notebook-push` defaults to a two-hour Kaggle kernel timeout (sync plus publication readback can exceed one hour) and renders only a plan unless `execute=1` is supplied. `just kaggle-example-notebooks-push` still defaults to one hour. Override `timeout` only for a deliberate longer maintenance run. ## CI/CD [#cicd] GitHub Actions mirrors local validation instead of inventing a second build path. CI should stay least-privilege, cache-aware, and split into independently debuggable lanes: * Python package tests and coverage through `uv`. * Lock-file consistency through `uv lock --check`. * OpenSpec strict validation for every active change. * Offline source-discovery schema, fixture, bundle, skill-eval, private-envelope, accounting, and promotion-preview gates with discovery network disabled. * Web generated data, MDX/type-check, build, tests, browser checks, and lint through `pnpm`. * Kaggle metadata generation for exported schema changes. * Dependency review on pull requests, supported Python 3.12/3.13/3.14 lanes, lowest-direct dependency testing, and network-dependent security audits. * A main-ref, non-pull-request supply-chain lane that builds one fresh hash-constrained Python wheel and one fresh sdist, rejects stale, oversized, unsafe, metadata-inconsistent, or mis-scoped contents, generates and semantically verifies one artifact-specific SPDX SBOM per distribution, attests both subjects separately, and retains the workflow evidence for 30 days. The SBOMs bind exact artifact/OpenOpps component identity; `uv.lock`, hash-locked installed-wheel smoke, and vulnerability audits provide dependency-closure evidence. `just build-release-artifacts` is the canonical local counterpart and refuses to reuse an existing `dist/`. The supply-chain lane does not build, attest, restore-test, or publish the v7 public-data recovery archive. That remains a separate production-readiness gate. The broader `rtk lint` docs checklist remains an explicit maintainer recipe (`just web-rtk-lint`) rather than a conditional CI step. When CI changes, update `Justfile`, this operations page, root README, and repo instructions in the same logical change. ## Troubleshooting [#troubleshooting] | Symptom | Check | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | A source sync returns zero boards | Run `uv run openopps admin sources test --page-size 5` and check provider health. | | A board has provider hints but no jobs | Run `admin providers probe-routes` and inspect unknown route candidates. | | A new source has low yield | Run `uv run openopps admin sources yield --source ` after route probing and job sync. | | Cached data looks stale | Rerun the operation with `--refresh-cache` or purge the relevant cache namespace. | | Local DB reports missing v0.1 columns | Reset the pre-release local DB file or set `OPENOPPS_DB_URL` to a new SQLite path, then rerun `admin db init`. | | Workday sync is slow | Lower `OPENOPPS_WORKDAY_CONCURRENCY` and narrow sync with `--source`, `--board`, or `--provider workday`. | | Export output is unexpectedly empty | Confirm the database has data with `uv run openopps admin db status` and inspect filters. | | Docs navigation is stale | Update `web/content/docs/meta.json`, then run `cd web && pnpm types:check`. | | `just source-policy-audit` exits 2 | Expected while 688 sources remain blocked pending written Getro/Consider grants. Structural `just source-policy-check` staying green is not eligibility. | | `discovery verify-scout` reports stale | Evaluation bundles use a 48-hour `max_evidence_age`. Re-scout; this is not a promotion rollback. | | A source-scout suggestion looks approved | The skill is advisory. Acceptance is only `launch_isolated_scout` / `validate_fixture.py`. Do not live-install harness projections. |