# CLI (/docs/cli) OpenOpps exposes the `openopps` console script from `openopps.cli:app`. Treat the CLI as a local data pipeline: source catalogs become durable board records, board records become executable provider routes, provider routes become normalized jobs, and exports preserve the resulting ledger for audit or analysis. ## Invocation [#invocation] Run through the repository environment during development: ```bash uv sync just --list uv run openopps --help uv run openopps status ``` Install the current checkout as an editable uv tool when you want to call `openopps` directly: ```bash uv tool install -e . openopps --help openopps status ``` The editable tool uses the same runtime settings as `uv run`, including `OPENOPPS_DB_URL`, plugin allow lists, cache settings, and current working-directory-relative SQLite paths. ## Workflow [#workflow] The normal sequence is: 1. Run `sync` for source discovery, board route resolution, and job sync in order. 2. Use `sources sync`, `boards sync`, or `jobs sync` when rerunning one stage. 3. Inspect `providers coverage`, `providers audit`, or `admin providers registry` when route metadata is incomplete. 4. Use `jobs list`, `boards export`, or `jobs export` for analysis. Use `status` or `doctor` between steps to see counts, route readiness, cache state, plugin load state, and the next recommended action. `openopps discovery` is not a step in this sequence. Scout, verify-scout, and preview-promotion are advanced admin commands and are not same-run with ingest (`openopps sync`, `sources sync`, `boards sync`, or `jobs sync`). ## Safety Classes [#safety-classes] | Class | Examples | What changes | | ----------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Read-only local inspection | `status`, `doctor`, `sources list`, `providers coverage`, `plugins list`, `cache status` | Reads local SQLite/catalog state only. | | Writes local SQLite state | `examples seed`, `sources sync`, `boards sync`, `jobs sync`, `admin boards add`, `admin db init` | Creates or updates records under `OPENOPPS_DB_URL` and cache settings. | | Live upstream diagnostics | `admin sources test`, `providers health`, `admin providers probe-routes` | Calls public source or provider endpoints; dry-run unless the command documents `--apply`. | | Quarantined discovery | `discovery scout`, `discovery verify-scout`, `discovery preview-promotion` (also `admin sources` aliases) | Scout writes only an explicit quarantine directory. Verify is offline and read-only. Preview is a read-only on-disk B699 identity-closure dry-run. No `--apply`. Not same-run with `sync`. | | Destructive local maintenance | `admin cache purge` | Deletes local cache records; scope with `--namespace` when possible. | `--apply` is intentionally absent from most everyday commands. When it appears on diagnostics such as route probing or health checks, treat it as the boundary between inspection and persistence. Quarantined discovery commands never expose `--apply`. ## Command Groups [#command-groups] | Surface | Commands | Purpose | | ----------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sync` | top-level | Run source discovery, board route resolution, and job sync in order. | | `status` | top-level | Report database, cache, plugin, route readiness, and next action. | | `doctor` | top-level | Same status payload with setup-oriented framing. | | `sources` | `list`, `show`, `sync` | Inspect aggregate discovery catalogs and import board records. | | `boards` | `sync`, `list`, `show`, `export` | Resolve, inspect, and export firm or company hiring board records. | | `jobs` | `sync`, `list`, `show`, `history`, `export` | Fetch, inspect, version, and export normalized public postings. | | `providers` | `health`, `coverage`, `audit` | Inspect live health, persisted coverage gaps, and adoption evidence. | | `cache` | `status` | Inspect the SQLite request cache. | | `plugins` | `list` | Inspect installed plugin entry points, capabilities, and failures. | | `examples` | `seed` | Seed deterministic synthetic demo data. | | `discovery` | `scout`, `verify-scout`, `preview-promotion` | Quarantined scout into an explicit output directory, offline bundle verify, and digest-bound promotion preview. Advanced admin; does not promote, sync, or activate candidates. | | `admin` | `sources`, `boards`, `providers`, `cache`, `db` | Advanced registration, quarantined scout aliases, route diagnostics, cache purge, and DB maintenance. | ## Admin inventory [#admin-inventory] Use these commands for registration, portable backups, and route metadata repair. They are grouped under `openopps admin` and are separate from everyday `sync` workflows. `admin sources add`, `test`, and `yield` stay the custom-catalog, adapter-sample, and persisted-yield commands. Quarantined scout, verify, and promotion preview are aliased under `admin sources` so OpenSpec command strings stay available; the primary group is `openopps discovery`. | Group | Command | Purpose | | ----------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------- | | `admin db` | `init` | Create or upgrade the local SQLite schema at `OPENOPPS_DB_URL`. | | `admin db` | `status` | JSON counts and path for the configured database. | | `admin db` | `export --output ` | Copy the local `sqlite:///` database to a portable SQLite file (requires a local SQLite URL). | | `admin db` | `vacuum` | Compact the configured SQLite database file. | | `admin boards` | `add-provider` | Attach explicit provider route metadata (token, hosted URL, or site fields) when probing did not resolve a board. | | `admin boards` | `detect-provider` | Detect provider metadata from one board URL; add `--apply` to persist. | | `admin providers` | `detect ` | Detect which packaged provider adapter matches a public board URL without a board scope. | | `admin providers` | `probe-routes` | Try route candidates and report boards that can fetch jobs; `--apply` persists matches. | | `admin providers` | `registry` | Inspect the durable `board_providers` route registry before job sync. | | `admin sources` | `scout --output ` | Same callback as `discovery scout`. Requires an explicit quarantine directory. No `--apply`. | | `admin sources` | `verify-scout ` | Same callback as `discovery verify-scout`. Offline verify of `manifest.json` or the bundle directory. | | `admin sources` | `preview-promotion [manifest]` | Same callback as `discovery preview-promotion`. Read-only on-disk B699 identity-closure dry-run. No `--apply`. | ```bash uv run openopps admin db export --output /tmp/openopps-backup.sqlite uv run openopps admin db vacuum uv run openopps admin boards add-provider a16z:example --provider ashbyhq --url 'https://jobs.ashbyhq.com/example' uv run openopps admin providers detect 'https://jobs.ashbyhq.com/example' ``` ## Quarantined source discovery [#quarantined-source-discovery] `openopps discovery` is the advanced-admin group for quarantined source discovery. Root help places it on the Advanced admin panel so everyday `sync`, `sources`, `boards`, and `jobs` stay primary. It does not promote, sync, or activate candidates, and it is not same-run with ingest (`openopps sync`) or with `sources sync` / `boards sync` / `jobs sync`. The same callbacks are aliased as `openopps admin sources scout`, `verify-scout`, and `preview-promotion`. None of these commands accept `--apply`. Scout requires `--output ` (`-o` / `-O`) and writes only there. `--json` (`-j` / `-J`) emits machine-readable JSON. There is no `--apply` option; passing `--apply` is rejected. Default unscoped `openopps sync` remains the snapshot writer and keeps using the last reviewed approved catalog. Ledger/L.2 snapshot tables are not landed. Scout output is not a snapshot and is not live publication evidence. ```bash uv run openopps discovery scout --output /absolute/quarantine-root --json uv run openopps discovery verify-scout /absolute/quarantine-root --json uv run openopps discovery preview-promotion --json uv run openopps admin sources scout --output /absolute/quarantine-root --json uv run openopps admin sources verify-scout /absolute/quarantine-root --json uv run openopps admin sources preview-promotion --json ``` | Command | Options and arguments | What it does | What it does not do | | ----------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `discovery scout` | `--output ` (required; `-o` / `-O`), `--json` (`-j` / `-J`) | Writes one evaluation quarantine bundle under the explicit directory. Selector-bound scout pins the private approved-ingestion envelope and does not accept the v7 public `SourceSelector`. | Mutate operational SQLite, catalogs, Git, Kaggle, or Cloudflare; accept `--apply`; run ingest or promotion in the same invocation. | | `discovery verify-scout` | `` (required path), `--json` (`-j` / `-J`) | Offline-verifies `manifest.json` or the bundle directory that contains it. Re-validates the private approved-ingestion envelope. | Rewrite or repair the bundle; activate candidates; accept `--apply`; run ingest in the same invocation. | | `discovery preview-promotion` | optional `[manifest]`, `--json` (`-j` / `-J`) | Dry-run a digest-bound repository promotion preview without applying. Omit the manifest to preview the on-disk B699 identity-closure envelope, decision, receipt, and ledger. Pass a quarantine manifest to offline-verify that bundle first, then preview an empty candidate selection bound to the verified digest. | Reserve, apply, acquire the promotion lock, or grant authority. Does not mutate Git remotes, operational SQLite, Kaggle, Cloudflare, or the catalog. There is no `--apply` option. | The current CLI scout publishes a bundle from an empty occurrence set against read-only v7 policy digests. Channel enumerators (`official`, `public_code`, `search`, `targeted_ats`) are replay-library surfaces, not a live crawl from this command. Public CI keeps `OPENOPPS_DISCOVERY_NETWORK=disabled`. Isolated scout limits live under `OPENOPPS_DISCOVERY_*` and do not change `openopps sync`; see [Configuration](/docs/configuration#isolated-discovery-scout). Scout JSON includes `activated=false` and `promoted=false`. Preview JSON includes `applied=false` and `grantsAuthority=false`. Omitting the preview manifest sets `identityClosure=true` for the on-disk B699 identity-closure path. Treat those fields as evidence, not as an apply path. ## Job inspection [#job-inspection] `jobs show` prints one normalized job record as JSON, including current lifecycle fields and enriched metadata. Use the job `id` from `jobs list --json`. `jobs history` lists normalized **content versions** for the same posting id (`job_versions`), not raw provider payload snapshots. Payload-only drift can update `job_payload_snapshots` and sync observations without creating a new content version. Add `--json` for automation; the default table view shows version, content hash prefix, first/last seen timestamps, and title per version. ```bash uv run openopps jobs show '' uv run openopps jobs history '' --json ``` ## Option Conventions [#option-conventions] Examples use long flags for readability. The CLI also exposes script-friendly short aliases for high-traffic options: | Long flag | Aliases | Used by | | ----------------- | ---------- | -------------------------------------------------------------------------------- | | `--source` | `-s`, `-S` | Source, board, job, provider, and audit scopes. | | `--board` | `-b`, `-B` | Job sync/list/export and route diagnostics. | | `--provider` | `-p`, `-P` | Board, job, provider, and route scopes. | | `--limit` | `-n`, `-N` | List and diagnostic result limits. | | `--json` | `-j`, `-J` | Machine-readable command output, including discovery scout, verify, and preview. | | `--output` | `-o`, `-O` | Export or no-DB output paths, and the required scout quarantine directory. | | `--format` | `-f`, `-F` | Export format selection. | | `--metrics-json` | `-m`, `-M` | Sync metrics output. | | `--refresh-cache` | `-r`, `-R` | Fresh upstream reads for cacheable request paths. | | `--verbose` | `-v`, `-V` | Detailed sync warnings instead of compact progress. | `--provider any` and `--provider all` both remove the provider filter. They are useful in reusable scripts that always pass a provider argument, but they do not mean “only providers named any/all.” ## Common Commands [#common-commands] ```bash uv run openopps status --json uv run openopps doctor --json uv run openopps sync a16z --metrics-json --refresh-cache uv run openopps sources sync a16z --metrics-json uv run openopps boards sync --source a16z --provider any --metrics-json uv run openopps jobs sync --provider ashbyhq --metrics-json --refresh-cache uv run openopps providers coverage --source a16z --provider any --json uv run openopps admin providers probe-routes --source a16z --provider any --limit 25 --json ``` Provider hints from source catalogs may lack the token or URL needed for job fetching. `admin providers probe-routes` tries candidate tokens from upstream slugs, remote ids, names, domains, and websites, then reports both matched routes and unknown boards. It is read-only unless `--apply` is passed. ## Board Filters [#board-filters] `boards list` and `boards export` share these filters: | Flag | Semantics | | ------------- | -------------------------------------------------------------------------- | | `--source` | Exact source key, such as `a16z` or `yc`. | | `--provider` | Exact detected board-provider route id. `any` and `all` remove the filter. | | `--market` | Case-insensitive substring match against board market tags. | | `--location` | Case-insensitive substring match against normalized board locations. | | `--domain` | Case-insensitive substring match against normalized board domains. | | `--has-jobs` | Keep boards with a source job hint, provider job hint, or synced job. | | `--min-staff` | Keep boards with `staff_count` greater than or equal to the value. | | `--max-staff` | Keep boards with `staff_count` less than or equal to the value. | | `--limit` | Apply a final limit after filters. | | `--json` | JSON output mode for `boards list`. | Use the exact persisted board key shown by `boards list`; provider requests are deduped before probing or job sync when overlapping source coverage points at the same provider route. ## Job Filters [#job-filters] `jobs list` and `jobs export` default to current active jobs with `--status open` across all boards and providers. Use `--status closed` or `--status all` for lifecycle audits. | Flag | Semantics | | ----------------------------- | ---------------------------------------------------------------------------------------------- | | `--source` | Exact source key via the job's board record. | | `--board` | Exact persisted board key from `boards list`. | | `--provider` | Exact job provider id. `any` and `all` remove the filter. | | `--location` | Case-insensitive substring match against normalized job locations. | | `--department` | Case-insensitive substring match against normalized department. | | `--team` | Case-insensitive substring match against normalized team. | | `--workplace-type` | Case-insensitive substring match against normalized workplace type. | | `--remote` | Case-insensitive exact match against normalized remote level: `Full`, `Hybrid`, or `None`. | | `--employment-type`, `--type` | Case-insensitive substring match against normalized employment type. | | `--salary-min` | Keep jobs whose normalized salary range overlaps this lower bound. | | `--salary-max` | Keep jobs whose normalized salary range overlaps this upper bound. | | `--skill` | Case-insensitive substring match against normalized skill name, level, or keywords. | | `--query` | Case-insensitive substring match across normalized title, company, and plain-text description. | | `--posted-after` | Inclusive `YYYY-MM-DD` lower bound for normalized `posted_at` dates. | | `--posted-before` | Inclusive `YYYY-MM-DD` upper bound for normalized `posted_at` dates. | | `--status` | Lifecycle filter: `open`, `closed`, or `all`. Defaults to `open`. | | `--limit` | Apply a final limit after filters. | | `--json` | JSON output mode for `jobs list`. | Date filters intentionally only match jobs whose normalized `posted_at` starts with `YYYY-MM-DD`, such as ISO timestamps. Relative provider text such as `Posted Yesterday` is not used for public filtering semantics. ## Exports [#exports] ```bash uv run openopps boards export --provider ashbyhq --has-jobs --format csv --output /tmp/openopps-boards.csv uv run openopps jobs export --source a16z --type full --format parquet --output /tmp/openopps-jobs.parquet ``` `boards export` and `jobs export` support: | Format | Use case | | --------- | ----------------------------------------------------------------------------------- | | `jsonl` | Streaming and audit-friendly line-delimited records. | | `csv` | Spreadsheet inspection and lightweight exchange. | | `parquet` | Analytics workflows with Polars, DuckDB, or warehouse ingestion. | | `sqlite` | Local relational handoff, reproducible filtered extracts, or direct SQLite clients. | JSONL exports stream line-delimited records and empty JSONL/CSV exports produce empty files. Empty Parquet exports produce a readable empty Parquet table. CSV exports neutralize spreadsheet formula-leading strings by prefixing a single quote. SQLite exports should keep the same flattened field contract as CSV/Parquet and store nested values as stable JSON strings. For schema, SQLite, and search-index details, see [Data Model](/docs/data-model). ## Troubleshooting Map [#troubleshooting-map] | Symptom | Command to run first | | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | Local state looks empty | `uv run openopps status --json` | | A source returns no boards | `uv run openopps admin sources test --page-size 5 --refresh-cache` | | A board has provider hints but no jobs | `uv run openopps admin providers registry --include-missing --json` | | Route metadata is missing | `uv run openopps admin providers probe-routes --source --provider any --limit 25 --json` | | Cached data looks stale | Rerun the command with `--refresh-cache` or purge a specific namespace. | | Need a quarantined candidate bundle | `uv run openopps discovery scout --output --json` then `discovery verify-scout` (or the `admin sources` aliases) | | Need a promotion dry-run | `uv run openopps discovery preview-promotion --json` (on-disk B699 identity closure; no `--apply`) | | Export output is empty | Check `status`, then rerun the matching `list` command with the same filters and `--json`. | # Configuration (/docs/configuration) OpenOpps loads settings from `OPENOPPS_` environment variables and an optional local `.env` file. Unknown `.env` keys are ignored. Treat configuration as local runtime wiring: the CLI writes to the SQLite URL you point it at and reads plugins from the Python environment that launched the command. ## Invocation Scope [#invocation-scope] Both invocation styles use the same settings model: ```bash uv run openopps status uv tool install -e . openopps status ``` The editable `uv tool install -e .` path makes the `openopps` command available directly from the current checkout. Relative SQLite URLs such as `sqlite:///openoppsdb.sqlite` are resolved by the running process, so keep your working directory and `OPENOPPS_DB_URL` explicit when switching between projects or smoke-test databases. ## Environment Variables [#environment-variables] | Variable | Default | Purpose | | -------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `OPENOPPS_DB_URL` | `sqlite:///openoppsdb.sqlite` | Database URL used by storage commands and sync jobs. | | `OPENOPPS_MAX_CONNECTIONS` | `40` | Maximum HTTP connection pool size. | | `OPENOPPS_SOURCE_CONCURRENCY` | `4` | Source adapter concurrency during source sync and provider health checks, not the isolated scout. | | `OPENOPPS_SOURCE_TIMEOUT_SECONDS` | `900.0` | Maximum seconds one source adapter can run before timed-out skip. | | `OPENOPPS_SOURCE_FRESHNESS_SECONDS` | `0.0` | Skip recently synced source catalogs during unscoped full-sync retries. | | `OPENOPPS_BOARD_CONCURRENCY` | `16` | Concurrent ready board routes and board-scoped listing/detail work during job sync, route health checks, and providers that fan out across job pages. | | `OPENOPPS_JOB_ROUTE_TIMEOUT_SECONDS` | `180.0` | Maximum seconds one executable provider route may run during job sync before OpenOpps records a timeout and continues with remaining routes. | | `OPENOPPS_JOB_ROUTE_FRESHNESS_SECONDS` | `0.0` | Skip recently synced provider routes during job sync when above `0`; `0` refreshes every selected route. | | `OPENOPPS_JOB_ROUTE_LIMIT` | unset | Cap stale or never-synced routes processed in one job sync; unset processes every selected route. | | `OPENOPPS_PROVIDER_CONCURRENCY` | `12` | Concurrent provider route probes when OpenOpps detects executable job-board routes (for example during `admin providers probe-routes`). Not job-fetch parallelism. | | `OPENOPPS_WORKDAY_CONCURRENCY` | `2` | Conservative limit for public Workday CXS careers-site requests. | | `OPENOPPS_DB_BATCH_SIZE` | `500` | Batch size for SQLite writes. | | `OPENOPPS_HTTP_TIMEOUT` | `30.0` | HTTP timeout in seconds. | | `OPENOPPS_RETRY_ATTEMPTS` | `3` | Number of retry attempts for retriable upstream requests. | | `OPENOPPS_USER_AGENT` | `openopps/0.1 openopps@wyattowalsh.com` | User agent sent to public source and provider endpoints. | | `OPENOPPS_CACHE_ENABLED` | `true` | Enables the shared JSON request cache in the SQLite database. | | `OPENOPPS_CACHE_TTL_SECONDS` | `3600` | Default cache freshness window in seconds. | | `OPENOPPS_CACHE_REFRESH` | `false` | Bypasses cache reads while still updating successful responses. | | `OPENOPPS_CACHE_STALE_ON_ERROR` | `false` | Allows eligible stale cache data after retryable upstream errors. | | `OPENOPPS_PLUGIN_AUTOLOAD` | `false` | Execute every discovered plugin entry point without allow-listing. | | `OPENOPPS_PLUGIN_DISABLED` | empty | Comma-separated plugin entry-point names to skip. | | `OPENOPPS_PLUGIN_ALLOWED` | empty | Comma-separated plugin entry-point names allowed to execute. | | `OPENOPPS_NO_INTRO` | unset | Suppress the interactive startup portal animation. | ### Isolated discovery scout [#isolated-discovery-scout] The isolated scout uses a separate settings model (`DiscoverySettings` in `src/openopps/discovery/settings.py`). `OPENOPPS_DISCOVERY_*` budget fields are not part of `OpenOppsSettings`, do not load `.env`, and do not change `openopps sync`. Values must be canonical positive decimals. Invalid values fail closed without echoing the input. Scout output is an explicit directory, not the operational database. `openopps discovery scout --output /absolute/quarantine-root --json` (also `openopps admin sources scout`) writes one evaluation quarantine bundle under that required `--output` path. There is no scout-output environment variable. `OPENOPPS_DB_URL` still names the CLI SQLite ledger for sync, cache, and `admin db` commands; the scout does not open or mutate that file, catalogs, Git, Kaggle, or Cloudflare. | Variable | Default | Purpose | | ------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------- | | `OPENOPPS_DISCOVERY_WHOLE_RUN_TIMEOUT_SECONDS` | `300` | Whole isolated invocation wall-clock limit (max `3600`). | | `OPENOPPS_DISCOVERY_CHANNEL_TIMEOUT_SECONDS` | `120` | Per-channel wall-clock limit (max `1800`). | | `OPENOPPS_DISCOVERY_CHANNEL_QUERY_LIMIT` | `20` | Trusted predeclared queries per channel (max `1000`). | | `OPENOPPS_DISCOVERY_CHANNEL_REQUEST_LIMIT` | `100` | Requests including retries, redirects, and pagination per channel (max `5000`). | | `OPENOPPS_DISCOVERY_ORIGIN_LIMIT` | `25` | Distinct validated public HTTPS origins per channel (max `500`). | | `OPENOPPS_DISCOVERY_REDIRECT_LIMIT` | `5` | Manually validated redirect hops per logical request (max `10`). | | `OPENOPPS_DISCOVERY_PER_HOST_CONCURRENCY` | `2` | Simultaneous connections per origin (max `16`). | | `OPENOPPS_DISCOVERY_OVERALL_CONCURRENCY` | `8` | Simultaneous requests across channels (max `64`). | | `OPENOPPS_DISCOVERY_RESPONSE_MAX_BYTES` | `1048576` | Decoded bytes admitted from one response (max `10485760`). | | `OPENOPPS_DISCOVERY_AGGREGATE_RESPONSE_MAX_BYTES` | `67108864` | Decoded bytes admitted across the invocation (max `268435456`). | | `OPENOPPS_DISCOVERY_CANDIDATE_LIMIT` | `1000` | Candidate occurrences per channel before remaining work is unstarted (max `10000`). | | `OPENOPPS_DISCOVERY_RETRY_ATTEMPT_LIMIT` | `3` | Total attempts per logical request (max `10`). | | `OPENOPPS_DISCOVERY_PAGINATION_LIMIT` | `20` | Pagination requests per logical enumeration (max `1000`). | | `OPENOPPS_DISCOVERY_PARSER_MAX_DEPTH` | `32` | Trusted structural nesting depth (max `128`). | | `OPENOPPS_DISCOVERY_EVIDENCE_RETENTION_SECONDS` | `86400` | Maximum age for exact verified quarantine evidence to be reusable by a later scout (max `604800`). | | Variable | Public/CI/offline value | Purpose | | ---------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | `OPENOPPS_DISCOVERY_NETWORK` | `disabled` | Gate flag for public CI, `just ci-discovery` / `just source-discovery-*`, and `scripts/source_discovery_gates.py`. Not a `DiscoverySettings` field. | Public CI (the `discovery` job in `.github/workflows/ci.yml`), local offline recipes, and other CI-shaped runs require `OPENOPPS_DISCOVERY_NETWORK=disabled`. The gate refuses any other value; unset defaults to `disabled`. Public CI replays committed sanitized fixtures and does not run a live scout. A live scout schedule is a separate, unexercised maintainer authority gate. This page does not provision GitHub Actions `schedule:`, Cloudflare Cron, hosted runners, or a private live-network profile, and it does not document a live-scheduler environment as if one were enabled. Remote content cannot raise a trusted limit. The scout does not use the runtime SQLite HTTP cache or stale-on-error behavior. See [CLI](/docs/cli#quarantined-source-discovery) for command strings. ### Web app public data release [#web-app-public-data-release] These variables configure the Fumadocs/Next.js host when it loads public search artifacts. They are **not** part of the CLI `OpenOppsSettings` model. Server and browser values should select the same v7 origin and channel; the `NEXT_PUBLIC_` values are embedded into the browser build. | Variable | Default | Purpose | | -------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `OPENOPPS_PUBLIC_DATA_ORIGIN` | site URL | Trusted origin for server-side snapshot reads (never the request Host). Must be `https://` in production unless the insecure override is explicit. | | `OPENOPPS_PUBLIC_DATA_CHANNEL` | unset | Safe lowercase v7 channel, normally `production`. When unset, server consumers use the bounded v6 transition reader. | | `OPENOPPS_PUBLIC_DATA_ORIGIN_ALLOW_INSECURE` | unset (`1` allows) | Allow non-HTTPS origins. Implicitly allowed in development and Vercel preview. | | `OPENOPPS_PUBLIC_DATA_ORIGIN_ALLOW_HOSTS` | empty | Comma-separated extra hostnames allowlisted in production (alongside the configured site host and `VERCEL_URL`). | | `NEXT_PUBLIC_OPENOPPS_PUBLIC_DATA_ORIGIN` | unset | Browser-visible v7 publication origin used by Jobs/Explorer and its search worker. | | `NEXT_PUBLIC_OPENOPPS_PUBLIC_DATA_CHANNEL` | unset | Browser-visible safe lowercase v7 channel; normally the same value as `OPENOPPS_PUBLIC_DATA_CHANNEL`. | When neither channel is set, the web app reads the committed v6 transition tree. When v7 is enabled, both browser and server clients resolve the channel once, validate the manifest, pin its immutable release, and verify each asset before use. Production origins require HTTPS. Server-side production origins must also pass the explicit hostname allowlist; add the Workers hostname to `OPENOPPS_PUBLIC_DATA_ORIGIN_ALLOW_HOSTS` when it differs from the site host. ```dotenv OPENOPPS_PUBLIC_DATA_ORIGIN=https://openopps-data-production..workers.dev OPENOPPS_PUBLIC_DATA_CHANNEL=production OPENOPPS_PUBLIC_DATA_ORIGIN_ALLOW_HOSTS=openopps-data-production..workers.dev NEXT_PUBLIC_OPENOPPS_PUBLIC_DATA_ORIGIN=https://openopps-data-production..workers.dev NEXT_PUBLIC_OPENOPPS_PUBLIC_DATA_CHANNEL=production ``` Local `next start` / Playwright e2e may use an explicitly allowed loopback origin. Do not enable `OPENOPPS_PUBLIC_DATA_ORIGIN_ALLOW_INSECURE=1` in production merely to bypass a configuration error. ## Examples [#examples] ```bash OPENOPPS_DB_URL=sqlite:///data/openopps.db uv run openopps admin db status OPENOPPS_BOARD_CONCURRENCY=8 OPENOPPS_JOB_ROUTE_LIMIT=100 uv run openopps jobs sync --provider workday --metrics-json OPENOPPS_PROVIDER_CONCURRENCY=4 uv run openopps admin providers probe-routes --source a16z --provider any --limit 25 --json OPENOPPS_USER_AGENT='openopps/0.1 (contact: jobs@example.com)' uv run openopps admin sources test yc OPENOPPS_DISCOVERY_NETWORK=disabled uv run python scripts/source_discovery_gates.py ci uv run openopps discovery scout --output /absolute/quarantine-root --json ``` For repeated local use from the repository checkout, place values in `.env` at the repository root: ```dotenv OPENOPPS_DB_URL=sqlite:///openoppsdb.sqlite OPENOPPS_SOURCE_CONCURRENCY=4 OPENOPPS_SOURCE_TIMEOUT_SECONDS=900 OPENOPPS_SOURCE_FRESHNESS_SECONDS=0 OPENOPPS_BOARD_CONCURRENCY=16 OPENOPPS_JOB_ROUTE_TIMEOUT_SECONDS=180 OPENOPPS_JOB_ROUTE_FRESHNESS_SECONDS=0 OPENOPPS_PROVIDER_CONCURRENCY=12 OPENOPPS_CACHE_TTL_SECONDS=3600 OPENOPPS_PLUGIN_ALLOWED=trusted-plugin OPENOPPS_PLUGIN_DISABLED=broken-plugin ``` `DiscoverySettings` ignores `.env`; do not put `OPENOPPS_DISCOVERY_*` or `OPENOPPS_DISCOVERY_NETWORK` in that file. Unscoped `jobs sync` targets all persisted ready routes. Use explicit CLI filters such as `--source`, `--board`, or `--provider` for one-off scoped syncs. The [Jobs](/) and [Explorer](/explorer) surfaces use the committed v6 snapshot only while no v7 channel is configured. Refresh that transition artifact with `just web-search-index` after updating the local `kaggle/openoppsdb.sqlite` export. See [Public Data Releases](/docs/public-data-releases) for v7 configuration, governance, delivery, and cutover gates. Docs-app telemetry is configured separately from CLI runtime settings. When instrumentation is enabled, keep collection env-gated: default to a no-op sink in local development, use `local-event-lake` for canonical raw events on persistent disk, and set `OPENOPPS_POSTHOG_PROJECT_API_KEY` only when a sanitized hosted product-analytics mirror is desired. Browser PostHog replay is a separate opt-in through `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY`; it remains gated by `NEXT_PUBLIC_OPENOPPS_TELEMETRY_ENABLED=true`, disables automatic PostHog pageview/autocapture events, masks text and inputs, disables network body/header capture, and leaves replay sampling plus trigger controls to the PostHog project. See [Data Model](/docs/data-model#telemetry-event-lake) for the event shape and [Operations](/docs/operations#telemetry-operations) for operating guidance. ## Flags vs Persistent Settings [#flags-vs-persistent-settings] Prefer flags for one run and environment variables for durable local policy: | Need | Prefer | | ---------------------------------------------- | -------------------------------------------- | | Refresh one upstream read | `--refresh-cache` | | Always bypass cache reads in a shell/session | `OPENOPPS_CACHE_REFRESH=true` | | Narrow one job sync | `--source`, `--board`, or `--provider` | | Lower Workday pressure across all runs | `OPENOPPS_WORKDAY_CONCURRENCY=1` or `2` | | Cap one unscoped job sync batch | `OPENOPPS_JOB_ROUTE_LIMIT=500` | | Skip recently synced routes in job sync | `OPENOPPS_JOB_ROUTE_FRESHNESS_SECONDS=86400` | | Lower concurrent job-route execution | `OPENOPPS_BOARD_CONCURRENCY=8` | | Lower concurrent route probing | `OPENOPPS_PROVIDER_CONCURRENCY=6` | | Hide the animation once | `--no-intro` | | Hide the animation everywhere | `OPENOPPS_NO_INTRO=1` | | Bound one isolated scout | `OPENOPPS_DISCOVERY_*` | | Write scout output | `--output ` (not `OPENOPPS_DB_URL`) | | Keep public/CI/offline discovery gates offline | `OPENOPPS_DISCOVERY_NETWORK=disabled` | By default, discovered plugins are visible but not executed. Set `OPENOPPS_PLUGIN_ALLOWED` to run specific trusted entry points. If `OPENOPPS_PLUGIN_AUTOLOAD=true`, every discovered plugin is eligible unless its entry-point name appears in `OPENOPPS_PLUGIN_DISABLED`. Disabled entries are still skipped even if also allow-listed. For CLI one-offs, prefer command flags such as `--source` and `--refresh-cache` over persistent environment changes. Do not commit `.env` files with private or environment-specific values. # Contributing (/docs/contributing) OpenOpps changes should keep the CLI, web app, generated artifacts, OpenSpec, and validation recipes aligned. The project is pre-release, but the local workflow should still be reproducible and reviewable. **Package vs URL:** The Next/Fumadocs package lives under `web/` in the repository. Public docs routes stay under `/docs/*` (for example `/docs/contributing`). ## Local Setup [#local-setup] ```bash uv sync just --list uv run openopps --help cd web && pnpm install ``` Use `uv run openopps ...` inside the repository checkout. Install the editable tool only when you want the `openopps` command available directly: ```bash uv tool install -e . openopps status ``` ## Validation [#validation] Use `just` from the repository root for local parity with GitHub Actions. Prefer the canonical **`web-*`** recipes; transitional **`docs-*`** aliases invoke the same `web-*` recipes. ```bash just quick just ci just ci-discovery just lock-check just openspec-validate-all just web-check just web-test just cli-help ``` The underlying commands remain direct and scriptable: ```bash uv run pytest uv run pytest --cov=openopps --cov-report=term-missing uv lock --check rtk npx -y @fission-ai/openspec@1.6.0 validate --all --strict cd web && pnpm types:check cd web && pnpm build cd web && pnpm lint cd web && pnpm test just web-search-index-check ``` `just ci` composes the `ci-python`, `ci-openspec`, `ci-discovery`, `ci-web`, and `ci-artifacts` lanes; `just ci-full` adds network-dependent security audits and lowest-direct dependency testing. The discovery lane is offline only (`OPENOPPS_DISCOVERY_NETWORK=disabled`); see [Quarantined discovery and the source-scout skill](#quarantined-discovery-and-the-source-scout-skill). `just web-rtk-lint` is the explicit optional maintainer lint for `rtk` and is not part of the default CI recipe. GitHub Actions adds the supported Python matrix, dependency review, and a non-PR wheel/SBOM attestation job. The current supply-chain job attests the Python wheel only, not a v7 public-data recovery archive. ## Web App Workflow [#web-app-workflow] Docs content lives in `web/content/docs/*.mdx`, and navigation order lives in `web/content/docs/meta.json`. ```bash cd web pnpm data:generate pnpm types:check pnpm build pnpm lint pnpm test ``` `pnpm data:generate` refreshes package-derived source/provider/export metadata. `pnpm types:check` also regenerates that metadata before Fumadocs MDX artifacts, Next.js route types, and TypeScript checks. Use `just web-build` from the repository root for production web build assurance; it also runs the API function trace check. The static jobs/explorer index is separate because regeneration requires a **clean** local public `kaggle/openoppsdb.sqlite` snapshot (ignored by git). Recipes fail loud if that file is missing: ```bash just web-search-index just web-search-index-check ``` CI never opens SQLite. It validates the **committed v6 transition** artifact graph with `just web-search-artifacts-check` and the schema check inside `just web-check` (`pytest -k committed`). Run `just web-search-index-check` only when intentionally regenerating `web/public/data/openopps-search/` from a clean local snapshot. Version 7 generation writes to a separate publication root and is fail-closed on freshness, source rights, required attribution, privacy, exact closure, provenance, and platform budgets: ```bash uv run python scripts/generate_docs_search_index.py \ --data-db kaggle/openoppsdb.sqlite \ --release-root /absolute/path/to/openopps-search-v7 \ --channel production \ --max-snapshot-age-hours 48 uv run python scripts/verify_docs_search_artifacts.py \ --root /absolute/path/to/openopps-search-v7 \ --channel production \ --max-snapshot-age-hours 48 uv run python scripts/docs_search_delivery.py \ validate-config deployment/openopps-data ``` See [Public Data Releases](/docs/public-data-releases) before changing artifact schema, rights metadata, public-data environment variables, the shared snapshot client, search worker, assets-only configs, archive contents, or v6 compatibility. Live upload/deploy, GitHub Release publication, v6 removal, and Git history rewriting are separate authority boundaries; a local green run does not authorize or prove them. ## Public Workflow Changes [#public-workflow-changes] Use OpenSpec for non-trivial changes to public workflows, generated asset formats, downstream agent tooling, docs generation, or validation behavior. Pin `@fission-ai/openspec@1.6.0` in copy-paste commands (not `@latest`); set `OPENOPPS_OPENSPEC` to align `just openspec-*` with the same pin. ```bash rtk npx -y @fission-ai/openspec@1.6.0 list --json rtk npx -y @fission-ai/openspec@1.6.0 validate --all --strict ``` When commands, workflows, or generated surfaces change, update the relevant MDX page, root README, nested `AGENTS.md`, `Justfile`, CI workflow, and OpenSpec change in the same logical workstream. ## Quarantined discovery and the source-scout skill [#quarantined-discovery-and-the-source-scout-skill] Source discovery is CLI-first and fail-closed. Use `openopps discovery scout|verify-scout|preview-promotion` (or the `admin sources` aliases). Do not add `--apply`, TUI, browser, or hosted-service flows. ### Local offline gates [#local-offline-gates] Contributor and public CI runs stay offline. The Just recipes are thin wrappers around `scripts/source_discovery_gates.py` and set `OPENOPPS_DISCOVERY_NETWORK=disabled`. Prefer the canonical graph; use a named recipe when you need a single gate: ```bash just ci-discovery just source-discovery-schema-check just source-discovery-fixtures-check just source-discovery-manifest-check manifest= just source-discovery-promotion-preview just source-discovery-private-envelope-check just source-discovery-accounting-check just source-discovery-benchmark-check just source-discovery-skill-eval-check ``` `just ci-discovery` is an alias of `just source-discovery-ci` and is the same offline graph GitHub Actions runs. `source-discovery-manifest-check` requires a quarantine manifest path and does not rewrite or activate it. `source-discovery-promotion-preview` is a digest-bound dry-run; omit `manifest=` to preview the on-disk identity closure. None of these recipes apply, upload, or open a live network path. ### Tests [#tests] Focused discovery tests live under `tests/unit/openopps/discovery/` and `tests/unit/openopps/test_discovery_cli.py`: ```bash uv run pytest tests/unit/openopps/discovery tests/unit/openopps/test_discovery_cli.py -q ``` ### Source-scout skill [#source-scout-skill] The skill SSOT at `skills/openopps-source-scout/` is **inert** and **advisory**. Skill prose does not confine tools already authorized in Codex, Cursor, or Grok Build, and a suggestion is never approval, policy permission, review, promotion, or runtime activation. Do not run `wagents --apply` or any live harness install in the contributor flow. Acceptance is only through the deterministic isolated validator `openopps.discovery.isolation.launch_isolated_scout` (via `skills/openopps-source-scout/scripts/validate_fixture.py` for committed fixtures). Do not live-install harness projections under `.agents/skills/` or `.cursor/skills/`. Selected Codex/Cursor copies must remain absent; Grok has no repository projection. Read-only skill helpers: ```bash uv run python skills/openopps-source-scout/scripts/validate_evals.py uv run python skills/openopps-source-scout/scripts/validate_frontmatter.py uv run python skills/openopps-source-scout/scripts/dry_run_projection.py uv run python skills/openopps-source-scout/scripts/resolve_docs_steward.py uv run pytest tests/unit/openopps/discovery/test_source_scout_skill.py tests/unit/openopps/test_discovery_cli.py -q ``` `resolve_docs_steward.py` searches for docs-steward with `uv run wagents skills search docs-steward --json` and skips when `wagents` is absent. Do not install `wagents`, run `wagents --apply`, or run a live skill install from this lane. Agent-fabricated `approved`, reviewer, signature, receipt, or promotion fields are rejected before evaluation. ### Public CI [#public-ci] Public CI stays offline. The discovery job in `.github/workflows/ci.yml` sets `OPENOPPS_DISCOVERY_NETWORK=disabled`, checks out with `persist-credentials: false`, and runs `just ci-discovery`. Do not add a live-scout `schedule:` trigger (or any live network dispatch) to that workflow. Contributor discovery work does not provision private schedulers, mutate Kaggle, or upload Workers. Those remain separate unexercised maintainer authority gates. Live scheduler provisioning, credential selection, activation, retention, and execution are separate unexercised authority gates. A local green `just ci-discovery` does not authorize a live scout, catalog apply, dataset publish, or deploy. When discovery commands, schemas, or the skill boundary change, update README, this page, [CLI](/docs/cli), [Operations](/docs/operations), [Configuration](/docs/configuration), [Providers](/docs/providers), and nested `AGENTS.md` in the same workstream. Do not regenerate `web/lib/generated/openopps-data.json` from this docs lane. ## Source and Provider Changes [#source-and-provider-changes] Source adapters discover candidate company boards. Provider adapters detect or fetch postings from public job-board providers. * Keep source adapters low-side-effect and explicit about upstream access. * Preserve source provenance in durable board records. * Keep route probing dry-run-first; persist with `--apply` only after matched routes are reviewed. * Add semantic tests for provider support and normalized output. * Use `providers coverage`, `providers audit`, and `admin sources yield` to evaluate persisted coverage before changing public claims. * Keep quarantined discovery off the ingest path: scout/verify/preview never share a run with `openopps sync`, and they have no `--apply` option. Installed Python plugins are not sandboxed and run in the same process as OpenOpps. Use `OPENOPPS_PLUGIN_ALLOWED` for trusted plugin entry points and `OPENOPPS_PLUGIN_AUTOLOAD=true` only in controlled environments. ## Data and Telemetry Contributions [#data-and-telemetry-contributions] Export and static-index changes must keep the data contract clear: * Update [Data Model](/docs/data-model) when entities, export formats, search-index fields, facets, suggestions, or telemetry events change. * Prefer generated counts and generated manifests over copied prose counts. * Keep SQLite, CSV, Parquet, and JSONL export semantics aligned. * Keep telemetry first-party, env-gated, size-capped, and sanitized. * Treat the local event lake as the canonical telemetry sink; optional dashboards or hosted adapters are mirrors. * Keep packaged source rights canonical for packaged sources. Missing or `needs_review` rights states fail v7 publication; required attribution must be present in the generated policy report. * Never hand-edit an immutable v7 release, channel pointer, policy report, or recovery archive. Correct the source of truth and regenerate. ## Secret Hygiene [#secret-hygiene] Keep credentials local. `.env`, `.env.*`, `.envrc`, Kaggle `kaggle.json`, local package-registry credential files, `.netrc`, key bundles, and token or credential JSON files are ignored; `.env.example` remains the tracked non-secret template. Do not print credentials in logs, docs, CI output, generated artifacts, or screenshots. Live Kaggle publishing remains a maintainer-only local action and is intentionally outside CI. # Data Model (/docs/data-model) OpenOpps keeps the operational ledger in local SQLite, publishes portable extracts for analysis, and serves the web Jobs/Explorer surfaces from static artifacts. Version 6 is the committed transition tree; version 7 is a separately delivered content-addressed publication. Use this page when you need to reason about data boundaries, counts, exports, or telemetry. ## Core Entities [#core-entities] | Entity | Meaning | Primary producer | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- | | `sources` | Aggregate discovery catalogs such as `a16z`, `accel`, `yc`, public-company indices, or ecosystem landscapes. | Packaged source adapters and plugins. | | `boards` | Firm or company hiring boards discovered from sources. | `sources sync`, `boards sync`, and admin board commands. | | `board_providers` | Provider route metadata between a board and a public job adapter. | Provider detection, route probing, and board sync. | | `jobs` | Stable posting identity and lifecycle fields. | Successful provider-route job syncs. | | `job_versions` | Normalized current or historical public posting content. | Job sync content hashing. | | `job_payload_snapshots` | Distinct raw listing/detail payload snapshots for replay and audit. | Job sync payload hashing. | | `job_sync_runs` and `job_sync_observations` | Per-route sync evidence for new, unchanged, changed, reopened, and closed outcomes. | Provider-route job syncs. | Job storage separates durable posting identity from changing content. The `jobs` table tracks status, first/last seen timestamps, close time, and current hashes. `job_versions` stores normalized user-visible content with child tables for locations, skills, skill keywords, and responsibility or qualification bullets. Raw payload drift updates payload snapshots and observations without creating a new user-visible version when normalized content does not change. ## Ingest surplus taxonomy (S1–S4) [#ingest-surplus-taxonomy-s1s4] Board ingestions capture more than first-class normalized columns. OpenOpps classifies the extra signal into four surplus classes: | Class | Where it lives today | Examples | | --------------------------- | ---------------------------------------------------- | ----------------------------------------------------- | | **S1 Raw-buried** | `raw_listing`, `raw_detail`, `job_payload_snapshots` | Greenhouse `metadata`, `requisition_id`, office trees | | **S2 Fetchable-not-called** | Board detail APIs not requested during sync | Greenhouse `pay_input_ranges`, org `/offices` trees | | **S3 Derivable** | Computable from normalized text | `seniority`, expanded skill catalog, `daysOpen` | | **S4 Sync-evidence** | `job_sync_runs`, `job_sync_observations` | Velocity, churn, reopen, median days open | Promotion policy: move high-cardinality S1 keys into typed `JobRecord` fields or `version.extra_payload` (`posting_kind`, `seniority`, `provider_extras`); keep full raw payloads in SQLite/Kaggle only. The docs search index uses **tiered detail shards**—T2 full body for indexable SEO jobs, T1 metadata-only for other open jobs—and never commits `payloadSnapshots`. ## Count Provenance [#count-provenance] OpenOpps has two generated count families: | Count family | Source | Used by | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Package catalog counts | `scripts/generate_docs_data.py` writes `web/lib/generated/openopps-data.json`. | Source catalog, provider registry, export format, docs summary cards. | | Snapshot/search counts | `scripts/generate_docs_search_index.py` reads `kaggle/openoppsdb.sqlite` and writes either the v6 `web/public/data/openopps-search/` tree or a separate v7 publication root. | Jobs workbench, `/explorer`, snapshot freshness, filters, row totals, and preview/detail shards. | Do not copy counts into prose unless the sentence explicitly names the snapshot date or package source. Prefer generated components and generated manifest values so the site does not drift when sources, providers, locations, departments, or exports change. ## Exports [#exports] `boards export` and `jobs export` provide filtered extracts of normalized records: ```bash uv run openopps boards export --provider ashbyhq --has-jobs --format jsonl --output /tmp/openopps-boards.jsonl uv run openopps jobs export --remote Full --skill Python --format csv --output /tmp/openopps-jobs.csv uv run openopps jobs export --source a16z --salary-min 150000 --format parquet --output /tmp/openopps-jobs.parquet uv run openopps jobs export --status all --format sqlite --output /tmp/openopps-jobs.sqlite ``` | Format | Use case | Notes | | --------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `jsonl` | Streaming audit logs and line-oriented processing. | Preserves normalized values as JSON records. | | `csv` | Spreadsheet review and lightweight exchange. | Formula-leading strings are neutralized before writing. | | `parquet` | Polars, DuckDB, and analytics workflows. | Empty exports should remain readable empty tables. | | `sqlite` | Relational handoff, filtered local extracts, and portable query files. | Entity tables should use the flattened CSV/Parquet contract; nested values should be stable JSON strings. | The full OpenOppsDB snapshot remains the durable SQLite ledger. The docs and Kaggle bundle flows use `kaggle/openoppsdb.sqlite` as the local ignored input for static search artifacts and public dataset generation. ## Static Jobs and Explorer Index [#static-jobs-and-explorer-index] The web app does not call provider APIs at runtime. It reads a generated static projection: | Artifact | Purpose | | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `search-manifest.json` (v7) / `manifest.json` (v6) | Snapshot time, counts, chunks, facets, and search projection metadata. | | Job chunks | Searchable latest job rows for the Jobs workbench and browser search worker. | | Detail shards | Posting details loaded after selection. T1 carries metadata only; T2 adds bounded plain text (≤4000 characters, HTML stripped). | | Board/provider chunks | Explorer coverage, route, and source/provider inspection. | | `publication-policy.json` (v7) | Sanitized source rights, required attribution, and snapshot-quality counts. | | `manifest.json` (v7 release root) | Exact file closure, byte lengths, SHA-256 values, semantic roles/counts, source provenance, generator provenance, and release identity. | The committed `web/public/data/openopps-search/` version 6 tree is a bounded transition fallback. A version 7 publication lives outside that tree under immutable `releases//` paths selected by `channels/production.json` schema version 2. Every v7 payload read must be declared by and match the pinned release manifest. **Description projection matrix:** Kaggle/SQLite bundle previews truncate normalized text at **512** characters; search **T2** detail shards cap plain `description` at **4000** characters after HTML stripping; **Parquet/JSONL** exports keep full normalized job fields from SQLite. Public artifacts never include `payloadSnapshots` or `descriptionHtml`. ### Indexable job criteria [#indexable-job-criteria] A job receives a **T2** detail shard (and appears in `jobs-indexable-ids.json`) only when all of the following hold at index generation time: * `status` is open (or unset) * `title`, `company`, and a non-empty description (plain or HTML) are present * at least one of `postedAt`, `firstSeenAt`, or `versionCreatedAt` is set * a safe absolute `postingUrl` or `applyUrl` is present (no credentials, fragments-only, or non-http schemes) All other open jobs still get T1 metadata-only detail shards for preview and filtering. Refresh the index after updating `kaggle/openoppsdb.sqlite`: ```bash just web-search-index just web-search-index-check ``` `just web-search-index-check` requires the ignored local SQLite snapshot, regenerates `web/public/data/openopps-search/`, and fails if the committed v6 transition artifacts still differ. If the SQLite file is absent, use package-derived web checks and record the missing snapshot as the blocker for v6 search-index parity. `just web-search-artifacts-check` validates the committed artifact graph (manifest paths, detail shard counts, Git tracking) without regenerating from SQLite. CI runs `uv run pytest tests/unit/openopps/test_docs_search_index.py -k committed` for committed manifest schema parity. Version 7 uses `--release-root` during generation and `--channel production` during verification; see [Public Data Releases](/docs/public-data-releases). ## Facets and Suggestions [#facets-and-suggestions] Jobs and explorer filters should be generated from the static search manifest, not hardcoded UI lists. The manifest is the right place for canonical values and counts for: * sources and source aliases * providers and support levels * locations * departments and teams * companies * skills and skill keywords * workplace and employment types * job status and salary currency **Seniority** (`seniorities` facet): populated from each job row's `seniority` column. Values come from `version.extra_payload.seniority` when promoted during ingest; otherwise the index generator derives a label from normalized title and `job_versions.experience` text using the same `derive_seniority()` rules as `openopps.models`. **Days open** (`daysOpen` column): computed at index generation from `snapshotAt` minus `firstSeenAt` (non-negative integer days). This is an S3 derivable field and is not stored in SQLite job rows. Fuzzy matching should rank normalized generated suggestions first and only fall back to typed substring matching when the user enters a value that is not in the generated index. ## Provider surplus promotion [#provider-surplus-promotion] Provider adapters preserve full upstream payloads on `raw_listing` and `raw_detail`. v0.1 promotion focuses on **S1 list-endpoint fields** already fetched during sync: | Provider | Promoted surplus (initial) | | ---------- | ------------------------------------------------------------------------------------------------------------------------- | | Greenhouse | `metadata`, `requisition_id`, `language`, department/office trees; `posting_kind=prospect` when `internal_job_id` is null | | Workable | Split `raw_listing` (list POST) vs `raw_detail` (per-job GET) | | Others | See `openspec/changes/ingest-data-surplus/provider-promotion-manifest.json` | Derived **S3** fields such as `seniority` are computed from title/experience text without extra HTTP. **S4** sync aggregates surface in the search manifest `dashboard.sync` block when `job_sync_runs` is present in the snapshot. ## Telemetry Event Lake [#telemetry-event-lake] The docs app telemetry source of truth should be a first-party event lake. The collector can run as a no-op locally, append raw events when persistent storage is configured, mirror sanitized events to PostHog when configured, and compact local events into Parquet for DuckDB analysis. Recommended event shape: | Field | Meaning | | -------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `event_id` | Client-generated unique id for de-duping. | | `schema_version` | Telemetry schema version, starting at `1`. | | `sent_at` and `received_at` | Client and collector timestamps. | | `event_name` | Allowlisted event name such as `jobs.filters_changed` or `explorer.lineage_loaded`. | | `anonymous_id`, `session_id`, `page_id` | First-party anonymous identifiers. | | `context.path`, `context.viewport`, `context.screen`, `context.timezone`, `context.language`, `context.languages`, `context.userAgent` | Browser context retained by the current route allowlist. Raw `search`, `url`, and `referrer` fields are dropped. | | `request.path`, `request.headers`, `request.ip` | Server-side request context after header allowlisting, query removal, and IP drop/hash/raw policy. | | `entity_context` | Job, source, provider, filter, or explorer entity ids when relevant. | | `properties` | Event-specific allowlisted metadata. | | `redaction_count`, `payload_truncated` | Evidence that sanitization and size limits were applied. | `page_engagement` events include both cumulative counters (`durationMs`, `visibleDurationMs`, `interactionCount`) and additive counters (`durationDeltaMs`, `visibleDurationDeltaMs`, `interactionDeltaCount`) with a `reason` and `sequence`. Use the additive counters for aggregate dwell-time stories so route changes, hidden-tab flushes, and unmounts do not double-count the same page interval. Runtime sink modes: | Sink | Use case | Cost posture | | ------------------ | --------------------------------------------------- | --------------------------------------------- | | `noop` | Local development or deployments without storage. | Free; records nothing. | | `local-event-lake` | Canonical first-party telemetry on persistent disk. | Free when running on existing infrastructure. | The current `/api/telemetry` route treats any unsupported sink value as `noop`. PostHog forwarding is controlled separately with `OPENOPPS_POSTHOG_PROJECT_API_KEY`, receives only allowlisted sanitized properties, and is bounded by `OPENOPPS_POSTHOG_TIMEOUT_MS`. Browser PostHog session replay is an opt-in mirror controlled by `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY`; replay masks text and inputs, disables network body/header capture, respects PostHog project sampling and trigger controls, and does not change the first-party event schema. Cloudflare, Umami, or BI dashboards can mirror sanitized summaries downstream, but the canonical raw sink remains `local-event-lake`. Sanitize obvious secret-like values, cap event size, and make raw IP retention an explicit opt-in. Hashing IPs with a deployment salt is the default compromise when approximate abuse/debugging signal is useful. # Start Here (/docs) OpenOpps v0.1 is a Python CLI for discovering firm hiring boards from aggregate sources, detecting public provider routes, syncing normalized jobs, caching repeated HTTP JSON requests, and exporting an auditable local opportunity ledger. Ingestion and durable state are CLI-driven; the published [Jobs](/) and [Explorer](/explorer) surfaces are a static docs workbench over a committed search snapshot, not a hosted sync service. There is no prompt UI or TUI.
Browse the committed snapshot Use the{" "} Jobs {" "} surface for open-role search and posting previews, or{" "} Explorer {" "} for source, provider, route, and data-quality analysis.
Discover Read source catalogs and persist durable board records.
Resolve Promote provider hints into executable public job routes.
Export Write normalized boards and jobs as JSONL, CSV, Parquet, or SQLite.
## Domain [#domain] * `sources` are aggregate discovery catalogs such as `a16z`, `accel`, `lsvp`, `sequoia`, `bvp`, `greylock`, `kleinerperkins`, `southparkcommons`, `signalfire`, and `yc`. * `boards` are firm/company hiring boards discovered from sources. * `jobs` are normalized postings fetched from boards. * `providers` are adapters that detect or fetch provider-specific boards, such as Ashby, Greenhouse, Lever, Workday, Workable, Teamtailor, BambooHR, Rippling, and WP Job Manager. * `cache`, `plugins`, and `examples` cover operational cache inspection, installed Python plugin discovery, and deterministic demo data. * `discovery` is an advanced quarantined scout (`openopps discovery scout|verify-scout|preview-promotion`). It does not write the daily snapshot and has no `--apply` path. ## Install and Run [#install-and-run] Use `uv run` while working inside the repository checkout: ```bash uv sync just --list uv run openopps --help uv run openopps status ``` If you want `openopps` available directly from this editable checkout, install the repo as a uv tool from the repository root: ```bash uv tool install -e . openopps --help openopps status ``` `uv tool install -e .` installs the console entry point from the current checkout. Runtime settings such as `OPENOPPS_DB_URL` still control where local SQLite state is read and written. ## First run [#first-run] Before live syncs against public upstreams, initialize the durable SQLite schema for the database you intend to use: ```bash uv run openopps admin db init uv run openopps admin db status ``` `OPENOPPS_DB_URL` defaults to `sqlite:///openoppsdb.sqlite` relative to the process working directory. For an isolated smoke database, point at a dedicated file before `admin db init` and sync: ```bash OPENOPPS_DB_URL=sqlite:///./tmp/openopps-smoke.sqlite uv run openopps admin db init OPENOPPS_DB_URL=sqlite:///./tmp/openopps-smoke.sqlite uv run openopps examples seed --json ``` See [Configuration](/docs/configuration) for concurrency and job-route settings. ## Contributor Command Map [#contributor-command-map] The root `Justfile` is the quickest way to discover local validation without hiding the underlying toolchain: ```bash just --list just quick just ci just openspec-validate-all just web-check just cli-help ``` `just ci` composes the `ci-python`, `ci-openspec`, `ci-web`, and `ci-artifacts` lanes. Those cover the Python release gate, strict OpenSpec validation, web type/build/unit/browser/accessibility/lint/search-artifact checks, Kaggle metadata/bundle smoke, and repository drift. Use the raw `uv`, `pnpm`, and OpenSpec commands from the reference pages when a CI failure needs exact reproduction; use `just ci-full` when network-dependent security audits and lowest-direct dependency tests are required. ## Safe Local Smoke Path [#safe-local-smoke-path] These commands seed deterministic demo records and do not hit upstream source or provider services: ```bash uv run openopps examples seed --seed 42 --boards 4 --jobs-per-board 2 --json uv run openopps jobs list --source example --json uv run openopps providers coverage --json ``` ## Live Quickstart [#live-quickstart] The commands below read public upstream catalogs and provider endpoints. They write local SQLite state unless a command is explicitly marked no-DB or dry-run. ```bash uv run openopps sources list uv run openopps admin sources test a16z uv run openopps sources sync a16z --metrics-json --refresh-cache uv run openopps sources sync accel --metrics-json uv run openopps sources sync greylock --metrics-json uv run openopps boards list --source a16z --limit 10 uv run openopps admin boards enrich --source a16z --json uv run openopps providers health --source a16z --provider any --limit 25 --json uv run openopps admin providers probe-routes --source a16z --provider any --limit 25 --json uv run openopps jobs sync --provider ashbyhq --metrics-json --refresh-cache uv run openopps cache status --json uv run openopps plugins list --json ``` Unscoped list, export, and sync commands use the full known superset unless narrowed by `--source`, `--board`, or `--provider`. Provider filters accept `any` and `all` as aliases for removing the provider filter, which keeps reusable scripts explicit without narrowing to one provider. Overlapping source coverage uses the persisted board keys shown by `boards list`, and provider requests are deduped before route probing or job sync. The relevant metrics include `duplicateRoutesSkipped`. ## Provider Levels [#provider-levels] | Level | Meaning | | ------------- | -------------------------------------------------------- | | `detect` | OpenOpps can preserve provider metadata and route hints. | | `jobs` | OpenOpps can fetch public jobs for the provider. | | `unsupported` | The provider is known only as raw metadata. | Ashby, Greenhouse, Lever, public Workday CXS, Workable, Teamtailor, BambooHR, Rippling, and explicit WP Job Manager boards are job-capable in v0.1 through public no-auth routes. Manatal and Gem hints remain detect-only until stable public fetching is added. See [Providers](/docs/providers) for the generated provider registry and route diagnostics. Ashby postings marked `isListed: false` are direct-link-only and are excluded from normal job sync output. ## Documentation Map [#documentation-map] | Page | Use it for | | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [CLI](/docs/cli) | Command groups, common flags, JSON output, filters, exports, and quarantined `discovery scout` / `verify-scout` / `preview-promotion`. | | [Configuration](/docs/configuration) | `OPENOPPS_` environment variables, isolated `OPENOPPS_DISCOVERY_*` scout limits, `.env` loading, and runtime policy. | | [Data Model](/docs/data-model) | Sources, boards, providers, jobs, exports, static indices, counts, and telemetry event-lake guidance. | | [Providers](/docs/providers) | Source catalogs, eight required taxonomy fields, route diagnostics, and limitations. | | [Operations](/docs/operations) | Storage, cache, exports, quarantined scout accounting and promotion preview, policy declaration versus verification, validation, telemetry operations, Kaggle, and troubleshooting workflows. | | [Public Data Releases](/docs/public-data-releases) | V7 manifests, source-rights gates, release-pinned web reads, static delivery, rollback, recovery, and the v6 cutover boundary. | | [Contributing](/docs/contributing) | Local setup, validation, docs generation, source-scout skill non-authority, and review expectations. | | [Jobs](/) | Filter and preview open roles from the static search index. | | [Explorer](/explorer) | Analyze source/provider coverage, route health, freshness, and data quality. | ## Public Docs Routes [#public-docs-routes] The canonical jobs workbench is `/`, the canonical analytics explorer is `/explorer`, and thin job detail pages live at `/jobs/:id` for direct posting previews. Legacy `/jobs` redirects to `/`, and legacy `/docs/explorer` redirects to `/explorer`; do not add broad `/jobs/:path` redirects because detail pages must remain addressable. ## LLM-readable docs [#llm-readable-docs] The docs app exposes machine-readable exports for agents and tooling: | URL | Purpose | | ---------------- | ----------------------------------------------------- | | `/llms.txt` | Compact index of documentation pages for LLM context. | | `/llms-full.txt` | Full concatenated docs text for deeper retrieval. | Per-page markdown routes are also available under `/llms.mdx/` when you need a single MDX page in isolation. ## Route Probing Summary [#route-probing-summary] Provider hints from source catalogs may not include the token or URL needed for job fetching. `admin providers probe-routes` tries candidate tokens from upstream slugs, remote ids, names, domains, and websites, then reports both matched routes and unknown boards with the candidate tokens it tried. Its JSON output includes per-provider selected/matched counts, unresolved reason counts, and duplicate route skips. ```bash uv run openopps admin providers probe-routes --source a16z --provider all --limit 25 --json ``` Probing is read-only unless `--apply` is passed. Use [CLI](/docs/cli) for command flags and [Operations](/docs/operations) for local-state runbooks. # Operations (/docs/operations) Use this page for local CLI operations and docs validation while working on OpenOpps v0.1. There is no hosted runtime to initialize; durable state lives in local SQLite files controlled by `OPENOPPS_` settings.
Local state Sync and seed commands write to the database selected by OPENOPPS\_DB\_URL.
Dry-run diagnostics Route probing and health checks report first and persist only with --apply.
Export boundary JSONL, CSV, Parquet, and SQLite exports preserve normalized records for audit or analysis.
## Command Modes [#command-modes] Use `uv run openopps ...` inside the checkout. After `uv tool install -e .` from the repo root, the same examples can be run as `openopps ...` directly. ## Public docs surfaces [#public-docs-surfaces] The docs app exposes three v1 URL surfaces plus exact legacy redirects for moved routes. | URL | Purpose | | ----------- | ----------------------------------------------------------------- | | `/` | Interactive jobs board (search, filters, inline preview) | | `/explorer` | Dataset explorer across jobs, boards, and providers | | `/jobs/:id` | Static job detail pages for shareable permalinks and SEO metadata | | `/?job=:id` | In-app deep link that opens a job preview on the board | Bare `/jobs` redirects to `/`, and `/docs/explorer` redirects to `/explorer`. Trailing-slash variants normalize through the same canonical destinations. Job detail routes and `/jobs/sitemap/*.xml` remain available under the `/jobs/` prefix; do not add broad `/jobs/:path` redirects. ## Storage [#storage] SQLite is the default DB-backed storage mode: ```bash uv run openopps admin db init uv run openopps admin db status uv run openopps sync a16z --metrics-json uv run openopps sources sync a16z --metrics-json uv run openopps boards sync --source a16z --provider any --metrics-json uv run openopps jobs sync --provider any --metrics-json ``` The default database path is controlled by `OPENOPPS_DB_URL` and resolves to `openoppsdb.sqlite` for `sqlite:///openoppsdb.sqlite`. All sync commands in this section write local SQLite records unless they explicitly use `--no-db` or are documented as dry-run diagnostics. `admin db init` applies the current v0.1 Alembic schema head for the durable app file-backed SQLite database. HTTP response cache rows live in that same SQLite database and are managed by `cache.py`, not by the Alembic schema history. Unsupported or partial pre-release schemas fail closed; reset that local database or point `OPENOPPS_DB_URL` at a new SQLite file instead of relying on silent repair. Job storage separates stable posting identity from changing content. The `jobs` table tracks lifecycle fields such as `status`, `first_seen_at`, `last_seen_at`, `closed_at`, and current hashes. `job_versions` stores one normalized content state per posting, with child tables for locations, skills, skill keywords, and responsibility or qualification bullets. `job_payload_snapshots` stores distinct raw listing/detail payloads so raw upstream JSON is never discarded. Every provider-route job sync creates a pending `job_sync_runs` row before network access and finishes it as succeeded or failed with bounded error classification, authority, timing, and committed-batch metadata. A route may close unseen jobs only after its provider returns a complete authoritative snapshot. Truncated pagination, count mismatch, continuation loops, access/schema failures, and ambiguous empty results preserve existing job state and fail the run. Successful runs write `job_sync_observations` rows for `new`, `unchanged`, `changed`, `reopened`, and `closed` outcomes. Raw payload drift means provider JSON changed but normalized content stayed the same; raw-only drift updates payload snapshots and observations but does not create a new `job_versions` row. Use `status` or `doctor` to inspect the full local runtime state: ```bash uv run openopps status uv run openopps doctor --json ``` ## Cache [#cache] The shared request cache stores successful JSON response payloads in the configured SQLite application database for source adapters and job providers. Cache schema version 2 stores a SHA-256 request identity and redacted canonical location, never a raw request body, URL user information, or credential-bearing query value. It records the response payload hash, an allowlist of response headers, freshness timestamps, ETag/Last-Modified validators, and stale-on-error eligibility. Legacy cache rows are invalidated. ```bash uv run openopps cache status uv run openopps admin cache purge --namespace http-json --json uv run openopps sources sync a16z --refresh-cache --metrics-json uv run openopps jobs sync --provider any --refresh-cache --metrics-json ``` `--refresh-cache` bypasses cache reads for that operation and lets successful responses update cache state. `admin cache purge` deletes cache rows. Scope it with `--namespace` when you only need to invalidate one cache family. ## Examples [#examples] Seed deterministic synthetic records for local demos and smoke tests without upstream network access: ```bash uv run openopps examples seed --seed 42 --boards 4 --jobs-per-board 2 --json uv run openopps jobs list --source example --json uv run openopps providers coverage --json uv run openopps admin sources yield --json ``` For a fully isolated smoke test, point `OPENOPPS_DB_URL` at a temporary SQLite file before seeding. ## No-DB Source Sync [#no-db-source-sync] For one-off source extraction without persistence, use explicit JSONL output: ```bash uv run openopps sources sync a16z --no-db --output /tmp/a16z-boards.jsonl ``` Human sync commands use a brief dynamic progress display by default. The top-level `sync` command reports source, board, and job stages in order; individual `sources sync`, `boards sync`, and `jobs sync` commands report the stage they own. Add `--verbose` when you need detailed provider warnings on stderr. Machine-readable modes such as `--metrics-json` keep stdout parseable. `jobSyncAttempts` counts durable route attempts, while `jobSyncRuns` counts only authoritative successes; failed, invalid, and partial snapshots therefore cannot masquerade as release-quality success evidence. ## Exports [#exports] ```bash uv run openopps boards export --format jsonl --output /tmp/openopps-boards.jsonl uv run openopps boards export --provider ashbyhq --has-jobs --format jsonl --output /tmp/ashby-boards.jsonl uv run openopps jobs export --remote Full --skill Python --format csv --output /tmp/openopps-jobs.csv uv run openopps jobs export --status all --format jsonl --output /tmp/openopps-all-jobs.jsonl uv run openopps jobs export --source a16z --salary-min 150000 --format parquet --output /tmp/openopps-jobs.parquet uv run openopps jobs export --status all --format sqlite --output /tmp/openopps-jobs.sqlite ``` Use `jsonl` for auditability, `csv` for spreadsheet inspection, `parquet` for analytics workflows, and `sqlite` for portable relational handoff. JSONL exports stream records as they are encoded. Empty JSONL and CSV exports produce empty files; empty Parquet exports produce a readable empty Parquet table. CSV exports neutralize spreadsheet formula-leading strings by prefixing a single quote; JSONL and Parquet preserve values as-is for machine processing. SQLite exports should preserve the flattened export contract while storing nested values as stable JSON strings. `boards export` accepts the same board filters as `boards list`, including `--source`, `--provider`, `--market`, `--location`, `--domain`, `--has-jobs`, `--min-staff`, `--max-staff`, and `--limit`. `jobs export` accepts the same job filters as `jobs list`, including `--source`, `--board`, `--provider`, `--location`, `--department`, `--team`, `--workplace-type`, `--remote`, `--employment-type`/`--type`, `--salary-min`, `--salary-max`, `--skill`, `--query`, `--posted-after`, `--posted-before`, `--status`, and `--limit`. These filters use normalized current-version job fields; raw upstream payloads remain exported for auditability but are not part of primary filtering semantics. Job list and export commands default to `--status open`; pass `--status all` when you intentionally need closed jobs in an audit extract. ## Route Probing [#route-probing] The stable everyday route resolution command is `boards sync`, which enriches boards and applies successful missing route probes: ```bash uv run openopps boards sync --source a16z --provider all --limit 25 --metrics-json ``` The admin route-probing command remains dry-run-first for diagnostics: ```bash uv run openopps admin providers probe-routes --source a16z --provider all --limit 25 --json ``` Use `--apply` only after inspecting matched routes. ## Source Yield [#source-yield] Use source-yield reporting after source syncs, route probing, and job syncs when you need to compare which source families are turning into active job routes: ```bash uv run openopps admin sources yield --json uv run openopps admin sources yield --source sec-company-tickers ``` The report is offline and reads persisted SQLite records only. It does not fetch sources, probe routes, or sync jobs. Treat `yieldScore` as a local snapshot metric rather than a global benchmark unless the database was refreshed from a representative source set. `openopps providers coverage --json` also exposes `gaps.sourceScope` for v0.1 startup-board exclusions (WorkAtAStartup), Wellfound/Angel unsupported rationales, and the Editorial label audit decision. ## Quarantined source discovery [#quarantined-source-discovery] `openopps discovery` is advanced admin, not an everyday sync stage. It is not same-run with `openopps sync`, `sources sync`, `boards sync`, or `jobs sync`. Default unscoped `openopps sync` remains the snapshot writer and keeps using the last reviewed approved catalog. Scout output is an explicit quarantine directory, not a snapshot and not live publication evidence. ```bash uv run openopps discovery scout --output /absolute/quarantine-root --json uv run openopps discovery verify-scout /absolute/quarantine-root --json uv run openopps discovery preview-promotion --json just ci-discovery just source-policy-check just source-policy-audit ``` The same callbacks are aliased as `openopps admin sources scout`, `verify-scout`, and `preview-promotion`. None accept `--apply`. Scout writes only the required `--output` directory. It does not mutate operational SQLite, Git, the packaged catalog, Kaggle, or Cloudflare, and it cannot activate candidates in the same invocation. The current CLI scout publishes an evaluation bundle from an empty occurrence set against read-only v7 policy digests. Channel enumerators stay replay-library surfaces. Command strings are on [CLI](/docs/cli#quarantined-source-discovery). Isolated `OPENOPPS_DISCOVERY_*` limits are on [Configuration](/docs/configuration#isolated-discovery-scout). Taxonomy fields are on [Providers](/docs/providers#required-discovery-taxonomy). Skill contributor gates are on [Contributing](/docs/contributing#quarantined-discovery-and-the-source-scout-skill). ### Policy declaration versus independent verification [#policy-declaration-versus-independent-verification] Declared catalog status is not verified live rights. v7 source-policy is a deny-only overlay. `just source-policy-check` runs `scripts/source_policy_review.py validate`: canonical evidence bytes, schema, and the exact committed-v6 corpus identity. That structural gate is part of `just ci` via `ci-artifacts`. It is not permission to publish. `just source-policy-audit` runs the same validator, prints the eligibility summary, and exits 2 while any source is blocked. The audit type fixes `independentlyVerifiedAllowedCount` at `0` and labels allowed keys `repository_catalog_declarations_not_independent_legal_review`. Current committed-v6 counts: 1787 sources, 7 catalog-declared allowed, 0 independently verified, 1780 blocked. The 1780 stay blocked pending written Getro/Consider grants (`getro-terms-v3-1`, `consider-terms-observed-2026-08-13`) plus snapshot keys without independent review (`historical-snapshot-sources-unreviewed`). Do not treat a green structural check as eligibility, and do not render a selector or publish while the audit is red. Discovery promotion binds those read-only v7 digests in `openopps.discovery.policy` and requires independent positive closure of five operations: `access`, `license`, `redistribution`, `sync`, and `publication`. Deny-overlay strings such as `allowed`, `public`, or `granted` stay `unresolved`. Untrusted observations and `sourceAttribution` never grant permission; attribution is a requirement only. A `DiscoveryPromotionPolicyDecision` plus the evidence-only receipt still set `grantsAuthority=false`. Scout, verify, preview, CI, and scheduled workflows cannot mint a positive decision or invoke apply. ### Complete versus degraded accounting [#complete-versus-degraded-accounting] Conservation of a planned set is not a complete attestation. Candidate, source, and route denominators are exact and mutually exclusive. Scout candidate accounting (`ScoutCandidateAccounting` in `src/openopps/discovery/models.py`) conserves: ```text observedCandidateOccurrences = invalidOccurrences + normalizedOccurrences normalizedOccurrences = duplicateOccurrences + uniqueCandidates uniqueCandidates = alreadyApproved + quarantinedCandidates quarantinedCandidates = promotable + blocked + unsupported + inconclusive ``` Source terminals (`SourceAccounting` in `src/openopps/discovery/accounting.py`): ```text planned = succeeded + failed + timedOut + freshSkipped + policyBlocked + rateLimited + cancelled + unstarted ``` `complete` is true only when the run is terminal `succeeded`, every planned id is accounted, none of `failed` / `timedOut` / `policyBlocked` / `rateLimited` / `cancelled` / `unstarted` are present, and every `succeeded` or `freshSkipped` outcome is authoritative with a freshness-context digest that matches the pin. Fresh skips are not a completeness exemption without that digest. Route terminals (`RouteAccounting`) add `deferred`, `duplicateSkipped`, and `missingMetadata`. `complete` additionally forbids deferred, missing-metadata, and the failed classes above. Each `duplicateSkipped` route must name exactly one canonical representative that is an authoritative `succeeded` or `freshSkipped` bound to the same freshness context. A duplicate group of only skips fails closed. `classify_typed_degraded` returns `complete` with no class only when source and route are both complete, every operation terminal is `succeeded`, the operation channel state is `complete`, and `run_state` is `succeeded`. Otherwise it returns `degraded` plus one of `TYPED_DEGRADED_CLASSES`: `failed`, `timed_out`, `policy_blocked`, `rate_limited`, `cancelled`, `unstarted`, `missing_metadata`, `deferred`, `partial`, `nonterminal`. Multiple count classes collapse to `partial`, or to `nonterminal` when that class is present. Complete source or route accounting cannot pair with a non-succeeded run, and a complete attestation cannot carry a degraded class. Selector-bound `openopps sync --metrics-json` attaches `attestation` and `degradedClass` from that classifier. `jobSyncAttempts` versus `jobSyncRuns` still counts durable route attempts versus authoritative successes; those counters are not a complete attestation. `just source-discovery-accounting-check` conserves the pinned envelope as unstarted terminals and does not ingest. ### Bundle freshness, replay, rollback, and revocation [#bundle-freshness-replay-rollback-and-revocation] `discovery verify-scout` calls `verify_bundle` with `evaluation_bundle_verification_policy`: timezone-aware `observedAt`, a supported schema and profile, and `max_evidence_age` of 48 hours. Future-dated or stale evidence fails closed. That window is not the v7 public-data snapshot freshness gate. `OPENOPPS_DISCOVERY_EVIDENCE_RETENTION_SECONDS` (default `86400`) is a separate scout setting: the maximum age for exact-verified quarantine evidence to be reusable by a later isolated scout. Conditional reuse still requires those prior exact-verified bytes; verify-scout does not refetch. Replay here is offline fixture replay, not a live crawl. Channel enumerators (`enumerate_official_channel`, `enumerate_public_code_channel`, `enumerate_search_channel`, `enumerate_targeted_ats_channel`) consume maintainer-owned seeds plus captured observations and emit a closed `ChannelReplayReceipt`. They do not open sockets. Public CI and `just ci-discovery` set `OPENOPPS_DISCOVERY_NETWORK=disabled` and replay committed sanitized fixtures. Promotion-intent replay is a different, forbidden operation. Durable state is the hash-chained JSONL ledger at `src/openopps/discovery/data/promotion_decision_ledger.jsonl`. The only states are `reserved`, `applied`, and `revoked`; transitions are append-only (`reserved` → `applied` or `revoked`, `applied` → `revoked`, `revoked` terminal). Global replay keys are `decisionId` and `promotionIntentDigest` only. A semantic manifest digest is reusable content identity, not a global blacklist. `BundleVerificationPolicy.replayed_manifest_ids` / `revoked_manifest_ids` are legacy fields; CLI verify passes empty sets. There is no `openopps discovery rollback` command and no `--apply` flag. `preview-promotion` is a digest-bound dry-run (`applied=false`, `grantsAuthority=false`) and writes no repository files. The committed evidence-only receipt at `src/openopps/discovery/data/evidence_only_decision_receipt.json` also has `grantsAuthority=false`. Reserve, apply, recover, and revoke require `invocation_mode="maintainer"` in `openopps.discovery.promotion_runtime`. Recovery uses `choose_recovery_action`: `finalize` only when every owned path matches the journal `after` bytes; otherwise `restore_and_revoke` restores each journal `before` image and appends `revoked`. `revoke_promotion` appends `revoked` under the repository lock. That is a forward compensating ledger operation plus restored bytes, not a Git history rewrite. Reverted promotion digests stay revoked and need a fresh bundle and review. Candidate manifests cannot carry approval, reviewer, signature, receipt, or revocation fields. Kaggle `allow_no_rollback=1` on this page is a first-create dataset acknowledgement. It is not discovery ledger rollback. ### Skill non-authority and isolated validator [#skill-non-authority-and-isolated-validator] The portable skill at `skills/openopps-source-scout/` is advisory. Skill prose does not confine tools already authorized in a parent Codex, Cursor, or Grok Build harness. A suggestion is never approval, policy permission, review, promotion, or runtime activation. The parent harness is outside OpenOpps enforcement. Acceptance is only through the deterministic isolated validator `openopps.discovery.isolation.launch_isolated_scout` (committed fixtures via `skills/openopps-source-scout/scripts/validate_fixture.py`). The worker is a credential-free, allowlisted, bounded process that writes only a parent-owned new quarantine file; that is an application contract, not an OS sandbox claim. Do not live-install harness projections under `.agents/skills/` or `.cursor/skills/`. Selected Codex/Cursor copies must remain absent; Grok has no repository projection. Do not run `wagents --apply` or a live skill install from this workflow. `just source-discovery-skill-eval-check` runs the read-only evals, frontmatter, dry-run projection, and docs-steward skip gates. `resolve_docs_steward.py` searches with `uv run wagents skills search docs-steward --json` and skips when `wagents` is absent. ## Validation [#validation] Use `just` from the repository root for the contributor validation graph: ```bash just quick just ci just ci-discovery just source-policy-check just lock-check just openspec-validate-all just web-check just web-build just web-test just web-e2e just web-a11y just web-lint just kaggle-meta just kaggle-bundle-check kaggle/openoppsdb.sqlite ``` `just ci` composes `ci-python`, `ci-openspec`, `ci-discovery`, `ci-web`, and `ci-artifacts`. These lanes cover the Python release gate, strict OpenSpec validation, offline source-discovery gates with `OPENOPPS_DISCOVERY_NETWORK=disabled`, web type/build/unit/browser/accessibility/lint/search-artifact checks, Kaggle metadata and clean-bundle smoke, and repository drift. `ci-artifacts` runs `source-policy-check` and does not run `source-policy-audit`. `just ci-full` runs `just ci` plus the network-dependent `security-audit` (Python `pip-audit` and the full web dependency graph via `pnpm audit --audit-level high`) and `test-lowest-direct`. Use `ci` for day-to-day PR confidence; use `ci-full` before release or when dependency/security surfaces change. ### Jobs search public data release [#jobs-search-public-data-release] The committed v6 tree is a transition fallback when no channel is configured. For v7, set matching server and browser origins/channels as described in [Configuration](/docs/configuration#web-app-public-data-release). The shared snapshot client resolves the mutable channel once, pins the immutable release, and verifies manifest membership, byte length, and SHA-256 before serving search, details, metadata, or sitemap reads. Jobs search runs in a dedicated browser Web Worker using release-pinned chunks. The `/api/jobs/search` route is retained only as a fail-closed stale-client boundary and returns HTTP `410` with `browser_worker_required`; it never loads or scans the full production corpus. Saved-search counts use the same worker/session snapshot. Local saved jobs, searches, and retained details commit to IndexedDB before visible state advances; validated replacement imports are transactional and retain at most three bounded pre-import backups. The default-off offline-search installer is implemented for v7. An explicit opt-in performs a two-times-size quota preflight, downloads and verifies a bounded search/metadata projection, pins one immutable release, preserves the previous verified release until replacement succeeds, and removes only OpenOpps-owned caches on opt-out. Unit tests cover those local invariants. Deployed offline readiness is still unproven: the release gate remains open until a real deployed v7 release passes install, disconnect/readback, update/rollback, and complete Chromium/Firefox/WebKit journeys. See [Public Data Releases](/docs/public-data-releases) and `deployment/openopps-data/README.md` for v7 generation, rights/freshness gates, assets-only staging, exact remote readback, promote/rollback/re-promote barriers, archive recovery, and v6 exit criteria. Those tools prepare or verify a rollout; they do not prove that a live rollout occurred. Run Python tests from the repository root: ```bash uv run pytest uv run pytest --cov=openopps --cov-report=term-missing uv lock --check ``` Run web checks from the web app: ```bash cd web pnpm data:generate pnpm data:generate:search pnpm types:check pnpm build pnpm lint pnpm test ``` `pnpm data:generate` refreshes package-derived provider/source metadata. `pnpm types:check` also regenerates that metadata before generating Fumadocs MDX artifacts, generating Next.js route types, and running TypeScript without emitting files. `pnpm lint` runs the Next ESLint surface. `pnpm test` runs the docs Vitest suite. `just web-build` runs the production docs build and the function trace guard that keeps committed search artifacts out of API route bundles. Use `just web-rtk-lint` from the repository root for the optional maintainer `rtk lint` surface; it requires `rtk` locally and is not silently skipped inside `just ci`. `pnpm data:generate:search` refreshes the committed v6 transition index used by [Jobs](/) and [Explorer](/explorer) when no v7 channel is configured. Run it after updating the local OpenOppsDB snapshot; from the repository root you can also use `just web-search-index`. Preview descriptions are limited to the normalized description fields present in that snapshot; when those fields are empty, the jobs surface shows metadata and links back to the source posting. `just web-search-index-check` requires the ignored local `kaggle/openoppsdb.sqlite` snapshot and fails clearly when that file is unavailable. Do not remove the v6 tree until every v7 cutover exit criterion in [Public Data Releases](/docs/public-data-releases#version-6-transition-exit-criteria) has passed for one exact SHA. Run OpenSpec checks from the repository root. Pin `@fission-ai/openspec@1.6.0` in copy-paste commands for parity with CI; do not use a floating `@latest`. Override the `Justfile` default with `OPENOPPS_OPENSPEC='npx -y @fission-ai/openspec@1.6.0'` when `just openspec-*` should run a different invocation. ```bash rtk npx -y @fission-ai/openspec@1.6.0 list --json rtk npx -y @fission-ai/openspec@1.6.0 validate --all --strict ``` OpenSpec-backed changes should keep proposal, design, specs, tasks, docs, nested `AGENTS.md`, CI, and just recipes synchronized. Use `instructions --json` when handing an OpenSpec task graph to another agent. ## Dependency And Secret Hygiene [#dependency-and-secret-hygiene] `uv.lock` and `web/pnpm-lock.yaml` are committed reproducibility artifacts. Local validation and CI both run `uv lock --check`, and web installs use `pnpm install --frozen-lockfile`. Renovate is configured at the repo root for Python `pyproject.toml`/`uv.lock` and web npm/pnpm dependency maintenance. Dependabot owns GitHub Actions pin updates only. Keep credentials local. `.env`, `.env.*`, `.envrc`, Kaggle `kaggle.json`, local package-registry credential files, `.netrc`, key bundles, and token or credential JSON files are ignored; `.env.example` remains the tracked non-secret template. Do not print secrets in logs, docs, CI output, or generated artifacts. ## Outbound fetch threat model [#outbound-fetch-threat-model] OpenOpps source and job providers fetch public HTTPS endpoints from the local CLI. `openopps.http.assert_public_fetch_url` is a defense-in-depth guard for that local threat model: * URLs must be public `https://` endpoints with a host. * DNS is resolved once before the request; any address that resolves to a non-global-routable IP is rejected. * Redirect hops are origin-checked, but each hop performs its own DNS lookup. This guard is **not** rebinding-proof. httpx resolves DNS again at connect time, so a hostile or compromised resolver could return a public address during the pre-check and a private/metadata address when the socket connects (classic DNS TOCTOU / rebinding). Redirect chains widen the window because every hop repeats the lookup. v0.1 intentionally stops at lightweight validation rather than connect-time IP pinning or a trusted resolver policy. Treat the check as reducing accidental SSRF against RFC1918/link-local targets, not as a network sandbox. If you need stronger outbound controls later, pin the vetted addresses into the client connect path or run ingestion behind an egress policy you control. ## Telemetry Operations [#telemetry-operations] Telemetry in this section applies **only** to the Fumadocs/Next.js docs app (Jobs, Explorer, and `/docs/*`). The `openopps` CLI does not collect, store, or forward usage telemetry; CLI behavior is controlled solely by `OPENOPPS_*` settings in [Configuration](/docs/configuration). The docs app telemetry source of truth should be a first-party event lake. The default local mode is no-op; production-like deployments should write append-only events to persistent storage, compact them into Parquet with a separate maintenance job, and query them with DuckDB or another local analytical engine. | Sink | Operating model | Notes | | ------------------ | ---------------------------------------------------------- | ------------------------------------------------------- | | `noop` | Collects nothing. | Default for local development or hosts without storage. | | `local-event-lake` | Appends sanitized NDJSON events under UTC date partitions. | Canonical free sink when persistent disk is available. | The current collector accepts only `noop` and `local-event-lake` as `OPENOPPS_TELEMETRY_SINK` values. Cloudflare, Umami, or BI dashboards should be treated as downstream mirrors fed from the event lake or from sanitized aggregates, not as the canonical raw telemetry store. PostHog can be enabled as a free hosted product-analytics mirror by setting `OPENOPPS_POSTHOG_PROJECT_API_KEY`. The route forwards only allowlisted, sanitized events after OpenOpps redaction and never forwards raw posting bodies, request headers, or raw IPs. Forwarding is bounded by `OPENOPPS_POSTHOG_TIMEOUT_MS` and runs after successful `local-event-lake` writes, so a slow hosted mirror cannot block the canonical sink. Browser session replay is a separate public-key opt-in through `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY`; the browser SDK masks all inputs and text, blocks elements marked `data-openopps-private`, `data-telemetry-private`, or `data-sensitive`, disables network body/header capture, leaves sampling plus URL/event/linked-flag controls to PostHog project configuration, and keeps PostHog automatic pageview/autocapture events off so OpenOpps remains the event schema source of truth. Use PostHog for dashboards and masked replay while keeping `local-event-lake` as the canonical raw sink when persistent storage is available. Telemetry operations should preserve the event lake, cap payload sizes, record redaction/truncation evidence, and avoid raw IP retention. Keep event names and property keys allowlisted so exploratory UI instrumentation does not accidentally persist secrets from query strings, stack traces, or copied provider payloads. Page-engagement stories should sum `visibleDurationDeltaMs` and `durationDeltaMs`; cumulative `visibleDurationMs` and `durationMs` are retained for last-known-state debugging. | Variable | Default | Production rule | | ---------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NEXT_PUBLIC_OPENOPPS_TELEMETRY_ENABLED` | disabled | Enables the browser client only when explicitly set. | | `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY` | unset | Optional browser PostHog key for masked session replay; ignored unless browser telemetry is enabled. | | `NEXT_PUBLIC_OPENOPPS_POSTHOG_RECORDING` | disabled | Explicit replay opt-in. The PostHog client is initialized with session recording disabled unless this is `true`. | | `NEXT_PUBLIC_OPENOPPS_POSTHOG_HOST` | `https://us.i.posthog.com` | Optional browser PostHog ingestion host. | | `OPENOPPS_TELEMETRY_SINK` | `noop` | Use `local-event-lake` only when persistent storage is available. | | `OPENOPPS_TELEMETRY_DIR` | unset | Required for `local-event-lake`; events are partitioned by UTC date. | | `OPENOPPS_TELEMETRY_SALT` | test/dev fallback | Required before `local-event-lake` accepts writes; rotate through deployment config, not source. | | `OPENOPPS_TELEMETRY_IP_MODE` | `hash` | `raw` is rejected when `NODE_ENV=production`; use `hash` or `drop`. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_MAX` | `120` | Public POST limit per salted request fingerprint. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_WINDOW_MS` | `60000` | Rate-limit window in milliseconds. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_MAX_BUCKETS` | `4096` | Maximum in-process rate-limit fingerprints retained before oldest buckets are evicted. | | `OPENOPPS_TELEMETRY_TRUSTED_PROXY` | `none` | Valid values are `none`, `cloudflare`, `vercel`, and `forwarded`. Client IP headers are ignored unless a mode is selected; `cloudflare` trusts `cf-connecting-ip`, `vercel` trusts `x-vercel-forwarded-for`, and `forwarded` trusts the first `x-forwarded-for` hop only when your ingress strips client-supplied values. `x-real-ip` is not trusted by this route. | | `OPENOPPS_POSTHOG_PROJECT_API_KEY` | unset | Optional PostHog project key for sanitized hosted product analytics. | | `OPENOPPS_POSTHOG_HOST` | `https://us.i.posthog.com` | Optional PostHog ingestion host. | | `OPENOPPS_POSTHOG_TIMEOUT_MS` | `1500` | Best-effort PostHog forwarding timeout, clamped to 100-10000 ms. | ## OpenOppsDB Kaggle [#openoppsdb-kaggle] The generated Kaggle bundle lives in `kaggle/`. It contains `dataset-metadata.json`, `kernel-metadata.json`, the connected `openoppsdb-manager.ipynb` notebook, public example notebooks under `kaggle/starter/` and `kaggle/examples/`, and generated SQLite/CSV/Parquet artifacts when a data DB is supplied. `dataset-metadata.json` is the Kaggle UI source of truth for public file descriptions and field-level descriptors for every CSV and Parquet export. `openoppsdb.sqlite` remains directly readable and carries `openopps_tables`/`openopps_columns` metadata for SQLite clients; if Kaggle does not expose nested SQLite table previews for a fresh upload, use the mirrored CSV/Parquet exports for Kaggle-rendered table previews and field metadata. Full snapshot runs write private `sync_metrics.json`, `status.json`, `coverage.json`, and `snapshot-quality.json` evidence while validating the bundle, then prune those files before publishing. The live dataset recipes stage a temporary upload directory containing only Kaggle dataset control files plus `openoppsdb.sqlite`, `exports/csv/*.csv`, and `exports/parquet/*.parquet`; notebooks are pushed as separate Kaggle kernels. Run the local non-live generation and validation path from the repository root: ```bash just kaggle-meta PYTHONPATH=scripts uv run python -m openopps_kaggle --data-db kaggle/openoppsdb.sqlite just kaggle-bundle-check kaggle/openoppsdb.sqlite ``` For a fresh local SQLite bundle seed, initialize a **clean-schema** DB (never a legacy root ledger with `sources.enabled`) and run a bounded jobs sync before bundle generation. This is a faster local maintainer path, not the manager contract: the scheduled manager runs the full `openopps sync --metrics-json` pipeline under its notebook time budget. ```bash OPENOPPS_DB_URL="sqlite:///$PWD/.tmp/openoppsdb-operational.sqlite" uv run openopps admin db init OPENOPPS_DB_URL="sqlite:///$PWD/.tmp/openoppsdb-operational.sqlite" uv run openopps jobs sync --metrics-json --freshness-seconds 86400 --limit 120 PYTHONPATH=scripts uv run python -m openopps_kaggle --data-db .tmp/openoppsdb-operational.sqlite ``` The manager notebook should be scheduled in Kaggle as one daily full-snapshot attempt, for example `0 6 * * *`. Each run reads public snapshot input from `wyattowalsh/openoppsdb`, copies the verified `openopps_kaggle` runtime package from the private `wyattowalsh/openoppsdb-manager-runtime` input, copies the newest `/kaggle/input/**/openoppsdb.sqlite` ledger into `/kaggle/working/openoppsdb/openoppsdb.sqlite`, may restore large columns from prior Parquet exports when upgrading legacy thin snapshots, rehydrates the public SQLite snapshot into a fresh operational Alembic schema when needed, and then runs `openopps sync --metrics-json` (packaged catalog sources, boards, and jobs) under a 6000s budget. It writes private status and coverage evidence, runs `python -m openopps_kaggle` to backfill derived tables, export parquet-first artifacts, regenerate metadata, prune private evidence, and stage the exact public upload, then publishes and reads back one immutable version through the publication ledger. Timeout recovery is an explicit bounded partial-publication policy. The manager may continue only when the timed invocation completed at least one fresh, non-example, successful authoritative run with positive durable job evidence and matching observations. Reconstructed metrics retain provider failures and add `partial_sync_timeout`; without qualifying evidence, the timeout fails closed. Because the current quality policy can publish that degraded snapshot with warnings, `partial_sync_timeout` means incomplete route coverage and must not be interpreted as a full daily-crawl attestation. Keep the scheduled manager as the sole writer to its working SQLite copy: recovery uses an append-only rowid/time window, not a cross-process invocation identifier, and a concurrent sync against the same file can invalidate attribution. The manager has no mutable OpenOpps package default. Set `OPENOPPS_PACKAGE_SPEC` to `git+https://github.com/wyattowalsh/openopps.git@` as a Kaggle notebook secret or environment variable (or let `just kaggle-notebook-push execute=1` bake the current `HEAD` SHA into a temporary push copy); branches, tags, ranges, and unpinned packages fail before installation. The generated notebook also pins the canonical runtime-package digest through `OPENOPPS_RUNTIME_PACKAGE_SHA256` (currently `2c73c33becdb0765f5f3e890bc92af6ac56b9c9bf1cfe9cc88ea9a9ae3353819`), which must match the private runtime dataset's `runtime-manifest.json`. Regenerate the manager notebook and refresh the private runtime dataset together whenever that package changes. Scheduled rehydrate accepts a pre-`0004` public `job_sync_runs` table by inserting the lifecycle columns from `synced_at` / `success` / `error`. The snapshot quality gate does not hard-block empty derived skill tables when every `job_versions.skills` value is null or `[]`. The notebook seeds bounded Kaggle runtime defaults for source freshness, job-route freshness, concurrency, connection limits, timeouts, and retries while allowing `OPENOPPS_` environment overrides. Live file/column metadata repair is not part of dataset publication or immutable-version readback. After a verified live publish, run `just kaggle-live-file-metadata` separately from a browser-authenticated maintainer environment when the Kaggle DataBundle checklist or column-description score needs authoritative repair, and record its outcome independently. The scheduled Kaggle notebook environment must expose Kaggle API credentials before the manager starts. Configure `KAGGLE_USERNAME` and `KAGGLE_KEY`, or `KAGGLE_API_TOKEN`, as Kaggle notebook secrets/environment variables; otherwise the manager fails fast before running the expensive sync. Live deployment remains a local maintainer action with Kaggle CLI credentials. Every create, version, and kernel-push recipe is dry-run by default. Create/version recipes rebuild from `db=` before staging so a stale `kaggle/` tree cannot silently ship (`allow_stale=1` is a loud override only): ```bash # Review dry-run ledgers first. Replace 42/7 with the exact live versions. just kaggle-dataset-version message="OpenOppsDB daily snapshot" db=.tmp/openoppsdb-operational.sqlite expected_current_version=42 just kaggle-runtime-generator-version message="OpenOppsDB manager runtime generator" expected_current_version=7 just kaggle-notebook-push just kaggle-example-notebooks-push # Execute only after reviewing those plans. just kaggle-dataset-version message="OpenOppsDB daily snapshot" db=.tmp/openoppsdb-operational.sqlite expected_current_version=42 execute=1 just kaggle-runtime-generator-version message="OpenOppsDB manager runtime generator" expected_current_version=7 execute=1 just kaggle-notebook-push execute=1 just kaggle-example-notebooks-push execute=1 just kaggle-example-notebooks-status just kaggle-example-notebooks-pull-check just kaggle-live-verify ``` The public example notebooks are generated from the repo source of truth and pushed as separate Kaggle kernels: `wyattowalsh/openoppsdb-starter-notebook`, `wyattowalsh/openoppsdb-advanced-usage`, `wyattowalsh/openoppsdb-hiring-market-map`, and `wyattowalsh/openoppsdb-skills-radar`. They are read-only, internet-disabled, credential-free, and attached only to `wyattowalsh/openoppsdb`. After pushing, run `just kaggle-example-notebooks-pull-check` to pull and verify the live source bundles; `just kaggle-example-notebooks-files page_size=200` lists output files emitted by those notebook runs. Use `just kaggle-dataset-create db=` only to prepare the first public upload and `just kaggle-runtime-generator-create` only to prepare the first private manager-runtime upload. After inspecting the dry-run ledger, a first create requires both `execute=1` and `allow_no_rollback=1`; this is the explicit acknowledgment that no immutable rollback target exists yet: ```bash just kaggle-dataset-create db=.tmp/openoppsdb-operational.sqlite just kaggle-dataset-create db=.tmp/openoppsdb-operational.sqlite execute=1 allow_no_rollback=1 just kaggle-runtime-generator-create just kaggle-runtime-generator-create execute=1 allow_no_rollback=1 ``` Subsequent version writes require `expected_current_version=` and `execute=1`; the live preflight must observe that exact version, and the ledger preserves it as the rollback target. The live write path stages a temporary upload directory, requires Kaggle CLI credentials, verifies the new immutable version by exact remote readback, and remains intentionally outside CI. `just kaggle-bundle-smoke` is the non-secret clean-DB stage smoke used for local/CI confidence. `just kaggle-notebook-push` defaults to a two-hour Kaggle kernel timeout (sync plus publication readback can exceed one hour) and renders only a plan unless `execute=1` is supplied. `just kaggle-example-notebooks-push` still defaults to one hour. Override `timeout` only for a deliberate longer maintenance run. ## CI/CD [#cicd] GitHub Actions mirrors local validation instead of inventing a second build path. CI should stay least-privilege, cache-aware, and split into independently debuggable lanes: * Python package tests and coverage through `uv`. * Lock-file consistency through `uv lock --check`. * OpenSpec strict validation for every active change. * Offline source-discovery schema, fixture, bundle, skill-eval, private-envelope, accounting, and promotion-preview gates with discovery network disabled. * Web generated data, MDX/type-check, build, tests, browser checks, and lint through `pnpm`. * Kaggle metadata generation for exported schema changes. * Dependency review on pull requests, supported Python 3.12/3.13/3.14 lanes, lowest-direct dependency testing, and network-dependent security audits. * A non-pull-request supply-chain lane that builds the Python wheel, generates its SPDX SBOM, attests the wheel subject, and retains the workflow artifact for 30 days. The supply-chain lane does not build, attest, restore-test, or publish the v7 public-data recovery archive. That remains a separate production-readiness gate. The broader `rtk lint` docs checklist remains an explicit maintainer recipe (`just web-rtk-lint`) rather than a conditional CI step. When CI changes, update `Justfile`, this operations page, root README, and repo instructions in the same logical change. ## Troubleshooting [#troubleshooting] | Symptom | Check | | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | A source sync returns zero boards | Run `uv run openopps admin sources test --page-size 5` and check provider health. | | A board has provider hints but no jobs | Run `admin providers probe-routes` and inspect unknown route candidates. | | A new source has low yield | Run `uv run openopps admin sources yield --source ` after route probing and job sync. | | Cached data looks stale | Rerun the operation with `--refresh-cache` or purge the relevant cache namespace. | | Local DB reports missing v0.1 columns | Reset the pre-release local DB file or set `OPENOPPS_DB_URL` to a new SQLite path, then rerun `admin db init`. | | Workday sync is slow | Lower `OPENOPPS_WORKDAY_CONCURRENCY` and narrow sync with `--source`, `--board`, or `--provider workday`. | | Export output is unexpectedly empty | Confirm the database has data with `uv run openopps admin db status` and inspect filters. | | Docs navigation is stale | Update `web/content/docs/meta.json`, then run `cd web && pnpm types:check`. | | `just source-policy-audit` exits 2 | Expected while 688 sources remain blocked pending written Getro/Consider grants. Structural `just source-policy-check` staying green is not eligibility. | | `discovery verify-scout` reports stale | Evaluation bundles use a 48-hour `max_evidence_age`. Re-scout; this is not a promotion rollback. | | A source-scout suggestion looks approved | The skill is advisory. Acceptance is only `launch_isolated_scout` / `validate_fixture.py`. Do not live-install harness projections. | # Providers (/docs/providers) OpenOpps v0.1 separates aggregate source adapters from job-capable provider adapters. Source adapters discover company boards; provider adapters detect or fetch postings from public board providers through the CLI and local SQLite state. Python plugins can contribute additional source adapters and job providers through the `openopps.plugins` entry point group. Plugin capabilities appear in `admin providers list` and registry surfaces when loaded successfully, and load failures, disabled entries, allow-list filters, and conflicts are visible in `plugins list`. Installed plugins are discovered by default but only executed when their entry-point names are listed in `OPENOPPS_PLUGIN_ALLOWED`, unless `OPENOPPS_PLUGIN_AUTOLOAD=true` is set. Plugin job providers implement `BoardJobProvider` and return `openopps.providers.JobFetchResult` from `fetch_jobs`. Mark a result `authoritative=True` only after a complete, verified traversal: authoritative snapshots may close jobs that were not returned, while partial and plain-list results are rejected fail closed. The minimal plugin template demonstrates a non-authoritative no-op provider. Use `examples/plugins/minimal-openopps-plugin/` as the starting template for a packaged plugin with a `pyproject.toml` entry point. ## Source Adapter Inventory [#source-adapter-inventory] ## Source Catalog Sample [#source-catalog-sample] ## Non-VC Source Families [#non-vc-source-families] OpenOpps packages a small set of low-friction non-VC source families as source adapters, not job providers. These sources add company candidates and provenance metadata; they only become job-yielding after route detection, route probing, and provider job syncs find public job-capable routes. | Source family | Packaged keys | Notes | | ----------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------- | | Official public-company index | `sec-company-tickers` | Included listed-company backbone; SEC fair-access controls can reject generic scheduled sync environments. | | Public index CSV | `sp500`, `nasdaq100` | Included seed data for index membership; `nasdaq100` remains manual until a reviewed CSV is configured. | | Employer ranking CSV | `fortune500` | Included ranking seed data; supports embedded user-supplied CSV rows for reviewed local refreshes. | | Ecosystem landscape | `cncf-landscape` | Reads CNCF `landscape.yml` public fields and intentionally excludes logos and Crunchbase-derived fields. | Each packaged source may carry taxonomy metadata in `raw_metadata`. Discovery promotion requires the eight required fields below as non-empty strings; packaged catalog values are not a promotion grant. ## Required Discovery Taxonomy [#required-discovery-taxonomy] Discovery candidates use `CandidateTaxonomy` (`src/openopps/discovery/data/candidate-taxonomy.schema.json`) and `REQUIRED_TAXONOMY_FIELDS` in `src/openopps/discovery/identity.py`. `validate_taxonomy` requires all eight required fields as non-empty strings. `sourceYear` is optional (four-digit integer 1900–9999 when present). Unknown field names fail closed. | Field | Required | | ------------------- | -------- | | `providerType` | yes | | `coverageMode` | yes | | `accessType` | yes | | `licenseStatus` | yes | | `refreshCadence` | yes | | `sourceCategory` | yes | | `sourceAttribution` | yes | | `inclusionReason` | yes | | `sourceYear` | no | Values may remain null while quarantined. `validate_taxonomy` reports `complete` only when every required field is a non-empty string. Missing or blank required fields leave taxonomy `incomplete`. Incomplete taxonomy cannot promote: evaluation will not mark the candidate `promotable`, and `revalidate_selected_candidates` in `src/openopps/discovery/promotion.py` raises `PromotionPreviewError` (`selected candidate taxonomy is incomplete`). These counts are frozen before-state evidence from barrier B000 (commit `8e3c797b975a1f79844c1906e96c0993d88ab1f1`), not current live catalog totals. At that baseline, 895 records have exactly 8/8 required taxonomy values, 1,975 have 0/8, no record is partially populated, and `sourceYear` is present on zero records. | Frozen taxonomy baseline | Count | | ---------------------------------------- | ----- | | Complete required taxonomy (exactly 8/8) | 895 | | No standard taxonomy fields (0/8) | 1,975 | | Partially populated records | 0 | | `sourceYear` present | 0 | Future promotion is expected to change counts. Do not treat 895/1,975 as a current production inventory. See [CLI](/docs/cli#quarantined-source-discovery) for scout, verify, and preview command strings. ## Support Levels [#support-levels] | Level | Meaning | | ------------- | ----------------------------------------------------------------------------------------- | | `detect` | OpenOpps can preserve provider metadata and route hints but does not reliably fetch jobs. | | `jobs` | OpenOpps can fetch public postings for the provider. | | `unsupported` | The provider is known only as raw upstream metadata. | ## Job-Capable Providers [#job-capable-providers] ## Detect-Only Providers [#detect-only-providers] `manatal` and `gem` are preserved as board metadata until stable public fetching is added. Health reports group these under `notCovered` so they are visible without being treated as failed job syncs. ## Startup-board source scope (v0.1) [#startup-board-source-scope-v01] YC is the preferred packaged startup-board source (`yc` / `ycombinator` via the public Algolia-backed companies index). WorkAtAStartup is intentionally **not** packaged; it duplicates that discovery surface without a better public no-auth path. Wellfound and Angel List startup discovery are **unsupported** for v0.1: public discovery depends on session or anti-bot protected pages rather than stable static no-auth assets or approved search-index endpoints. OpenOpps does not ship a Wellfound/Angel source adapter. Provider coverage JSON includes `gaps.sourceScope.unsupportedSourceDiscovery` with the release rationale. Consider-backed sources may emit `Editorial` or misspelled `Editiorial` `job_sources` labels without a generic public ATS route. OpenOpps keeps those hints as detect-only metadata and does not register an `editorial` job provider until route-probe evidence proves a repeatable public fetch path. Audit notes live in `openspec/changes/archive/2026-07-13-provider-source-scope-hygiene/editorial-label-audit.md`. ## Public No-Auth Board Providers [#public-no-auth-board-providers] Workable, Teamtailor, BambooHR, Rippling, and WP Job Manager are job-capable in v0.1 only through public unauthenticated board routes. Workable fetches list and per-job detail endpoints separately so `raw_listing` and `raw_detail` stay distinct for audit replay. BambooHR uses public careers endpoints such as `/careers/list` and `/careers/{job_id}/detail`; OpenOpps does not call authenticated BambooHR ATS APIs. WP Job Manager requires an explicit `/wp-json/wp/v2/job-listings` or `/jm-ajax/get_listings/` endpoint and is not inferred from arbitrary WordPress sites. ## Surplus field promotion [#surplus-field-promotion] Greenhouse list responses with `content=true` promote `metadata`, `requisition_id`, `language`, and department/office hierarchy into `provider_extras`; prospect posts without `internal_job_id` are tagged `posting_kind=prospect`. See the S1–S4 surplus taxonomy in [Data Model](/docs/data-model#ingest-surplus-taxonomy-s1s4) and `openspec/changes/ingest-data-surplus/` for the full promotion manifest. | Provider | Shipped in v0.1 | Planned (manifest) | | -------------- | --------------------------------------------------------------------------------- | ----------------------------------------- | | Greenhouse | `metadata`, `requisition_id`, `language`, department/office trees, `posting_kind` | `pay_input_ranges` (optional N+1 fetch) | | Workable | `raw_listing` / `raw_detail` split | — | | Lever | — | `categories`, epoch dates, `sections` | | Ashby | — | `isListed`, compensation, `workplaceType` | | BambooHR | — | `jobOpening`, requisition ids | | Workday | — | `postedOn`, `jobDescription` | | Rippling | — | `payRangeDetails`, `workplaceType` | | Teamtailor | — | RSS limits; optional HTML detail fetch | | WP Job Manager | — | meta keys | ## Provider Coverage [#provider-coverage] Provider coverage reports on the persisted SQLite dataset only. It does not perform live HTTP checks, source fetches, route probes, or job syncs. The deterministic smoke data proves the report shape; published real-world percentages must be measured from representative persisted source snapshots: ```bash uv run openopps providers coverage uv run openopps providers coverage --source a16z --provider any --json uv run openopps providers coverage --source a16z --provider greenhouse uv run openopps providers audit --source a16z --json uv run openopps admin sources yield --json ``` Use coverage to answer whether the local data is complete enough for analysis. The JSON report includes source, board, route, and job counts; route counts by provider, support level, and last status; executable route counts; missing route metadata counts; duplicate route skips derived from the durable route registry; job counts by provider, source, and board; non-supported provider coverage; detect-only provider examples; boards with job-capable hints but no executable route; and boards with executable routes but zero persisted jobs. `admin sources yield` reports source-family conversion from persisted records only: company candidates, canonical boards, provider hints, job-capable routes, route-ready routes, active job routes, duplicate board rate, active boards added, yield score, and taxonomy totals by provider type and access type. Coverage also reports enrichment completeness percentages from deterministic provider-field mapping. These data-quality metrics cover posting URLs, apply URLs, locations, departments, descriptions, normalized compensation or salary, remote level, and employment type. ## Route Probing [#route-probing] Some sources report provider hints without the public route token required for job fetching. Route probing tries candidate tokens derived from upstream slugs, remote ids, names, domains, and websites. Board keys such as `a16z:acme` are durable record identifiers, not provider route-token candidates.
Candidate tokens Derived from source slugs, remote ids, names, domains, and websites.
Dry-run first Route probing reports matches and unknowns without persistence by default.
Apply deliberately Add --apply only after inspecting matched route metadata.
```bash uv run openopps admin providers probe-routes --source a16z --provider any --limit 25 --json ``` By default, probing only checks routes missing token or URL metadata and does not persist results. Use `--include-existing` to recheck existing routes and `--apply` to persist matched metadata. ## Provider Health [#provider-health] Provider health samples source adapters and job-capable routes, then reports status counts: ```bash uv run openopps providers health --source a16z --provider any --limit 25 --json ``` Health checks report `active`, `empty`, `error`, `missing_route`, `not_covered`, and duplicate route skips. Add `--apply` to persist source health under `raw_metadata.health` and board-provider route health under `last_status`. Use provider health for live sampled HTTP status. Use provider coverage for offline persisted route coverage and enrichment quality. Use provider audit for candidate-provider adoption evidence. ## Board Route Registry [#board-route-registry] The `board_providers` registry is the executable route layer between discovered boards and job sync. Use it before large syncs to confirm which routes are ready: ```bash uv run openopps admin providers registry --provider any uv run openopps admin providers registry --provider any --passed-probe-only --json uv run openopps admin providers registry --provider any --include-missing --limit 50 ``` Without `--include-missing`, `admin providers registry` skips job-capable hints that still lack executable metadata, such as an Ashby board token or complete Workday CXS route. `--passed-probe-only` narrows output to routes verified by a persisted `admin providers probe-routes --apply` result. Use [Explorer](/explorer) to inspect the generated static snapshot of persisted boards, board-provider routes, latest job rows, source/provider coverage, and data-quality signals. ## Provider Limits [#provider-limits] * Workday support is limited to public postings visible through careers sites; it is not official tenant API access. * Ashby sync excludes `isListed: false` direct-link-only postings from normal output. * Overlapping board records from multiple sources are merged by company domain. Board JSON includes `source_keys` and `source_board_keys` for every source currently represented by the canonical board, and provider requests are deduped before route probes and job syncs. * Installed Python plugins are not sandboxed and run in the same process as OpenOpps. * Use `OPENOPPS_PLUGIN_ALLOWED` to opt in trusted entry-point names before plugin code executes. * Use `OPENOPPS_PLUGIN_AUTOLOAD=true` only in controlled environments where every installed `openopps.plugins` entry point is trusted. # Public Data Releases (/docs/public-data-releases) OpenOpps can build a version 7 public-data publication without changing the legacy version 6 tree. Version 7 is content-addressed: a small mutable channel pointer selects one immutable release, and every consumer verifies that release before reading its assets. This is an operator and governance contract, not a deployment announcement. The repository contains local generation, verification, staging, rollout-plan, remote-readback, and recovery-archive tooling. A successful live Workers rollout, production-snapshot rights approval, and a GitHub Release attestation for the public-data archive have **not** been established by those local tools or by the current CI workflow. ## Publication layout [#publication-layout] ```text publication-root/ ├── channels/ │ └── production.json └── releases/ ├── / │ ├── manifest.json │ ├── search-manifest.json │ ├── publication-policy.json │ └── ...verified search assets... └── / └── ...independently verified prior release... ``` `releaseId` is the lowercase SHA-256 digest of the canonical manifest body. The release directory name, `releaseId`, and `rootDigest.value` must agree. `manifest.json` is self-excluding; its `files` array closes over every other payload file and records each safe relative path, byte length, media type, SHA-256, semantic role, and semantic count. The manifest also records: * canonical UTC `snapshotAt`; * SQLite input path, byte length, and SHA-256; * generator name, entry point, payload schema version, and component digests; and * exact `fileCount` and `totalBytes` values. Validation rejects missing, extra, duplicate, case-colliding, symlinked, non-regular, unsafe, oversized, or hash-mismatched files. JSON is decoded strictly, private payload keys such as `payloadSnapshots` and `descriptionHtml` are forbidden, and secret-like keys and credential-bearing values fail the release. ## Channel schema version 2 [#channel-schema-version-2] `channels/production.json` has exactly these fields: | Field | Contract | | -------------------- | ---------------------------------------------------------------------------------------- | | `schemaVersion` | Integer `2`. | | `channel` | Safe lowercase channel name, normally `production`. | | `releaseId` | Current immutable release SHA-256. | | `rootDigest` | `{ "algorithm": "sha256", "value": releaseId }`. | | `snapshotAt` | Canonical UTC snapshot time copied from the release manifest. | | `manifestPath` | Exactly `releases//manifest.json`. | | `priorReleaseId` | Distinct previous release SHA-256, or `null` before the first transition release exists. | | `degradedReason` | Auditable stale-snapshot reason, otherwise `null`. | | `promotedAt` | Canonical UTC pointer promotion time. | | `snapshotAgeSeconds` | Non-negative difference between `promotedAt` and `snapshotAt`. | The pointer is written with a sibling temporary file and atomic replacement. A repeated build that selects the same release preserves its prior release identity instead of rotating it away. The static delivery stage is stricter than a first publication: it requires two distinct, valid releases and contains **exactly** the current and prior release trees plus `channels/production.json` and `_headers`. A first v7 build with `priorReleaseId: null` can be verified, but it cannot satisfy the dual-release delivery gate. ## Freshness and degraded operation [#freshness-and-degraded-operation] Ordinary v7 generation rejects a snapshot older than 48 hours: ```bash uv run python scripts/generate_docs_search_index.py \ --data-db kaggle/openoppsdb.sqlite \ --release-root /absolute/path/to/openopps-search-v7 \ --channel production \ --max-snapshot-age-hours 48 uv run python scripts/verify_docs_search_artifacts.py \ --root /absolute/path/to/openopps-search-v7 \ --channel production \ --max-snapshot-age-hours 48 ``` An incident owner may explicitly allow stale data with `--allow-stale-reason`. The non-empty reason and computed age become public channel metadata. The override bypasses only the age limit: source rights, attribution, privacy, secret scanning, exact-set integrity, provenance, and file/platform budgets remain non-bypassable. ```bash uv run python scripts/generate_docs_search_index.py \ --data-db kaggle/openoppsdb.sqlite \ --release-root /absolute/path/to/openopps-search-v7 \ --channel production \ --allow-stale-reason "Upstream incident INC-123; approved by release owner" ``` Do not use a vague reason such as “temporary.” Record an incident or change identifier, approving role, observed snapshot age, and the planned refresh time in the external release ledger. A degraded release is not evidence that the data is current. ## Source rights and attribution [#source-rights-and-attribution] `publication-policy.json` is generated from the sources that actually occur in the snapshot. Publication is fail-closed: | `licenseStatus` | Publication behavior | | ------------------------------------------- | ------------------------------------------------- | | `official_public` | Allowed. | | `oss_attribution_required` | Allowed only when `sourceAttribution` is present. | | `public_attribution_required` | Allowed only when `sourceAttribution` is present. | | Missing, `needs_review`, or any other value | Rejected. | For a packaged source, repository metadata is the positive inclusion boundary, but it is not independent legal permission. A persisted database row cannot grant a packaged source rights that the catalog does not grant. Persisted metadata is considered only for an explicit user or plugin source that is absent from the packaged catalog. Only the sanitized source key, allowed rights state, required attribution, and safe source URL enter the policy report; arbitrary stored metadata does not. The separate source-policy evidence models license, access, redistribution, synchronization, and publication independently. It is a deny-only overlay: provider-scoped and exact-source denials supersede catalog or stored metadata, while an uncovered source receives no grant. `publication-policy.json` records the policy ID and review date, and its policy module, evidence, schema, and reference-corpus SHA-256 values must match generator components hashed into the v7 release ID. Run `just source-policy-check` for the structural CI contract and `just source-policy-audit` for release eligibility. The current committed-v6 policy is structurally valid but release-ineligible: 7 of 1787 sources only mirror repository catalog declarations, 0 are independently verified, and 1780 are blocked. The audit exits 2, so do not render a selector, generate or upload a production corpus, bootstrap a Worker, or publish it. The report also records snapshot-quality counts and detail-tier counts. Review the report itself before any upload. Fix source metadata at its canonical source and regenerate; never hand-edit the generated policy report or a release directory. Passing policy-unit tests proves the gate's behavior, not the rights status of the current production data. Rights readiness remains open until the exact real snapshot intended for upload generates and verifies without blocked sources. ## Privacy, correction, and takedown procedure [#privacy-correction-and-takedown-procedure] Public search releases intentionally exclude raw provider payload snapshots and HTML descriptions. T2 details contain bounded normalized plain text. Telemetry is a separate, opt-in surface and must not be joined into a public-data release. Report a correction, attribution concern, or takedown request through the repository issue tracker. If the request contains personal or sensitive information, email `openopps@wyattowalsh.com` instead and include only the source URL and identifiers needed to locate the record. The release owner must: 1. Record the request time, affected source/job identifiers, requested action, and decision without copying unnecessary personal data into the ledger. 2. Pause publication for the affected source while rights or accuracy are unresolved. A takedown request is a fail-closed rights state, not a reason for a degraded freshness override. 3. Correct the canonical source catalog, ingestion normalization, or source data; then rebuild from a clean SQLite snapshot. Never mutate an immutable release in place. 4. Produce two distinct verified releases that both exclude the withdrawn content before staging the replacement dual-release Worker version. This prevents the served prior release from retaining the affected record. 5. Promote the replacement, verify every remote file and critical web route, and confirm the withdrawn identifiers are absent from both served releases. 6. Remove or quarantine superseded recovery archives containing withdrawn data according to the request and record the archive digests affected. Preserve only non-content audit metadata needed to prove the action. The tooling verifies artifacts but does not automate request intake, legal review, Worker-version deletion, CDN purge, or GitHub Release deletion. Treat those actions as explicit operator steps and preserve their external request/deployment IDs. ## Retention boundary [#retention-boundary] * Each staged Worker asset tree retains exactly two releases: current and previous. Extra releases fail verification. * Each recovery archive closes over the same dual-release tree. Retain the previous-good stage or its verified archive until promote, rollback, re-promotion, and restore checks for the successor complete. * Candidate directories and Wrangler machine-output/ledger files are local operational evidence. Store them outside Git, protect them from secrets, and remove them under the maintainer retention policy after evidence is recorded. * The legacy v6 tree remains in Git during the bounded transition. Its removal is a normal follow-up commit only after every cutover exit gate passes. * A Git history rewrite is not normal retention. It requires separate explicit approval and its own protected backup, full ref inventory, old-to-new SHA mapping, freeze window, fresh-clone validation, recovery instructions, and force-push authorization. ## Release-pinned web consumers [#release-pinned-web-consumers] Set the server and browser origins/channels to the same publication: ```dotenv OPENOPPS_PUBLIC_DATA_ORIGIN=https://openopps-data-production..workers.dev OPENOPPS_PUBLIC_DATA_CHANNEL=production NEXT_PUBLIC_OPENOPPS_PUBLIC_DATA_ORIGIN=https://openopps-data-production..workers.dev NEXT_PUBLIC_OPENOPPS_PUBLIC_DATA_CHANNEL=production ``` The server values configure details, metadata, and sitemaps. The `NEXT_PUBLIC_` values are embedded into the browser build and configure Jobs/Explorer search. Production server origins must be HTTPS and host-allowlisted. The browser and server resolve the mutable channel once and then read the immutable release. Every fetched v7 JSON asset must appear in the manifest and match its declared byte length and SHA-256. When no channel is configured, the app uses the legacy v6 tree as a bounded transition fallback. The production `/api/jobs/search` compatibility route returns `410 browser_worker_required`; it does not scan the full corpus. Browser search runs in a dedicated release-pinned Web Worker using the accepted columnar postings/bitset design documented in `web/docs/adr/0001-browser-jobs-search-engine.md`. Saved jobs, saved searches, and retained details are browser-local. IndexedDB mutations commit before visible state advances, replacement imports are validated and transactional, and at most three bounded pre-import backups are retained. The browser now implements an explicit, default-off offline-search installer for v7. It selects only the bounded search/metadata roles, enforces deterministic file/byte caps and two-times-size quota headroom, verifies manifest bytes plus every selected file before recording readiness, pins one immutable release, preserves the prior verified cache until replacement succeeds, and deletes only OpenOpps-owned caches on opt-out. Unit tests assure opt-out, quota, integrity failure, partial-write rollback, replacement, tamper detection, and ownership behavior. That implementation is not deployed offline-readiness evidence. A real deployed v7 release still must pass install, offline disconnect/readback, update/rollback, and complete Chromium/Firefox/WebKit journeys. Until those results are recorded for the exact release and web revision, report the feature as locally implemented and unit-assured, with the production offline gate still open. ## Delivery runbook [#delivery-runbook] The detailed operator sequence is maintained in `deployment/openopps-data/README.md`. Its safe local phases are: ```bash uv run python scripts/docs_search_delivery.py validate-config deployment/openopps-data uv run python scripts/docs_search_delivery.py stage \ /absolute/path/to/openopps-search-v7 \ deployment/openopps-data/staging/assets uv run python scripts/docs_search_delivery.py verify-stage \ deployment/openopps-data/staging/assets ``` The script will stage only to the owned `deployment/openopps-data/{staging,production}/assets` paths. It renders upload and rollout commands but never executes Wrangler. Live commands require a separately authorized maintainer session with the exact account, Worker, previous-good version, and rollback evidence established first. If a target Worker is freshly proven absent, `scripts/docs_search_bootstrap.py` and `deployment/openopps-data/BOOTSTRAP.md` are the sole dry-run-first bootstrap exception: they bind the frozen candidate to the intended account/name, use pinned Wrangler for one initial deploy, require exact version/deployment readback, and record that initial version as `rollbackWorkerVersionId`. Tooling presence is not live bootstrap evidence. Cloudflare credentials must remain outside Git and logs. For non-interactive Wrangler authentication, the relevant names are `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID`; do not put values in `.env.example`, generated output, command history, or issue comments. Prefer a narrowly scoped API token over a global API key. The checked-in staging and production configs are distinct assets-only Workers on `workers.dev`. Their contract forbids a script entry point, bindings, `run_worker_first`, preview URLs, and metrics. Release assets receive one-year immutable caching; channel pointers revalidate; every asset receives CORS, `nosniff`, and `noindex` headers. Remote verification reads every file and checks status, exact bytes, SHA-256, headers, a non-empty ETag, redirect absence, and a deterministic missing-path 404. Cloudflare documents that Workers versions separate upload from traffic deployment and that a deployment can atomically route 100% to one version. The Free-plan static-assets limit remains 20,000 files per Worker version with an individual asset limit of 25 MiB. OpenOpps enforces a conservative 20,000-file and strict-less-than-24-MiB stage budget before upload. See the official [versions and deployments](https://developers.cloudflare.com/workers/versions-and-deployments/), [Wrangler configuration](https://developers.cloudflare.com/workers/wrangler/configuration/), and [static-asset limits](https://developers.cloudflare.com/changelog/post/2025-09-02-increased-static-asset-limits/) references. ## Archive and attestation status [#archive-and-attestation-status] `scripts/docs_search_delivery.py bundle` creates one deterministic `openopps-data-.tar.gz` containing the staged dual-release assets, `SHA256SUMS`, `bundle-manifest.json`, `sbom.spdx.json`, and `provenance.json`. The filename addresses the exact bytes, including provenance time and source revision; the release tag separately addresses the stage-root digest. `just public-data-archive-restore` requires the independently recorded archive SHA-256, stage-root digest, source revision, and current/prior release IDs. It enforces bounded regular members and a 4-GiB expanded-byte ceiling, streams through exclusive no-follow creates into a private sibling candidate, validates checksum plus bundle/provenance/SPDX semantics, reruns the complete stage verifier, and uses OS-native no-replace rename so a concurrent destination cannot be overwritten. The ordinary GitHub Actions `supply-chain` job remains wheel-only. Public-data recovery uses the separate manual `public-data-archive.yml` workflow. An authorized maintainer must first enable immutable releases and create a non-latest draft for `openopps-data-v7-` whose only asset is the exact content-addressed archive and whose target is the current `main` source SHA. The workflow itself cannot create the draft or upload/replace/delete an asset. The workflow first checks the repository immutable-release setting and draft identity with `contents: read`, freshly downloads and restores the asset, and creates an SPDX v2.3 attestation with only the additional `id-token: write` and `attestations: write` permissions. A separate `contents: write` job re-downloads and verifies that attestation before publishing with `latest=false`. A fresh read-only job then requires the immutable record and exact tag target, runs GitHub release and asset attestation verification, enforces the signer workflow/source SHA/source ref/SPDX predicate, and restores current plus prior again. Workflow presence or draft creation is not release evidence: keep task 5.7 open until an exact promoted-SHA run and immutable-release readback succeed and are recorded. ## Version 6 transition exit criteria [#version-6-transition-exit-criteria] Do not remove `web/public/data/openopps-search/` or disable the v6 reader until all of these are true for one exact source revision and release identity: 1. A fresh, rights-approved v7 current release and a distinct valid prior release pass exact local verification. 2. The full dual-release corpus uploads to the intended Workers **Free** staging target with the checked-in assets-only config; undocumented aggregate limits do not reject it. 3. Staging readback verifies every byte, digest, required header, ETag, redirect behavior, and missing-path 404. 4. The web app uses one v7 release across search, details, metadata, sitemaps, and build checks; production build, routes, unit tests, and browser E2E pass with the full v6 tree absent. 5. Production promotion, remote verification, rollback to the retained previous-good Worker version, verification of that version, re-promotion, and final verification all succeed with exact version/deployment IDs recorded. 6. A content-addressed recovery archive is independently downloaded, checksum/attestation verified, and restore-tested for current and previous releases. 7. Canonical local `just` gates and origin GitHub Actions are green for the exact SHA. Current CI wheel attestation alone is insufficient. 8. Only then may an ordinary commit remove the v6 data tree. Any history rewrite remains a separate destructive project requiring new explicit approval. If any gate fails, keep v6 available, leave production unchanged or roll back to the recorded previous-good version, and report the missing evidence without claiming a live rollout.