okami.mom/web-extract-go

git clone https://git.nkp.pet/okami.mom/web-extract-go

nekomimi.petadmin: show what each request's decisions would cost on Jev8af1784

main
10.1 KiB101 linesraw

web extract

an exa-shaped retrieval service: kagi search via browser rendering and logged in session from cookies, jev reranking and verbatim evidence selection, and LLM generated summaries.

layout

cmd/web-extract/        executable entrypoint
internal/app/          service, http/mcp transports, retrieval, models, and tests
internal/app/testdata/  frozen source fixtures
web/                   shadcn dashboard and embedded production assets
scripts/build          rebuild frontend and go binary
docs/admin.md          dashboard setup and security

build and run

requires go 1.25 and a c compiler for sqlite. pages are rendered with an existing chrome installation; no bundled browser download is required. PDF text is extracted in-process, with no external pdftotext dependency.

browser rendering is stealthy by default. it presents a desktop chrome identity matching the host it runs on, and removes the automation fingerprints that bot walls read: navigator.webdriver, the empty plugin list, the software-rendered WebGL strings, and the Headless_Chromium user agent.

the identity follows the host deliberately. claiming a different platform than the machine actually is — a linux user agent beside an Apple GPU and macOS fonts — describes hardware that cannot exist, and the stealth chromium builds take the same approach for the same reason. client-hints headers (Sec-CH-UA-*) are also left to the browser, because they are opt-in per origin and a browser that volunteers them unrequested is anomalous.

set STEALTH=false in .env to render with the browser's true identity, for example when diagnosing what a page sees. BROWSER_HEADLESS=false launches a visible window, which needs a logged-in desktop session.

a page is rendered in a real browser and converted from the resulting DOM. if the render produces less than 80 non-whitespace characters — a PDF served without a .pdf suffix, or a page that only serves text to non-browser clients — the same URL is retried once with a plain fetch. .pdf URLs skip the browser entirely, since a PDF has no DOM to render.

set PROXY_POOL to a comma-separated list of upstream proxies to route egress through them in rotation, which is the lever for a blocked or rate-limited egress address. one proxy is selected per request so a page, its subresources, and any retry keep the same address.

go test ./...
go vet ./...
go build -trimpath -ldflags='-s -w' -o bin/web-extract ./cmd/web-extract
./scripts/mcp          # codex normally owns this process
./bin/web-extract      # alternatively serve http on 127.0.0.1:8788

copy .env.example to .env for a new installation, fill credentials, and make it owner-readable only (chmod 600 .env). do not overwrite an existing configured .env.

public access

PUBLIC_ACCESS=true serves callers who send no token, for a shared instance (search.nkp.pet). a request with a token is unaffected; a wrong token is still refused.

  • GET / returns a notice explaining the service and its data policy.
  • each client address may make PUBLIC_RATE_PER_MINUTE requests a minute (default 20). PUBLIC_DAILY_LIMIT optionally caps public tool calls a day across all clients (default 0, no cap; reset at midnight UTC; MCP handshakes and listings do not count). refusals are 429 with Retry-After.
  • nothing about a public call is logged unless it opts in, with the header X-Contribute: 1 or by calling /mcp/contribute. opted-in calls go to their own log, PUBLIC_LOG_PATH (default data/public.db, same schema).
  • PUBLIC_JEV_URL / PUBLIC_JEV_KEY / PUBLIC_JEV_MODEL give public calls their own decision model, any server speaking the SystemOne API. cloudflare/clef is a Worker that serves Cloudflare's Clef-Flash this way. PUBLIC_JEV_FALLBACK_URL / PUBLIC_JEV_FALLBACK_KEY / PUBLIC_JEV_FALLBACK_MODEL (default jev-1.13-free) name a second SystemOne model used when the first fails after its retries, for example OpenCode Zen (https://opencode.ai/zen/v1), so a Worker or allocation outage does not take public calls down.
  • LOCAL_DECISION_URL names a local SystemOne-compatible model asked before the remote one (serve/decision.py serves a fine-tuned EmbeddingGemma 2 typed-decision checkpoint). Its answers to the yes/no questions named in LOCAL_DECISION_QUESTIONS (default evidence,useful) are kept when they are at or below LOCAL_DECISION_LOW (0.1) or at or above LOCAL_DECISION_HIGH (0.9); every other question goes to the remote model, as does everything if the local model fails. LOCAL_DECISION_SCOPE is public (default) or all.
  • TRUSTED_PROXIES lists addresses or prefixes whose X-Forwarded-For is believed (its last entry), so the rate limit sees the real client behind a reverse proxy.

admin dashboard

set ADMIN_PASSWORD in the root .env and run ./bin/web-extract, then open http://127.0.0.1:8788/admin/. an empty admin password disables the dashboard. this is separate from the stdio mcp process; both share the same logs.

the shadcn ui provides request activity, model/error/token totals, jev cost per request and per day, jev and evidence inspection, and a search/contents/answer playground for your own queries. it uses password sessions, not the api key. jev cost counts every decision call's input tokens, whichever model answered, at JEV_PRICE_PER_MILLION (default 0.042 dollars per million; jev does not bill output); calls that report no usage, such as the local model's, are estimated at a quarter of their request size. see admin setup and security for configuration, limits, and browser tests.

api shape

http exposes POST /search, POST /contents, POST /answer, and POST /mcp. use x-api-key or bearer authentication. GET /healthz is public. mcp tool arguments use the same request objects as their http counterparts.

mcp tools/call accepts prime-style _meta in the call parameters or top-level tool arguments and drops it before decoding or logging the actual tool input. other unknown arguments remain invalid; the http routes do not accept _meta.

AUTH_STORE_PATH optionally points to the existing service's auth.json. issued user tokens retain their sha256 verification and registered caller identity; they do not grant dashboard access. the file is reread for each authentication so revocations take effect immediately.

text.maxCharacters and highlights.maxCharacters accept 0–10000. positive values cap returned characters; 0 or omission leaves text uncapped and uses the configured evidence budget for highlights. these limits apply to both contents and search contents options.

search with snippets selected from the original pages:

{"query":"sqlite wal concurrent readers and writers","numResults":2,"contents":{"highlights":true,"summary":true}}

read known pages with query-focused evidence:

{"urls":["https://sqlite.org/wal.html"],"highlights":{"query":"can there be multiple writers?","maxCharacters":4000},"summary":{"query":"can there be multiple writers?"}}

search results and contents return requestId and results, with id, url, title, optional metadata and requested text, highlights, highlightScores, and summary. contents also has per-url statuses; inspect these even when the overall call succeeds. answers return requestId, answer, and citations.

this is an intentional subset of exa search, contents, and answer, not full compatibility. only type: "auto" is supported; no streaming, json-schema summaries, deep search, or invite-management routes. highlightScores are jev evidence probabilities, not exa cosine scores. search with contents may also return per-url statuses. web_answer with text: true includes selected citation evidence rather than full pages.

single-page aliases

the old names route to the new contents handler; they do not restore the old response contract or retrieval pipeline.

mcp toolhttp routeinputcontents behavior
web_extractPOST /v1/extracturl, optional max_charstext, without model calls
web_previewPOST /v1/previewurl, required queryquery-focused summary
web_summarizePOST /v1/summarizeurl, optional query or instructionssummary; query takes precedence over instructions

preview/summarize also accept include_markdown: true to include results[].text, and max_chars to cap that returned text (0 means uncapped; otherwise 1–10000). this cap does not truncate the evidence used for summarization. queries/instructions are limited to 4000 bytes. browser/cookie/proxy settings and other old options are not supported; unknown fields are rejected rather than silently ignored. use web_contents for batch urls, highlights, and explicit content options.

all aliases return requestId, results, and per-url statuses; read results[].text or results[].summary, not top-level markdown/summary. inspect statuses even on success. authentication, jev selection, summaries, and training rows use the existing handlers unchanged. the dashboard records alias calls as contents.

batched judgments and deadlines

jev ranking and passage checks use parallel questions in bounded batches (at most 96 questions and 100,000 encoded bytes per call). each question carries only its candidate/passage in structured instructions; the query is shared state. all four passage checks remain, and no candidates are removed just to reduce call counts. a normal 20-result rerank takes one call; 24 passages with four checks each fit one call unless their encoded size requires splitting. JEV_CONCURRENCY controls in-flight batches independently of fetching and summarization (default 8, maximum 16). the existing pacing and retry limits still apply.

contents/search preserve completed results at the overall deadline, with explicit per-url timeout statuses for unfinished work. no successful results means the timeout still fails the request. request logs retain per-url statuses and mark partial failures as partial; completion/training schemas and prose tasks are unchanged. jev decision rows represent actual batch calls, not individual questions.

1# web extract
2
3an exa-shaped retrieval service: kagi search via browser rendering and logged in session from cookies, jev reranking and verbatim evidence selection, and LLM generated summaries.
4
5## layout
6
7```text
8cmd/web-extract/        executable entrypoint
9internal/app/          service, http/mcp transports, retrieval, models, and tests
10internal/app/testdata/  frozen source fixtures
11web/                   shadcn dashboard and embedded production assets
12scripts/build          rebuild frontend and go binary
13docs/admin.md          dashboard setup and security
14```
15
16## build and run
17
18requires go 1.25 and a c compiler for sqlite. pages are rendered with an existing chrome installation; no bundled browser download is required. PDF text is extracted in-process, with no external `pdftotext` dependency.
19
20browser rendering is stealthy by default. it presents a desktop chrome identity matching the host it runs on, and removes the automation fingerprints that bot walls read: `navigator.webdriver`, the empty plugin list, the software-rendered WebGL strings, and the `Headless_Chromium` user agent.
21
22the identity follows the host deliberately. claiming a different platform than the machine actually is — a linux user agent beside an Apple GPU and macOS fonts — describes hardware that cannot exist, and the stealth chromium builds take the same approach for the same reason. client-hints headers (`Sec-CH-UA-*`) are also left to the browser, because they are opt-in per origin and a browser that volunteers them unrequested is anomalous.
23
24set `STEALTH=false` in `.env` to render with the browser's true identity, for example when diagnosing what a page sees. `BROWSER_HEADLESS=false` launches a visible window, which needs a logged-in desktop session.
25
26a page is rendered in a real browser and converted from the resulting DOM. if the render produces less than 80 non-whitespace characters — a PDF served without a `.pdf` suffix, or a page that only serves text to non-browser clients — the same URL is retried once with a plain fetch. `.pdf` URLs skip the browser entirely, since a PDF has no DOM to render.
27
28set `PROXY_POOL` to a comma-separated list of upstream proxies to route egress through them in rotation, which is the lever for a blocked or rate-limited egress address. one proxy is selected per request so a page, its subresources, and any retry keep the same address.
29
30```sh
31go test ./...
32go vet ./...
33go build -trimpath -ldflags='-s -w' -o bin/web-extract ./cmd/web-extract
34./scripts/mcp          # codex normally owns this process
35./bin/web-extract      # alternatively serve http on 127.0.0.1:8788
36```
37
38copy `.env.example` to `.env` for a new installation, fill credentials, and make it owner-readable only (`chmod 600 .env`). do not overwrite an existing configured `.env`.
39
40## public access
41
42`PUBLIC_ACCESS=true` serves callers who send no token, for a shared instance (search.nkp.pet). a request with a token is unaffected; a wrong token is still refused.
43
44- `GET /` returns a notice explaining the service and its data policy.
45- each client address may make `PUBLIC_RATE_PER_MINUTE` requests a minute (default 20). `PUBLIC_DAILY_LIMIT` optionally caps public tool calls a day across all clients (default 0, no cap; reset at midnight UTC; MCP handshakes and listings do not count). refusals are 429 with `Retry-After`.
46- nothing about a public call is logged unless it opts in, with the header `X-Contribute: 1` or by calling `/mcp/contribute`. opted-in calls go to their own log, `PUBLIC_LOG_PATH` (default `data/public.db`, same schema).
47- `PUBLIC_JEV_URL` / `PUBLIC_JEV_KEY` / `PUBLIC_JEV_MODEL` give public calls their own decision model, any server speaking the SystemOne API. `cloudflare/clef` is a Worker that serves Cloudflare's Clef-Flash this way. `PUBLIC_JEV_FALLBACK_URL` / `PUBLIC_JEV_FALLBACK_KEY` / `PUBLIC_JEV_FALLBACK_MODEL` (default `jev-1.13-free`) name a second SystemOne model used when the first fails after its retries, for example OpenCode Zen (`https://opencode.ai/zen/v1`), so a Worker or allocation outage does not take public calls down.
48- `LOCAL_DECISION_URL` names a local SystemOne-compatible model asked before the remote one (`serve/decision.py` serves a fine-tuned EmbeddingGemma 2 typed-decision checkpoint). Its answers to the yes/no questions named in `LOCAL_DECISION_QUESTIONS` (default `evidence,useful`) are kept when they are at or below `LOCAL_DECISION_LOW` (0.1) or at or above `LOCAL_DECISION_HIGH` (0.9); every other question goes to the remote model, as does everything if the local model fails. `LOCAL_DECISION_SCOPE` is `public` (default) or `all`.
49- `TRUSTED_PROXIES` lists addresses or prefixes whose `X-Forwarded-For` is believed (its last entry), so the rate limit sees the real client behind a reverse proxy.
50
51## admin dashboard
52
53set `ADMIN_PASSWORD` in the root `.env` and run `./bin/web-extract`, then open `http://127.0.0.1:8788/admin/`. an empty admin password disables the dashboard. this is separate from the stdio mcp process; both share the same logs.
54
55the shadcn ui provides request activity, model/error/token totals, jev cost per request and per day, jev and evidence inspection, and a search/contents/answer playground for your own queries. it uses password sessions, not the api key. jev cost counts every decision call's input tokens, whichever model answered, at `JEV_PRICE_PER_MILLION` (default 0.042 dollars per million; jev does not bill output); calls that report no usage, such as the local model's, are estimated at a quarter of their request size. see [admin setup and security](docs/admin.md) for configuration, limits, and browser tests.
56
57## api shape
58
59http exposes `POST /search`, `POST /contents`, `POST /answer`, and `POST /mcp`. use `x-api-key` or bearer authentication. `GET /healthz` is public. mcp tool arguments use the same request objects as their http counterparts.
60
61mcp `tools/call` accepts prime-style `_meta` in the call parameters or top-level tool arguments and drops it before decoding or logging the actual tool input. other unknown arguments remain invalid; the http routes do not accept `_meta`.
62
63`AUTH_STORE_PATH` optionally points to the existing service's `auth.json`. issued user tokens retain their sha256 verification and registered caller identity; they do not grant dashboard access. the file is reread for each authentication so revocations take effect immediately.
64
65`text.maxCharacters` and `highlights.maxCharacters` accept 0–10000. positive values cap returned characters; 0 or omission leaves text uncapped and uses the configured evidence budget for highlights. these limits apply to both contents and search contents options.
66
67search with snippets selected from the original pages:
68
69```json
70{"query":"sqlite wal concurrent readers and writers","numResults":2,"contents":{"highlights":true,"summary":true}}
71```
72
73read known pages with query-focused evidence:
74
75```json
76{"urls":["https://sqlite.org/wal.html"],"highlights":{"query":"can there be multiple writers?","maxCharacters":4000},"summary":{"query":"can there be multiple writers?"}}
77```
78
79search results and contents return `requestId` and `results`, with `id`, `url`, `title`, optional metadata and requested `text`, `highlights`, `highlightScores`, and `summary`. contents also has per-url `statuses`; inspect these even when the overall call succeeds. answers return `requestId`, `answer`, and `citations`.
80
81this is an intentional subset of [exa search](https://exa.ai/docs/reference/search), [contents](https://exa.ai/docs/reference/get-contents), and [answer](https://exa.ai/docs/reference/answer), not full compatibility. only `type: "auto"` is supported; no streaming, json-schema summaries, deep search, or invite-management routes. `highlightScores` are jev evidence probabilities, not exa cosine scores. search with contents may also return per-url statuses. `web_answer` with `text: true` includes selected citation evidence rather than full pages.
82
83### single-page aliases
84
85the old names route to the new `contents` handler; they do not restore the old response contract or retrieval pipeline.
86
87| mcp tool | http route | input | contents behavior |
88| --- | --- | --- | --- |
89| `web_extract` | `POST /v1/extract` | `url`, optional `max_chars` | text, without model calls |
90| `web_preview` | `POST /v1/preview` | `url`, required `query` | query-focused summary |
91| `web_summarize` | `POST /v1/summarize` | `url`, optional `query` or `instructions` | summary; query takes precedence over instructions |
92
93preview/summarize also accept `include_markdown: true` to include `results[].text`, and `max_chars` to cap that returned text (0 means uncapped; otherwise 1–10000). this cap does not truncate the evidence used for summarization. queries/instructions are limited to 4000 bytes. browser/cookie/proxy settings and other old options are not supported; unknown fields are rejected rather than silently ignored. use `web_contents` for batch urls, highlights, and explicit content options.
94
95all aliases return `requestId`, `results`, and per-url `statuses`; read `results[].text` or `results[].summary`, not top-level `markdown`/`summary`. inspect statuses even on success. authentication, jev selection, summaries, and training rows use the existing handlers unchanged. the dashboard records alias calls as `contents`.
96
97### batched judgments and deadlines
98
99jev ranking and passage checks use parallel questions in bounded batches (at most 96 questions and 100,000 encoded bytes per call). each question carries only its candidate/passage in structured instructions; the query is shared state. all four passage checks remain, and no candidates are removed just to reduce call counts. a normal 20-result rerank takes one call; 24 passages with four checks each fit one call unless their encoded size requires splitting. `JEV_CONCURRENCY` controls in-flight batches independently of fetching and summarization (default 8, maximum 16). the existing pacing and retry limits still apply.
100
101contents/search preserve completed results at the overall deadline, with explicit per-url timeout statuses for unfinished work. no successful results means the timeout still fails the request. request logs retain per-url statuses and mark partial failures as `partial`; completion/training schemas and prose tasks are unchanged. jev decision rows represent actual batch calls, not individual questions.