okami.mom/web-extract-go
git clone https://git.nkp.pet/okami.mom/web-extract-go
8af1784
main
web extract
an exa-shaped retrieval service: kagi search via browser rendering and logged in session from cookies, jev reranking and verbatim evidence selection, and LLM generated summaries.
layout
cmd/web-extract/ executable entrypoint internal/app/ service, http/mcp transports, retrieval, models, and tests internal/app/testdata/ frozen source fixtures web/ shadcn dashboard and embedded production assets scripts/build rebuild frontend and go binary docs/admin.md dashboard setup and security
build and run
requires go 1.25 and a c compiler for sqlite. pages are rendered with an existing chrome installation; no bundled browser download is required. PDF text is extracted in-process, with no external pdftotext dependency.
browser rendering is stealthy by default. it presents a desktop chrome identity matching the host it runs on, and removes the automation fingerprints that bot walls read: navigator.webdriver, the empty plugin list, the software-rendered WebGL strings, and the Headless_Chromium user agent.
the identity follows the host deliberately. claiming a different platform than the machine actually is — a linux user agent beside an Apple GPU and macOS fonts — describes hardware that cannot exist, and the stealth chromium builds take the same approach for the same reason. client-hints headers (Sec-CH-UA-*) are also left to the browser, because they are opt-in per origin and a browser that volunteers them unrequested is anomalous.
set STEALTH=false in .env to render with the browser's true identity, for example when diagnosing what a page sees. BROWSER_HEADLESS=false launches a visible window, which needs a logged-in desktop session.
a page is rendered in a real browser and converted from the resulting DOM. if the render produces less than 80 non-whitespace characters — a PDF served without a .pdf suffix, or a page that only serves text to non-browser clients — the same URL is retried once with a plain fetch. .pdf URLs skip the browser entirely, since a PDF has no DOM to render.
set PROXY_POOL to a comma-separated list of upstream proxies to route egress through them in rotation, which is the lever for a blocked or rate-limited egress address. one proxy is selected per request so a page, its subresources, and any retry keep the same address.
go test ./...go vet ./...go build-trimpath -ldflags= '-s -w' -o bin/web-extract ./cmd/web-extract./scripts/mcp # codex normally owns this process ./bin/web-extract # alternatively serve http on 127.0.0.1:8788
copy .env.example to .env for a new installation, fill credentials, and make it owner-readable only (chmod 600 .env). do not overwrite an existing configured .env.
public access
PUBLIC_ACCESS=true serves callers who send no token, for a shared instance (search.nkp.pet). a request with a token is unaffected; a wrong token is still refused.
GET /returns a notice explaining the service and its data policy.- each client address may make
PUBLIC_RATE_PER_MINUTErequests a minute (default 20).PUBLIC_DAILY_LIMIToptionally caps public tool calls a day across all clients (default 0, no cap; reset at midnight UTC; MCP handshakes and listings do not count). refusals are 429 withRetry-After. - nothing about a public call is logged unless it opts in, with the header
X-Contribute: 1or by calling/mcp/contribute. opted-in calls go to their own log,PUBLIC_LOG_PATH(defaultdata/public.db, same schema). PUBLIC_JEV_URL/PUBLIC_JEV_KEY/PUBLIC_JEV_MODELgive public calls their own decision model, any server speaking the SystemOne API.cloudflare/clefis a Worker that serves Cloudflare's Clef-Flash this way.PUBLIC_JEV_FALLBACK_URL/PUBLIC_JEV_FALLBACK_KEY/PUBLIC_JEV_FALLBACK_MODEL(defaultjev-1.13-free) name a second SystemOne model used when the first fails after its retries, for example OpenCode Zen (https://opencode.ai/zen/v1), so a Worker or allocation outage does not take public calls down.LOCAL_DECISION_URLnames a local SystemOne-compatible model asked before the remote one (serve/decision.pyserves a fine-tuned EmbeddingGemma 2 typed-decision checkpoint). Its answers to the yes/no questions named inLOCAL_DECISION_QUESTIONS(defaultevidence,useful) are kept when they are at or belowLOCAL_DECISION_LOW(0.1) or at or aboveLOCAL_DECISION_HIGH(0.9); every other question goes to the remote model, as does everything if the local model fails.LOCAL_DECISION_SCOPEispublic(default) orall.TRUSTED_PROXIESlists addresses or prefixes whoseX-Forwarded-Foris believed (its last entry), so the rate limit sees the real client behind a reverse proxy.
admin dashboard
set ADMIN_PASSWORD in the root .env and run ./bin/web-extract, then open http://127.0.0.1:8788/admin/. an empty admin password disables the dashboard. this is separate from the stdio mcp process; both share the same logs.
the shadcn ui provides request activity, model/error/token totals, jev cost per request and per day, jev and evidence inspection, and a search/contents/answer playground for your own queries. it uses password sessions, not the api key. jev cost counts every decision call's input tokens, whichever model answered, at JEV_PRICE_PER_MILLION (default 0.042 dollars per million; jev does not bill output); calls that report no usage, such as the local model's, are estimated at a quarter of their request size. see admin setup and security for configuration, limits, and browser tests.
api shape
http exposes POST /search, POST /contents, POST /answer, and POST /mcp. use x-api-key or bearer authentication. GET /healthz is public. mcp tool arguments use the same request objects as their http counterparts.
mcp tools/call accepts prime-style _meta in the call parameters or top-level tool arguments and drops it before decoding or logging the actual tool input. other unknown arguments remain invalid; the http routes do not accept _meta.
AUTH_STORE_PATH optionally points to the existing service's auth.json. issued user tokens retain their sha256 verification and registered caller identity; they do not grant dashboard access. the file is reread for each authentication so revocations take effect immediately.
text.maxCharacters and highlights.maxCharacters accept 0–10000. positive values cap returned characters; 0 or omission leaves text uncapped and uses the configured evidence budget for highlights. these limits apply to both contents and search contents options.
search with snippets selected from the original pages:
{"query" :"sqlite wal concurrent readers and writers" ,"numResults" :2 ,"contents" :{"highlights" :true ,"summary" :true }}
read known pages with query-focused evidence:
{"urls" :["https://sqlite.org/wal.html" ],"highlights" :{"query" :"can there be multiple writers?" ,"maxCharacters" :4000 },"summary" :{"query" :"can there be multiple writers?" }}
search results and contents return requestId and results, with id, url, title, optional metadata and requested text, highlights, highlightScores, and summary. contents also has per-url statuses; inspect these even when the overall call succeeds. answers return requestId, answer, and citations.
this is an intentional subset of exa search, contents, and answer, not full compatibility. only type: "auto" is supported; no streaming, json-schema summaries, deep search, or invite-management routes. highlightScores are jev evidence probabilities, not exa cosine scores. search with contents may also return per-url statuses. web_answer with text: true includes selected citation evidence rather than full pages.
single-page aliases
the old names route to the new contents handler; they do not restore the old response contract or retrieval pipeline.
| mcp tool | http route | input | contents behavior |
|---|---|---|---|
web_extract | POST /v1/extract | url, optional max_chars | text, without model calls |
web_preview | POST /v1/preview | url, required query | query-focused summary |
web_summarize | POST /v1/summarize | url, optional query or instructions | summary; query takes precedence over instructions |
preview/summarize also accept include_markdown: true to include results[].text, and max_chars to cap that returned text (0 means uncapped; otherwise 1–10000). this cap does not truncate the evidence used for summarization. queries/instructions are limited to 4000 bytes. browser/cookie/proxy settings and other old options are not supported; unknown fields are rejected rather than silently ignored. use web_contents for batch urls, highlights, and explicit content options.
all aliases return requestId, results, and per-url statuses; read results[].text or results[].summary, not top-level markdown/summary. inspect statuses even on success. authentication, jev selection, summaries, and training rows use the existing handlers unchanged. the dashboard records alias calls as contents.
batched judgments and deadlines
jev ranking and passage checks use parallel questions in bounded batches (at most 96 questions and 100,000 encoded bytes per call). each question carries only its candidate/passage in structured instructions; the query is shared state. all four passage checks remain, and no candidates are removed just to reduce call counts. a normal 20-result rerank takes one call; 24 passages with four checks each fit one call unless their encoded size requires splitting. JEV_CONCURRENCY controls in-flight batches independently of fetching and summarization (default 8, maximum 16). the existing pacing and retry limits still apply.
contents/search preserve completed results at the overall deadline, with explicit per-url timeout statuses for unfinished work. no successful results means the timeout still fails the request. request logs retain per-url statuses and mark partial failures as partial; completion/training schemas and prose tasks are unchanged. jev decision rows represent actual batch calls, not individual questions.
1# web extract 2 3an exa-shaped retrieval service: kagi search via browser rendering and logged in session from cookies, jev reranking and verbatim evidence selection, and LLM generated summaries. 4 5## layout 6 7``` text 8cmd/web-extract/ executable entrypoint 9internal/app/ service, http/mcp transports, retrieval, models, and tests 10internal/app/testdata/ frozen source fixtures 11web/ shadcn dashboard and embedded production assets 12scripts/build rebuild frontend and go binary 13docs/admin.md dashboard setup and security 14``` 15 16## build and run 17 18requires go 1.25 and a c compiler for sqlite. pages are rendered with an existing chrome installation; no bundled browser download is required. PDF text is extracted in-process, with no external `pdftotext` dependency. 19 20browser rendering is stealthy by default. it presents a desktop chrome identity matching the host it runs on, and removes the automation fingerprints that bot walls read: `navigator.webdriver`, the empty plugin list, the software-rendered WebGL strings, and the `Headless_Chromium` user agent. 21 22the identity follows the host deliberately. claiming a different platform than the machine actually is — a linux user agent beside an Apple GPU and macOS fonts — describes hardware that cannot exist, and the stealth chromium builds take the same approach for the same reason. client-hints headers (`Sec-CH-UA-*`) are also left to the browser, because they are opt-in per origin and a browser that volunteers them unrequested is anomalous. 23 24set `STEALTH=false` in `.env` to render with the browser's true identity, for example when diagnosing what a page sees. `BROWSER_HEADLESS=false` launches a visible window, which needs a logged-in desktop session. 25 26a page is rendered in a real browser and converted from the resulting DOM. if the render produces less than 80 non-whitespace characters — a PDF served without a `.pdf` suffix, or a page that only serves text to non-browser clients — the same URL is retried once with a plain fetch. `.pdf` URLs skip the browser entirely, since a PDF has no DOM to render. 27 28set `PROXY_POOL` to a comma-separated list of upstream proxies to route egress through them in rotation, which is the lever for a blocked or rate-limited egress address. one proxy is selected per request so a page, its subresources, and any retry keep the same address. 29 30``` sh 31go test ./... 32go vet ./... 33go build -trimpath -ldflags='-s -w' -o bin/web-extract ./cmd/web-extract 34./scripts/mcp # codex normally owns this process 35./bin/web-extract # alternatively serve http on 127.0.0.1:8788 36``` 37 38copy `.env.example` to `.env` for a new installation, fill credentials, and make it owner-readable only (`chmod 600 .env`). do not overwrite an existing configured `.env`. 39 40## public access 41 42`PUBLIC_ACCESS=true` serves callers who send no token, for a shared instance (search.nkp.pet). a request with a token is unaffected; a wrong token is still refused. 43 44- `GET /` returns a notice explaining the service and its data policy. 45- each client address may make `PUBLIC_RATE_PER_MINUTE` requests a minute (default 20). `PUBLIC_DAILY_LIMIT` optionally caps public tool calls a day across all clients (default 0, no cap; reset at midnight UTC; MCP handshakes and listings do not count). refusals are 429 with `Retry-After`. 46- nothing about a public call is logged unless it opts in, with the header `X-Contribute: 1` or by calling `/mcp/contribute`. opted-in calls go to their own log, `PUBLIC_LOG_PATH` (default `data/public.db`, same schema). 47- `PUBLIC_JEV_URL` / `PUBLIC_JEV_KEY` / `PUBLIC_JEV_MODEL` give public calls their own decision model, any server speaking the SystemOne API. `cloudflare/clef` is a Worker that serves Cloudflare's Clef-Flash this way. `PUBLIC_JEV_FALLBACK_URL` / `PUBLIC_JEV_FALLBACK_KEY` / `PUBLIC_JEV_FALLBACK_MODEL` (default `jev-1.13-free`) name a second SystemOne model used when the first fails after its retries, for example OpenCode Zen (`https://opencode.ai/zen/v1`), so a Worker or allocation outage does not take public calls down. 48- `LOCAL_DECISION_URL` names a local SystemOne-compatible model asked before the remote one (`serve/decision.py` serves a fine-tuned EmbeddingGemma 2 typed-decision checkpoint). Its answers to the yes/no questions named in `LOCAL_DECISION_QUESTIONS` (default `evidence,useful`) are kept when they are at or below `LOCAL_DECISION_LOW` (0.1) or at or above `LOCAL_DECISION_HIGH` (0.9); every other question goes to the remote model, as does everything if the local model fails. `LOCAL_DECISION_SCOPE` is `public` (default) or `all`. 49- `TRUSTED_PROXIES` lists addresses or prefixes whose `X-Forwarded-For` is believed (its last entry), so the rate limit sees the real client behind a reverse proxy. 50 51## admin dashboard 52 53set `ADMIN_PASSWORD` in the root `.env` and run `./bin/web-extract`, then open `http://127.0.0.1:8788/admin/`. an empty admin password disables the dashboard. this is separate from the stdio mcp process; both share the same logs. 54 55the shadcn ui provides request activity, model/error/token totals, jev cost per request and per day, jev and evidence inspection, and a search/contents/answer playground for your own queries. it uses password sessions, not the api key. jev cost counts every decision call's input tokens, whichever model answered, at `JEV_PRICE_PER_MILLION` (default 0.042 dollars per million; jev does not bill output); calls that report no usage, such as the local model's, are estimated at a quarter of their request size. see [admin setup and security](docs/admin.md) for configuration, limits, and browser tests. 56 57## api shape 58 59http exposes `POST /search`, `POST /contents`, `POST /answer`, and `POST /mcp`. use `x-api-key` or bearer authentication. `GET /healthz` is public. mcp tool arguments use the same request objects as their http counterparts. 60 61mcp `tools/call` accepts prime-style `_meta` in the call parameters or top-level tool arguments and drops it before decoding or logging the actual tool input. other unknown arguments remain invalid; the http routes do not accept `_meta`. 62 63`AUTH_STORE_PATH` optionally points to the existing service's `auth.json`. issued user tokens retain their sha256 verification and registered caller identity; they do not grant dashboard access. the file is reread for each authentication so revocations take effect immediately. 64 65`text.maxCharacters` and `highlights.maxCharacters` accept 0–10000. positive values cap returned characters; 0 or omission leaves text uncapped and uses the configured evidence budget for highlights. these limits apply to both contents and search contents options. 66 67search with snippets selected from the original pages: 68 69``` json 70{ "query" : "sqlite wal concurrent readers and writers" , "numResults" : 2 , "contents" :{ "highlights" : true , "summary" : true }} 71``` 72 73read known pages with query-focused evidence: 74 75``` json 76{ "urls" :[ "https://sqlite.org/wal.html" ], "highlights" :{ "query" : "can there be multiple writers?" , "maxCharacters" : 4000 }, "summary" :{ "query" : "can there be multiple writers?" }} 77``` 78 79search results and contents return `requestId` and `results`, with `id`, `url`, `title`, optional metadata and requested `text`, `highlights`, `highlightScores`, and `summary`. contents also has per-url `statuses`; inspect these even when the overall call succeeds. answers return `requestId`, `answer`, and `citations`. 80 81this is an intentional subset of [exa search](https://exa.ai/docs/reference/search), [contents](https://exa.ai/docs/reference/get-contents), and [answer](https://exa.ai/docs/reference/answer), not full compatibility. only `type: "auto"` is supported; no streaming, json-schema summaries, deep search, or invite-management routes. `highlightScores` are jev evidence probabilities, not exa cosine scores. search with contents may also return per-url statuses. `web_answer` with `text: true` includes selected citation evidence rather than full pages. 82 83### single-page aliases 84 85the old names route to the new `contents` handler; they do not restore the old response contract or retrieval pipeline. 86 87| mcp tool | http route | input | contents behavior | 88| --- | --- | --- | --- | 89| `web_extract` | `POST /v1/extract` | `url`, optional `max_chars` | text, without model calls | 90| `web_preview` | `POST /v1/preview` | `url`, required `query` | query-focused summary | 91| `web_summarize` | `POST /v1/summarize` | `url`, optional `query` or `instructions` | summary; query takes precedence over instructions | 92 93preview/summarize also accept `include_markdown: true` to include `results[].text`, and `max_chars` to cap that returned text (0 means uncapped; otherwise 1–10000). this cap does not truncate the evidence used for summarization. queries/instructions are limited to 4000 bytes. browser/cookie/proxy settings and other old options are not supported; unknown fields are rejected rather than silently ignored. use `web_contents` for batch urls, highlights, and explicit content options. 94 95all aliases return `requestId`, `results`, and per-url `statuses`; read `results[].text` or `results[].summary`, not top-level `markdown`/`summary`. inspect statuses even on success. authentication, jev selection, summaries, and training rows use the existing handlers unchanged. the dashboard records alias calls as `contents`. 96 97### batched judgments and deadlines 98 99jev ranking and passage checks use parallel questions in bounded batches (at most 96 questions and 100,000 encoded bytes per call). each question carries only its candidate/passage in structured instructions; the query is shared state. all four passage checks remain, and no candidates are removed just to reduce call counts. a normal 20-result rerank takes one call; 24 passages with four checks each fit one call unless their encoded size requires splitting. `JEV_CONCURRENCY` controls in-flight batches independently of fetching and summarization (default 8, maximum 16). the existing pacing and retry limits still apply. 100 101contents/search preserve completed results at the overall deadline, with explicit per-url timeout statuses for unfinished work. no successful results means the timeout still fails the request. request logs retain per-url statuses and mark partial failures as `partial`; completion/training schemas and prose tasks are unchanged. jev decision rows represent actual batch calls, not individual questions.