okami.mom/web-extract-go
git clone https://git.nkp.pet/okami.mom/web-extract-go
20f80af
main
files
| .impeccable/ | |
| cloudflare/clef/ | |
| cmd/web-extract/ | |
| docs/ | |
| internal/app/ | |
| scripts/ | |
| serve/ | |
| web/ | |
| .env.example | 1.3 KiB |
| .gitignore | 128 B |
| DESIGN.md | 7.2 KiB |
| PRODUCT.md | 1.0 KiB |
| README.md | 10.1 KiB |
| go.mod | 874 B |
| go.sum | 3.8 KiB |
web extract
an exa-shaped retrieval service: kagi search via browser rendering and logged in session from cookies, jev reranking and verbatim evidence selection, and LLM generated summaries.
layout
cmd/web-extract/ executable entrypoint internal/app/ service, http/mcp transports, retrieval, models, and tests internal/app/testdata/ frozen source fixtures web/ shadcn dashboard and embedded production assets scripts/build rebuild frontend and go binary docs/admin.md dashboard setup and security
build and run
requires go 1.25 and a c compiler for sqlite. pages are rendered with an existing chrome installation; no bundled browser download is required. PDF text is extracted in-process, with no external pdftotext dependency.
browser rendering is stealthy by default. it presents a desktop chrome identity matching the host it runs on, and removes the automation fingerprints that bot walls read: navigator.webdriver, the empty plugin list, the software-rendered WebGL strings, and the Headless_Chromium user agent.
the identity follows the host deliberately. claiming a different platform than the machine actually is — a linux user agent beside an Apple GPU and macOS fonts — describes hardware that cannot exist, and the stealth chromium builds take the same approach for the same reason. client-hints headers (Sec-CH-UA-*) are also left to the browser, because they are opt-in per origin and a browser that volunteers them unrequested is anomalous.
set STEALTH=false in .env to render with the browser's true identity, for example when diagnosing what a page sees. BROWSER_HEADLESS=false launches a visible window, which needs a logged-in desktop session.
a page is rendered in a real browser and converted from the resulting DOM. if the render produces less than 80 non-whitespace characters — a PDF served without a .pdf suffix, or a page that only serves text to non-browser clients — the same URL is retried once with a plain fetch. .pdf URLs skip the browser entirely, since a PDF has no DOM to render.
set PROXY_POOL to a comma-separated list of upstream proxies to route egress through them in rotation, which is the lever for a blocked or rate-limited egress address. one proxy is selected per request so a page, its subresources, and any retry keep the same address.
go test ./...go vet ./...go build-trimpath -ldflags= '-s -w' -o bin/web-extract ./cmd/web-extract./scripts/mcp # codex normally owns this process ./bin/web-extract # alternatively serve http on 127.0.0.1:8788
copy .env.example to .env for a new installation, fill credentials, and make it owner-readable only (chmod 600 .env). do not overwrite an existing configured .env.
public access
PUBLIC_ACCESS=true serves callers who send no token, for a shared instance (search.nkp.pet). a request with a token is unaffected; a wrong token is still refused.
GET /returns a notice explaining the service and its data policy.- each client address may make
PUBLIC_RATE_PER_MINUTErequests a minute (default 20).PUBLIC_DAILY_LIMIToptionally caps public tool calls a day across all clients (default 0, no cap; reset at midnight UTC; MCP handshakes and listings do not count). refusals are 429 withRetry-After. - nothing about a public call is logged unless it opts in, with the header
X-Contribute: 1or by calling/mcp/contribute. opted-in calls go to their own log,PUBLIC_LOG_PATH(defaultdata/public.db, same schema). PUBLIC_JEV_URL/PUBLIC_JEV_KEY/PUBLIC_JEV_MODELgive public calls their own decision model, any server speaking the SystemOne API.cloudflare/clefis a Worker that serves Cloudflare's Clef-Flash this way.PUBLIC_JEV_FALLBACK_URL/PUBLIC_JEV_FALLBACK_KEY/PUBLIC_JEV_FALLBACK_MODEL(defaultjev-1.13-free) name a second SystemOne model used when the first fails after its retries, for example OpenCode Zen (https://opencode.ai/zen/v1), so a Worker or allocation outage does not take public calls down.LOCAL_DECISION_URLnames a local SystemOne-compatible model asked before the remote one (serve/decision.pyserves a fine-tuned EmbeddingGemma 2 typed-decision checkpoint). Its answers to the yes/no questions named inLOCAL_DECISION_QUESTIONS(defaultevidence,useful) are kept when they are at or belowLOCAL_DECISION_LOW(0.1) or at or aboveLOCAL_DECISION_HIGH(0.9); every other question goes to the remote model, as does everything if the local model fails.LOCAL_DECISION_SCOPEispublic(default) orall.TRUSTED_PROXIESlists addresses or prefixes whoseX-Forwarded-Foris believed (its last entry), so the rate limit sees the real client behind a reverse proxy.
admin dashboard
set ADMIN_PASSWORD in the root .env and run ./bin/web-extract, then open http://127.0.0.1:8788/admin/. an empty admin password disables the dashboard. this is separate from the stdio mcp process; both share the same logs.
the shadcn ui provides request activity, model/error/token totals, jev cost per request and per day, jev and evidence inspection, and a search/contents/answer playground for your own queries. it uses password sessions, not the api key. jev cost counts every decision call's input tokens, whichever model answered, at JEV_PRICE_PER_MILLION (default 0.042 dollars per million; jev does not bill output); calls that report no usage, such as the local model's, are estimated at a quarter of their request size. see admin setup and security for configuration, limits, and browser tests.
api shape
http exposes POST /search, POST /contents, POST /answer, and POST /mcp. use x-api-key or bearer authentication. GET /healthz is public. mcp tool arguments use the same request objects as their http counterparts.
mcp tools/call accepts prime-style _meta in the call parameters or top-level tool arguments and drops it before decoding or logging the actual tool input. other unknown arguments remain invalid; the http routes do not accept _meta.
AUTH_STORE_PATH optionally points to the existing service's auth.json. issued user tokens retain their sha256 verification and registered caller identity; they do not grant dashboard access. the file is reread for each authentication so revocations take effect immediately.
text.maxCharacters and highlights.maxCharacters accept 0–10000. positive values cap returned characters; 0 or omission leaves text uncapped and uses the configured evidence budget for highlights. these limits apply to both contents and search contents options.
search with snippets selected from the original pages:
{"query" :"sqlite wal concurrent readers and writers" ,"numResults" :2 ,"contents" :{"highlights" :true ,"summary" :true }}
read known pages with query-focused evidence:
{"urls" :["https://sqlite.org/wal.html" ],"highlights" :{"query" :"can there be multiple writers?" ,"maxCharacters" :4000 },"summary" :{"query" :"can there be multiple writers?" }}
search results and contents return requestId and results, with id, url, title, optional metadata and requested text, highlights, highlightScores, and summary. contents also has per-url statuses; inspect these even when the overall call succeeds. answers return requestId, answer, and citations.
this is an intentional subset of exa search, contents, and answer, not full compatibility. only type: "auto" is supported; no streaming, json-schema summaries, deep search, or invite-management routes. highlightScores are jev evidence probabilities, not exa cosine scores. search with contents may also return per-url statuses. web_answer with text: true includes selected citation evidence rather than full pages.
single-page aliases
the old names route to the new contents handler; they do not restore the old response contract or retrieval pipeline.
| mcp tool | http route | input | contents behavior |
|---|---|---|---|
web_extract | POST /v1/extract | url, optional max_chars | text, without model calls |
web_preview | POST /v1/preview | url, required query | query-focused summary |
web_summarize | POST /v1/summarize | url, optional query or instructions | summary; query takes precedence over instructions |
preview/summarize also accept include_markdown: true to include results[].text, and max_chars to cap that returned text (0 means uncapped; otherwise 1–10000). this cap does not truncate the evidence used for summarization. queries/instructions are limited to 4000 bytes. browser/cookie/proxy settings and other old options are not supported; unknown fields are rejected rather than silently ignored. use web_contents for batch urls, highlights, and explicit content options.
all aliases return requestId, results, and per-url statuses; read results[].text or results[].summary, not top-level markdown/summary. inspect statuses even on success. authentication, jev selection, summaries, and training rows use the existing handlers unchanged. the dashboard records alias calls as contents.
batched judgments and deadlines
jev ranking and passage checks use parallel questions in bounded batches (at most 96 questions and 100,000 encoded bytes per call). each question carries only its candidate/passage in structured instructions; the query is shared state. all four passage checks remain, and no candidates are removed just to reduce call counts. a normal 20-result rerank takes one call; 24 passages with four checks each fit one call unless their encoded size requires splitting. JEV_CONCURRENCY controls in-flight batches independently of fetching and summarization (default 8, maximum 16). the existing pacing and retry limits still apply.
contents/search preserve completed results at the overall deadline, with explicit per-url timeout statuses for unfinished work. no successful results means the timeout still fails the request. request logs retain per-url statuses and mark partial failures as partial; completion/training schemas and prose tasks are unchanged. jev decision rows represent actual batch calls, not individual questions.