← back

okami.mom/web-extract-go

git clone https://git.nkp.pet/okami.mom/web-extract-go

albedosummaries: LLM_TEMPERATURE sets the summarizer temperature (default 1.0)20f80af

main
8 folders7 files
languages

files

.impeccable/
cloudflare/clef/
cmd/web-extract/
docs/
internal/app/
scripts/
serve/
web/
.env.example1.3 KiB
.gitignore128 B
DESIGN.md7.2 KiB
PRODUCT.md1.0 KiB
README.md10.1 KiB
go.mod874 B
go.sum3.8 KiB

web extract

an exa-shaped retrieval service: kagi search via browser rendering and logged in session from cookies, jev reranking and verbatim evidence selection, and LLM generated summaries.

layout

cmd/web-extract/        executable entrypoint
internal/app/          service, http/mcp transports, retrieval, models, and tests
internal/app/testdata/  frozen source fixtures
web/                   shadcn dashboard and embedded production assets
scripts/build          rebuild frontend and go binary
docs/admin.md          dashboard setup and security

build and run

requires go 1.25 and a c compiler for sqlite. pages are rendered with an existing chrome installation; no bundled browser download is required. PDF text is extracted in-process, with no external pdftotext dependency.

browser rendering is stealthy by default. it presents a desktop chrome identity matching the host it runs on, and removes the automation fingerprints that bot walls read: navigator.webdriver, the empty plugin list, the software-rendered WebGL strings, and the Headless_Chromium user agent.

the identity follows the host deliberately. claiming a different platform than the machine actually is — a linux user agent beside an Apple GPU and macOS fonts — describes hardware that cannot exist, and the stealth chromium builds take the same approach for the same reason. client-hints headers (Sec-CH-UA-*) are also left to the browser, because they are opt-in per origin and a browser that volunteers them unrequested is anomalous.

set STEALTH=false in .env to render with the browser's true identity, for example when diagnosing what a page sees. BROWSER_HEADLESS=false launches a visible window, which needs a logged-in desktop session.

a page is rendered in a real browser and converted from the resulting DOM. if the render produces less than 80 non-whitespace characters — a PDF served without a .pdf suffix, or a page that only serves text to non-browser clients — the same URL is retried once with a plain fetch. .pdf URLs skip the browser entirely, since a PDF has no DOM to render.

set PROXY_POOL to a comma-separated list of upstream proxies to route egress through them in rotation, which is the lever for a blocked or rate-limited egress address. one proxy is selected per request so a page, its subresources, and any retry keep the same address.

go test ./...
go vet ./...
go build -trimpath -ldflags='-s -w' -o bin/web-extract ./cmd/web-extract
./scripts/mcp          # codex normally owns this process
./bin/web-extract      # alternatively serve http on 127.0.0.1:8788

copy .env.example to .env for a new installation, fill credentials, and make it owner-readable only (chmod 600 .env). do not overwrite an existing configured .env.

public access

PUBLIC_ACCESS=true serves callers who send no token, for a shared instance (search.nkp.pet). a request with a token is unaffected; a wrong token is still refused.

admin dashboard

set ADMIN_PASSWORD in the root .env and run ./bin/web-extract, then open http://127.0.0.1:8788/admin/. an empty admin password disables the dashboard. this is separate from the stdio mcp process; both share the same logs.

the shadcn ui provides request activity, model/error/token totals, jev cost per request and per day, jev and evidence inspection, and a search/contents/answer playground for your own queries. it uses password sessions, not the api key. jev cost counts every decision call's input tokens, whichever model answered, at JEV_PRICE_PER_MILLION (default 0.042 dollars per million; jev does not bill output); calls that report no usage, such as the local model's, are estimated at a quarter of their request size. see admin setup and security for configuration, limits, and browser tests.

api shape

http exposes POST /search, POST /contents, POST /answer, and POST /mcp. use x-api-key or bearer authentication. GET /healthz is public. mcp tool arguments use the same request objects as their http counterparts.

mcp tools/call accepts prime-style _meta in the call parameters or top-level tool arguments and drops it before decoding or logging the actual tool input. other unknown arguments remain invalid; the http routes do not accept _meta.

AUTH_STORE_PATH optionally points to the existing service's auth.json. issued user tokens retain their sha256 verification and registered caller identity; they do not grant dashboard access. the file is reread for each authentication so revocations take effect immediately.

text.maxCharacters and highlights.maxCharacters accept 0–10000. positive values cap returned characters; 0 or omission leaves text uncapped and uses the configured evidence budget for highlights. these limits apply to both contents and search contents options.

search with snippets selected from the original pages:

{"query":"sqlite wal concurrent readers and writers","numResults":2,"contents":{"highlights":true,"summary":true}}

read known pages with query-focused evidence:

{"urls":["https://sqlite.org/wal.html"],"highlights":{"query":"can there be multiple writers?","maxCharacters":4000},"summary":{"query":"can there be multiple writers?"}}

search results and contents return requestId and results, with id, url, title, optional metadata and requested text, highlights, highlightScores, and summary. contents also has per-url statuses; inspect these even when the overall call succeeds. answers return requestId, answer, and citations.

this is an intentional subset of exa search, contents, and answer, not full compatibility. only type: "auto" is supported; no streaming, json-schema summaries, deep search, or invite-management routes. highlightScores are jev evidence probabilities, not exa cosine scores. search with contents may also return per-url statuses. web_answer with text: true includes selected citation evidence rather than full pages.

single-page aliases

the old names route to the new contents handler; they do not restore the old response contract or retrieval pipeline.

mcp toolhttp routeinputcontents behavior
web_extractPOST /v1/extracturl, optional max_charstext, without model calls
web_previewPOST /v1/previewurl, required queryquery-focused summary
web_summarizePOST /v1/summarizeurl, optional query or instructionssummary; query takes precedence over instructions

preview/summarize also accept include_markdown: true to include results[].text, and max_chars to cap that returned text (0 means uncapped; otherwise 1–10000). this cap does not truncate the evidence used for summarization. queries/instructions are limited to 4000 bytes. browser/cookie/proxy settings and other old options are not supported; unknown fields are rejected rather than silently ignored. use web_contents for batch urls, highlights, and explicit content options.

all aliases return requestId, results, and per-url statuses; read results[].text or results[].summary, not top-level markdown/summary. inspect statuses even on success. authentication, jev selection, summaries, and training rows use the existing handlers unchanged. the dashboard records alias calls as contents.

batched judgments and deadlines

jev ranking and passage checks use parallel questions in bounded batches (at most 96 questions and 100,000 encoded bytes per call). each question carries only its candidate/passage in structured instructions; the query is shared state. all four passage checks remain, and no candidates are removed just to reduce call counts. a normal 20-result rerank takes one call; 24 passages with four checks each fit one call unless their encoded size requires splitting. JEV_CONCURRENCY controls in-flight batches independently of fetching and summarization (default 8, maximum 16). the existing pacing and retry limits still apply.

contents/search preserve completed results at the overall deadline, with explicit per-url timeout statuses for unfinished work. no successful results means the timeout still fails the request. request logs retain per-url statuses and mark partial failures as partial; completion/training schemas and prose tasks are unchanged. jev decision rows represent actual batch calls, not individual questions.