# pub-search

> pub-search indexes long-form writing across AT Protocol publishing platforms
> (leaflet, pckt, offprint, greengale, whitewind, and self-hosted
> standard.site sites) as one curated corpus. it exposes that corpus over an
> HTTP API, an MCP server, and an AT Protocol labeler. bulk/scraper sources are
> banned at ingest, so results are human writing. keyword (FTS5/BM25),
> semantic (voyage-4-lite + turbopuffer), and hybrid (RRF) search modes.

## Project contracts

- [Site](https://pub-search.waow.tech) — the human surface (search + atlas)
- [Adopting pub-search for agents](https://github.com/zzstoatzz/pub-search/blob/main/docs/agent-surfaces.md) — start here; which surface to pick and why
- [HTTP API reference](https://github.com/zzstoatzz/pub-search/blob/main/docs/api.md)
- [Search syntax](https://github.com/zzstoatzz/pub-search/blob/main/docs/search-syntax.md) — quotes, OR, prefix behavior
- [Content extraction](https://github.com/zzstoatzz/pub-search/blob/main/docs/content-extraction.md) — why full text is per-platform work
- [Source repository](https://github.com/zzstoatzz/pub-search)

## Surfaces you can consume

### MCP server — default for tool-calling agents

- hosted: `claude mcp add-json pub-search '{"type": "http", "url": "https://pub-search-by-zzstoatzz.fastmcp.app/mcp"}'`
- local stdio: `claude mcp add pub-search -- uvx --from 'git+https://github.com/zzstoatzz/pub-search#subdirectory=pub-search-mcp/server' pub-search`
- a curated layer, not a 1:1 of the HTTP API. tools: `search` (hybrid by
  default, no offset), `get_document` (fetches + flattens the live record from
  the author's PDS), `author_profile`, `find_similar`, `describe_cluster`,
  `discover_focal_post`, `recommended_by_top_authors`. prompts:
  `usage_guide`, `search_tips`. resources: `pub-search://stats`,
  `pub-search://tags`, `pub-search://popular` — reference data is read on
  demand rather than charged against every reasoning cycle.
- publications that set `preferences.showInDiscover=false` are indexed but kept
  out of every discovery surface. `include_undiscoverable=true` opts back in
  and is scoped to ONE identity: it requires `author=` on `/search`, requires a
  single repo on `/document`, and is rejected on `/similar`. Requests without
  that scope get a 400 rather than being silently downgraded. This is scoping,
  not authentication — it closes enumeration, but a caller who already knows a
  DID can still read that author's unlisted writing.

### HTTP API — for code-mode agents, scripts, pipelines

- base URL: `https://api.pub-search.waow.tech` (no auth, GET only)
- `curl -s 'https://api.pub-search.waow.tech/search?q=relay&mode=hybrid&format=v2&limit=5' | jq`
- search returns snippets; `/document?uri=<at-uri>[,...]` returns the FULL
  extracted article text (up to 25 uris per request) — use it to read what
  search found instead of re-fetching records from each author's PDS.
- endpoints: `/search`, `/document`, `/similar`, `/tags`, `/popular`,
  `/stats`, `/activity`, `/api/dashboard`, `/subscribed`, `/subscribers`,
  `/wrapped`, `/api/labeler`, `/health`. full parameters + response shapes
  in api.md.
- publications that set `preferences.showInDiscover=false` are indexed but
  excluded from every discovery surface (search, /similar, /document). their
  author asked to stay out of them. `include_undiscoverable=true` opts back in
  and REQUIRES `author=` (400 without it); on `/document` every uri must share
  one repo; `/similar` rejects it outright. scoping, not authentication.

### AT Protocol labeler — a subscribable trust signal

- `labeler.pub-search.waow.tech` serves standard XRPC
  `com.atproto.label.queryLabels` (HTTP) and `com.atproto.label.subscribeLabels`
  (WebSocket). an autonomous classifier emits the `bulk-generated` label on
  accounts that generate documents from a dataset rather than composing them;
  those accounts are excluded from search. subscribe if you want the same
  signal in your own pipeline. see [/labels](https://pub-search.waow.tech/labels).

### the corpus itself lives on AT Protocol

- every result is an `at://` record on its author's PDS. search returns enough
  metadata (`uri`, `did`, `rkey`, `basePath`, `path`, `platform`) to build a
  web URL or fetch the record directly (`com.atproto.repo.getRecord`, or
  `uvx pdsx`). for reading, prefer the API's `/document` (indexed text,
  batched) or the MCP `get_document` (live record) over rolling your own
  extraction.

## Agent guidance

- read [agent-surfaces.md](https://github.com/zzstoatzz/pub-search/blob/main/docs/agent-surfaces.md)
  first — it has a decision table (MCP vs HTTP) and the sharp edges.
- MCP if the consumer is a tool-calling agent and you want it productive with
  no code of yours. HTTP if your agent writes code anyway or needs pagination /
  raw response shapes the tools don't expose.
- use `api.pub-search.waow.tech` for API calls. the site domain
  `pub-search.waow.tech` only proxies the API under `/api/*` (edge-cached,
  rate-limited 30 req/10s per IP → 429); every other unknown path there returns
  the SPA's HTML with a 200, so trusting status codes will "succeed" against a
  page of HTML. inspect bodies.
- at least one of `q`, `tag`, or `author` is required; an empty query is
  rejected by design.
- filters vary by mode: `tag`/`since` apply to keyword only; `platform`/`author`
  work everywhere; hybrid applies each filter to the half that supports it.
- `author` accepts a handle or a DID (handles resolve server-side).
  `search("", author="someone.com")` browses an author's everything.
- freshness is bounded, not real-time — the keyword replica refreshes by
  snapshot swap (minutes-to-hours of lag). `pub-search://stats` / `/api/dashboard` show
  current counts if staleness matters.

## Source docs

- `README.md` documents the overall architecture (ingester → backend → builder
  → labeler → site → mcp).
- `docs/agent-surfaces.md` is the canonical guide for consuming pub-search.
- `docs/api.md` is the full HTTP endpoint reference, including URL-building
  patterns per platform.
- `docs/search-syntax.md` documents query syntax; `docs/search-architecture.md`
  documents how the three modes dispatch.
- `pub-search-mcp/server/` is the installable MCP server.