Ryuu Search

Ryuu Search is a self-hosted web crawler and search engine backed by PostgreSQL full-text search. It includes a focused search UI, an authenticated indexer panel, ranked results with highlighted snippets, and crawl progress tracking.

What it does

• Crawls HTTP/HTTPS pages while ignoring robots.txt by default, with an explicit per-site option to enforce crawler rules
• Discovers XML sitemap indexes and Sitemap: entries from robots.txt
• Canonicalizes links, removes configured tracking parameters, and blocks common crawl traps
• Persists pending URLs with priority, retry time, leases, depth, referrer, and discovery source
• Runs the API and crawler as isolated processes with globally enforced PostgreSQL site leases
• Uses fair 500-page/10-minute crawl quanta, priority aging, adaptive per-domain concurrency, and circuit breakers
• Uses conditional requests and adaptive recrawl dates
• Groups exact and near-duplicate content with SHA-256 and SimHash fingerprints
• Optionally renders JavaScript-dependent pages in isolated Playwright Chromium contexts
• Extracts main content, structured metadata, headings, authors, dates, breadcrumbs, alternate languages, plain text, and PDFs
• Uses bounded two-stage ranking with text relevance, title phrases, link authority, freshness, quality, URL depth, and duplicate penalties
• Provides accent-aware and multilingual vectors, synonyms, autocomplete, and typo correction
• Detects content language, penalizes spam/thin/ad-heavy pages, diversifies domains, and clusters canonical results
• Supports optional pgvector semantic retrieval with Reciprocal Rank Fusion and automatic lexical fallback
• Supports cursor pagination, site/language/date filters, capped totals, and safe highlighted snippets
• Supports quoted phrases and PostgreSQL web-style query syntax
• Supports site:example.com in the web interface
• Limits crawl depth, page count, response size, concurrency, and request duration
• Classifies network/TLS/DNS/redirect/challenge failures and retries with full-jitter and server-directed backoff
• Supports crawl priority, pause, resume, cancellation, retry, and crash-safe continuation
• Stores replayable dead letters, cold document bodies, and change-version history
• Checks HTTP status, content type, noindex directives, and common soft-error pages; failed pages are automatically hidden from search
• Maintains per-site index settings for search visibility, robots policy, page limit, and crawl depth
• Revalidates one site or the entire index through persistent background health-check jobs
• Blocks loopback and private-network crawl targets by default
• Protects APIs with rate limits, rotating admin tokens, and mutation audit logging
• Tracks operational alerts, slow statements, partitioned telemetry, verified backups, and relevance benchmarks
• Caches permanent redirects, distinguishes true loops from long chains, and quarantines repeating redirect failures
• Scores crawl value, enforces per-domain bandwidth budgets, and exposes independent pipeline-stage backlogs
• Stores durable end-to-end traces from search/crawl work through database and ranking spans
• Enforces immediate domain blocks, approved takedowns, and sensitive-content exclusions in public results
• Provides explainable page-quality penalties and periodic domain reliability/duplicate-ratio refreshes

Run it

Requirements: Node.js 22+ and PostgreSQL 18 (older supported PostgreSQL releases should also work).

npm install
npm run dev

Open http://localhost:3000. The production commands are:

npm run build
npm start

Admin console

Open http://localhost:3000/admin and unlock it with ADMIN_TOKEN from .env. The token is kept in session storage by default; selecting Remember on this device stores it locally until Lock admin is used.

The console provides live index metrics, job priority and lifecycle controls, retry diagnostics, outbound-domain discoveries, first-time indexing, reindexing, ignore/restore controls, bulk actions, and removal of indexed pages. Site indexes adds independent search visibility, robots policy, URL rules, ignored parameters, trap limits, recrawl scheduling, JavaScript rendering, health checks, and excluded-page reports. Telemetry reports queue delay, throughput, latency, outcomes, storage, popular queries, clicks, and content gaps. Operations exposes alerts, leases, rich PostgreSQL diagnostics, dead-letter replay, slow SQL, verified backups, and audit history. Intelligence & safety exposes redirect chains and rule suggestions, canonical mappings, quarantines, pipeline backlogs, traces, SafeSearch policy, domain blocks, takedowns, sensitive findings, and controlled-change history. Search settings manages query assistance, synonyms, relevance judgments, baseline-versus-candidate benchmark gates, and relevance recomputation.

The current machine-specific connection settings are in .env, which is ignored by Git. Copy .env.example when deploying elsewhere. PostgreSQL uses a TCP connection string such as postgresql://user:password@host:5433/database; it is not an HTTP URL.

Index a site

Open Indexer in the top-right of the UI and enter the ADMIN_TOKEN value from .env, a seed URL, page limit, and depth. New jobs default to 100,000 pages and support up to 1,000,000 pages. Up to four sites run concurrently; each site has adaptive request pacing and its pending URLs remain durable in PostgreSQL. Large sites yield after a configurable page/time quantum so older and smaller queued jobs continue moving.

The crawler ignores robots.txt by default so it can discover and index all linked subpages. Enable Respect robots.txt on a new job or site index when crawler rules should be enforced. The setting applies to subsequent crawls and reindexes.

Login, logout, account, cart, payment, OAuth/session, credential-bearing, and token-bearing URLs are always rejected even when robots.txt is ignored. This non-overridable rule prevents accidental indexing of stateful or private endpoints.

Newly crawled pages are health-classified as they are fetched. To validate content already in the database, open Site indexes and choose Check pages for one site or Check all pages for the complete index. HTTP errors, non-HTML responses, meta-robots noindex, and recognized soft-error pages remain in the diagnostic record but are excluded from public search results. A later successful check makes a recovered page searchable again.

JavaScript rendering is configured per site as Off, Fallback only, or Always. On a new host, install the included Playwright Chromium runtime with:

npx playwright install chromium

To crawl a trusted LAN-only site, set ALLOW_PRIVATE_NETWORKS=true. Do not enable this on an internet-exposed instance because the crawler can then reach internal services.

The API equivalent is:

$headers = @{ "x-admin-token" = "your-admin-token" }
$body = @{ url = "https://example.com"; maxPages = 100; maxDepth = 3; sameOrigin = $true; respectRobots = $false } | ConvertTo-Json
Invoke-RestMethod -Method Post -Uri http://localhost:3000/api/crawl -Headers $headers -ContentType application/json -Body $body

API

• GET /api/search?q=terms&limit=10&site=example.com&language=en&safeSearch=moderate&from=2026-01-01&cursor=...
• GET /api/autocomplete?q=prefix
• GET /api/suggest?q=misspelling
• POST /api/search/click
• GET /api/stats
• GET /api/health
• GET /api/jobs (admin token required)
• GET /api/jobs/:id/failures (admin token required; latest 100 detailed errors)
• POST /api/crawl (admin token required)
• GET /api/admin/site-indexes (admin token required)
• PATCH /api/admin/site-indexes/:domain (admin token required)
• POST /api/admin/health-checks (admin token required; pass an optional domain)
• GET /api/admin/health-checks (admin token required)
• GET /api/admin/domains/:domain/errors (admin token required)
• GET /api/admin/telemetry (admin token required)
• PATCH /api/admin/jobs/:id and POST /api/admin/jobs/:id/action (admin token required)
• GET|PUT /api/admin/search-settings (admin token required)
• POST /api/admin/relevance/recompute (admin token required)
• GET /api/admin/operations/health and POST /api/admin/operations/backup (admin token required)
• GET /api/admin/dead-letters and POST /api/admin/dead-letters/replay (admin token required)
• GET|PUT /api/admin/relevance/judgments and POST /api/admin/relevance/evaluate (admin token required)
• POST /api/admin/relevance/compare (admin token required; compares selected or latest two runs)
• GET /api/admin/redirects, DELETE /api/admin/redirects/mappings, and quarantine release (admin token required)
• GET /api/admin/pipeline and GET /api/admin/traces[/:traceId] (admin token required)
• GET /api/admin/safety plus policy, domain-block, takedown, finding, and bandwidth mutations (admin token required)
• GET /api/admin/audit-log (admin token required)

Database objects

The service creates document, body/version, crawl frontier, domain throttle, link graph, site policy, relevance, audit, alert, health-check, search-settings, and partitioned telemetry tables plus GIN, B-tree, and trigram indexes on startup. It enables PostgreSQL's pg_trgm, unaccent, and pg_stat_statements extensions when permitted. Expired URL leases and interrupted work return safely to the queue after restart.

Optional semantic search uses an OpenAI-compatible embeddings endpoint configured with EMBEDDING_BASE_URL, EMBEDDING_MODEL, and EMBEDDING_DIMENSIONS. Without these values or pgvector, search automatically stays lexical.

Search accepts quoted phrases, site:example.com, intitle:term, filetype:pdf, and -excluded operators. Operator values are parsed into bound SQL parameters rather than interpolated into queries.

PIPELINE_MODE=inline keeps the current all-in-one indexing path. The durable discovery/fetch/render/extract/embed/index queues and budgets are initialized for a staged deployment; switch to staged only when independent stage workers are deployed. Durable PostgreSQL traces are always available and retained for TRACE_RETENTION_DAYS. OTEL_SERVICE_NAME and OTEL_EXPORTER_OTLP_ENDPOINT reserve configuration for an optional OTLP exporter adapter; the database trace recorder does not depend on an external collector.

Create and restore-verify a PostgreSQL custom-format backup with:

powershell -File scripts/backup-postgres.ps1 -VerifyRestore

Security notes

• Rotate the PostgreSQL password shown in the original screenshot; screenshots and chat transcripts are not secret stores.
• Replace the admin token when deploying or sharing the service.
• Put the app behind TLS and authentication before exposing it outside the LAN.
• Keep ALLOW_PRIVATE_NETWORKS=false unless LAN crawling is specifically required.