TorEdge

TorEdge is a Rust end-to-end Tor search and research platform. The first milestone includes a bounded crawler, policy engine, HTML extractor, Redis frontier, PostgreSQL schema, Meilisearch adapter, Axum web UI, admin route surface, and toredge CLI.

TorEdge is designed for legitimate search, security research, OSINT, private research, and archival workflows. It intentionally avoids features for marketplace promotion, credential collection, authentication bypass, exploit automation, or evading site owner controls.

Local Services

Start dependencies:

docker compose up -d postgres redis meilisearch tor

Run migrations:

cargo run -p toredge-cli -- migrate

Start the web app:

cargo run -p toredge-cli -- serve

Local-Only Mode

Run without Redis, PostgreSQL, Meilisearch, or Docker:

cargo run -p toredge-cli -- local

Then open http://127.0.0.1:8080/admin, log in with the default local password toredge-local, add a .onion seed, and press Start Crawl. The local UI stores queued URLs and crawled pages in SQLite so you can stop and restart without losing the crawl frontier.

The local admin dashboard also shows active crawl limits, recent fetch attempts, recent crawl errors, recent indexed pages, and locally persisted blocklist entries. Use the page removal buttons to delete a page from SQLite and the embedded search index. Use --admin-password <password> to change the local admin password.

Local mode uses:

• SQLite at data/toredge.sqlite
• SQLite-backed persistent URL frontier
• Embedded in-process search

Index a local HTML fixture and start the UI:

cargo run -p toredge-cli -- local --fixture-html .\sample.html

Run a one-shot Tor crawl without Redis/PostgreSQL/Meilisearch:

cargo run -p toredge-cli -- local --crawl-once --seed http://exampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexampleexample.onion/

If you are using Tor Browser instead of a standalone Tor service, point local mode at Tor Browser's SOCKS port:

cargo run -p toredge-cli -- local --tor-socks-proxy socks5h://127.0.0.1:9150

Run one crawler pass:

cargo run -p toredge-cli -- crawl --once

The web app listens on http://localhost:8080 by default. Runtime defaults live in .env.example; copy it to .env before making local secrets or production overrides.

Verification

Run the full local verification suite:

cargo fmt --all --check
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings

Conservative Defaults

The default crawl policy is intentionally bounded:

• Maximum depth: 2
• Maximum pages per host: 100
• Maximum response size: 2 MB
• Global concurrent fetches: 4
• Per-host cooldown: 60 seconds
• Fetch timeout: 45 seconds
• Retries: 2

Only HTTP and HTTPS .onion URLs are accepted. Non-HTML responses, oversized responses, blocked URLs, blocked hosts, and denied content patterns are rejected before indexing.

Current Milestone Notes

The codebase is split into focused Rust crates:

• toredge-common: shared config and records.
• toredge-policy: onion URL validation, blocklists, MIME checks, and response limits.
• toredge-extract: title, text, snippet, and onion-link extraction.
• toredge-frontier: in-memory and Redis URL frontier implementations.
• toredge-index: Meilisearch document mapping and HTTP adapter.
• toredge-db: PostgreSQL migration and repository helpers.
• toredge-crawler: one-pass crawler orchestration with injectable fetcher and sink.
• toredge-app: Axum web UI and admin route surface.
• toredge-cli: serve, crawl, and migrate commands.