pagesift
High-performance, memory-efficient Rust web crawler & content extraction engine for AI agents and RAG. Adaptive crawling, clean Markdown, structured data, PDF/RSS parsing, browser rendering, Firecrawl-compatible APIs, and Python/Node.js SDKs.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
- html-to-markdown
- markdown
- mcp
- pdf-extraction
- rag
- rust
- web-crawler
- web-scraping
- Stars
- 104
- Forks
- 17
- + today
- +1
- Created
- 7d
Ranking data as of October 27, 2026 (UTC).
Star History
Today, hour by hour
01
Overview
High-performance, memory-efficient Rust web crawler & content extraction engine for AI agents and RAG. Adaptive crawling, clean Markdown, structured data, PDF/RSS parsing, browser rendering, Firecrawl-compatible APIs, and Python/Node.js SDKs. It ranks #3727 on GitTiger, gaining +1 star on October 27, 2026 (UTC).
The project is written in Rust and has 17 forks. It was created 7d ago.
Installation
git clone https://github.com/YOLOW92/pagesift.git
cd pagesift
# see README for setup