Index crawled docs for RAG
Connect
SPIRITT
ScrapflyScrapfly
Any site, crawled and structured
SPIRITT uses Scrapfly to get through the hardest pages. It crawls whole sites within a credit budget, renders JavaScript, scrapes as any country, extracts structured data and keeps full archives, then loads it all where you work.
What you can do with Scrapfly.
Whole sites, on a budget
Recursive crawls with depth, path filters, page limits and a credit ceiling.
Pages that fight back
JavaScript rendering, anti-scraping bypass, country proxies and POST or PUT requests.
Structured data and proof
AI extraction to JSON, screenshots and WARC or HAR archives.
Works with your other apps.
Get started in three steps.
01
Open your workspace
Start a SPIRITT workspace for the web data you need.
02
Connect Scrapfly
Add your Scrapfly key so SPIRITT can scrape and crawl in your account.
03
Name the sites and output
Share the sites, fields and destination. SPIRITT crawls, extracts and loads the data.
Hand over a mission.
Choose a mission. Make it yours.
Put Scrapfly to work.
Get data from any site
Questions
Scrapfly, with SPIRITT.
01What can SPIRITT do with Scrapfly?+
Scrape pages with GET, POST or PUT, render JavaScript, use proxies and anti-scraping bypass, crawl sites, read crawl contents and URLs, download WARC or HAR archives, extract structured data, capture screenshots and check account usage.
02Can it get through anti-bot protection?+
Yes. Scrapes and crawls can enable Scrapfly's anti-scraping protection bypass with rendering and proxy rotation.
03How are crawl costs controlled?+
Crawls take page limits, depth limits and a credit budget that stops them automatically.
04Can extraction results arrive by webhook?+
Yes. Extraction can run asynchronously and send results to a named Scrapfly webhook.
05What output formats are available?+
Markdown, text, HTML and JSON contents, extracted data, page metadata, screenshots and WARC or HAR archives.
06How do I get started with Scrapfly?+
Open a SPIRITT workspace, connect Scrapfly and name a site and the data you need. SPIRITT crawls it and delivers the result.