Store crawled content for search
Connect
SPIRITT
WebCrawlerAPIWebCrawlerAPI
Whole sites, turned into data
SPIRITT crawls entire sites into clean content, extracts the fields you define from any page, and runs change feeds that turn website updates into actions in your apps.
What you can do with WebCrawlerAPI.
Full sites, clean content
SPIRITT crawls with depth, URL filters and main-content mode, then uses every page it collects.
Fields from any page
A prompt or JSON schema turns a single page into structured data on the spot.
Changes caught on schedule
Recurring feeds track sites, and SPIRITT acts on each detected change.
Works with your other apps.
Get started in three steps.
01
Open your workspace
Start a SPIRITT workspace for the websites your work depends on.
02
Connect WebCrawlerAPI
Connect WebCrawlerAPI so SPIRITT can crawl, scrape and run change feeds.
03
Delegate an outcome
Name the sites and what matters. SPIRITT crawls, watches and acts.
Hand over a mission.
Choose a mission. Make it yours.
Put WebCrawlerAPI to work.
Know every site you depend on
Questions
WebCrawlerAPI, with SPIRITT.
01What can SPIRITT do with WebCrawlerAPI?+
Crawl whole sites, scrape single pages with structured extraction, run recurring change feeds and track your spend.
02How are crawls controlled?+
With page limits, depth, URL allow and block patterns, robots.txt handling and main-content mode.
03Can it return structured data?+
Yes. SPIRITT passes a prompt or JSON schema and gets the fields back from the page.
04How does change monitoring work?+
Feeds recrawl a site on a schedule. SPIRITT reads each detected change and acts on it in your apps.
05Can feeds be paused or stopped?+
Yes. SPIRITT pauses, resumes or cancels feeds and checks their run history and cost.
06How do I get started with WebCrawlerAPI?+
Open a SPIRITT workspace, connect WebCrawlerAPI and name a site to crawl or watch. SPIRITT starts the first run.