Our Independent CRM Data Vault
How SPIRITT moved millions of records, relationships, and history entries from our customer platform into an independent PostgreSQL data layer with complete coverage evidence and hands-off refreshes.
The challenge
At SPIRITT, we depended on a cloud customer platform as the operational home for our customer records, companies, deals, activity, forms, lists, conversations, file metadata, ownership, and custom fields.
Over time, that system accumulated millions of records, relationship edges, history entries, and supporting resources. It remained useful for day-to-day operations, but it was difficult to reuse. Our analytics and AI products had to work through product-specific APIs. Schemas changed over time. Relationships lived behind separate endpoints. Some objects were available only when an account feature was enabled. Archived data behaved differently from active data. A successful request did not prove the entire source had been copied.
We needed our own independent, organization-controlled database that could answer a harder question than “did the script run?”:
Can we account for every available source surface, and distinguish unavailable data from failed imports?
What we built
Our SPIRITT Agent created the data system end to end.
A live schema and relationship map
Our importer discovered standard, optional, internal, and custom object types at runtime. It brought across property definitions, value history, archive state, owners, pipelines, directional association labels, and adjacent resources rather than assuming a fixed CRM model.
A lossless but usable PostgreSQL model
We normalized frequently queried entities and relationships into relational tables and readable views. Raw source payloads remained beside them in JSON, giving our teams a stable query layer without discarding source detail.
Coverage as a first-class dataset
We gave every source surface a ledger entry with its completion state and source-versus-imported reconciliation. Account-disabled features and official API limitations stayed visible as explicit exclusions. Nothing silently passed.
A secure operational product
We added private PostgreSQL storage, organization-only authentication, a searchable read-only grid, typed columns, recurring jobs, lifecycle reconciliation, and persistent verified backups.
The interface
The data vault is intentionally simple: choose a logical table or source object, search, filter archive state, resize or reorder columns, and inspect typed values. Dates, booleans, numbers, links, images, phone numbers, and multi-value fields render in their natural form.
The screenshot below is from our shipped system. Workspace identity, source vendor, record values, raw counts, IDs, and timestamps have been redacted.
The hard parts the agent resolved
The work was not a straightforward API export.
The source returned different behavior across object families, treated archived records inconsistently, exposed feature-gated endpoints, and repeated a terminal pagination cursor after an empty page. An overlapping scan also demonstrated why lifecycle reconciliation must be isolated: two correct imports can produce false disappearance signals when they compete to mark the latest observation.
We reproduced each issue, changed the import contract, and reran an isolated reconciliation. Empty terminal pages became an explicit stopping condition. Concurrent full scans were prohibited. Missing records were reconciled only after an object scan completed. Account-disabled features were reclassified as unavailable rather than failed.
The result
Our final completion gate reported no unresolved failures, no pending work, no source-versus-import mismatches, and no false lifecycle tombstones.
More importantly, we no longer pay the integration tax for every new data product. Our dashboards, internal tools, retrieval systems, agents, and analysis workflows can start from one governed data layer instead of rebuilding the source connection.
The recurring refresh also removed the operator time we previously spent exporting, joining, validating, and redistributing source data. The system now runs whether anyone is watching or not.