back
ENGINEERING
Yashraj Singh
AI Engineer, Morphic
8 min read
Recently our CEO, Joshua, shared a tweet on Slack from @DBredvick on X laying out a six-step ingestion pipeline architecture. It sent me down a rabbit hole, and I ended up writing the whole story of how we actually built ours at @morphic.io, the architecture we chose, which sources are live in production today for ingestion, and the iterations we made over time.
Introduction - The problem nobody wants to say out loud
Every CRM on earth has the same dirty secret: it is empty because a human has to fill it.
Not empty of rows. Empty of truth. The deal says "Proposal" because someone updated it three weeks ago and then went on holiday. The contact record has a name and an email and nothing else. The note that actually mattered - "they're worried the migration will take six months" - is sitting in someone's inbox, in a reply, on a Tuesday, unread by the system that is supposed to be the company's memory.
Meanwhile, the work already happened. It happened in Gmail. It happened in a Slack channel. It happened on a call that got transcribed and then never read again. The information exists. It's just scattered across twelve tools, none of which talk to each other, and the only integration between them is a person typing.
So we asked the obvious question: what if the CRM filled itself? And then we ran into the reason nobody had done it.
Chapter 1 - The economics of ingesting everything
Take a normal ten-person customer. Here's roughly what they generate in a single day:
The naive design is: point an AI at all of it. Let it read everything and keep the CRM up to date.
Do the arithmetic. 2,850 events a day, each needing real context to understand, run through a frontier model:
≈ $470 per day. ≈ $14,100 per month. Per customer.
We sell the product for $1,000–$2,500 a month. So the naive version loses about twelve thousand dollars per customer, per month, and loses it faster the more successful the customer is.
And onboarding is worse. A new customer connects two years of Gmail - around 200,000 messages. Reading all of that with AI to build their history:
≈ $33,000. For one customer. On day one. Before they've paid you anything.
These numbers come from our own production workload, not a universal law - a Slack-heavy customer or an email-heavy one will move the multiplier around. But the shape of the problem doesn't change: naive ingestion scales cost with volume, and volume scales with how well the product is doing.
This is the point where most teams quietly reduce the ambition. They ship "AI summarise this email" as a button and call it a platform.
We didn't want to do that. So the whole pipeline - every design decision in this post - exists to kill those two numbers without killing the ambition.
Where it landed
$14,100 → $58. Roughly 240× cheaper. Backfill: $33,000 → about $85.
This isn't a trick or a corner cut. It's a well-known pattern - Stanford's FrugalGPT research showed "cheap first, escalate rarely" cascades hitting up to 98% cost reduction at the same accuracy. We just applied it ruthlessly.
Chapter 2 - The principle
We spent a long time arguing about a false question: should we ingest everything, or only the important stuff?
Both answers are wrong, and the argument only ended when we split the question in two:
Be generous with facts. Be strict with commitments.
Facts are things that simply happened. A person exists. A company exists. An email was sent on Tuesday at 4pm. A meeting occurred. These are cheap, they're safe, and more of them makes the system smarter. Collect all of them.
Commitments are things that make the system act. Creating a deal. Moving a stage. Booking something. Drafting an outbound message. These are expensive, and - far more importantly - one wrong commitment breaks trust permanently. If the CRM invents a deal called "DocuSign" because someone signed a contract through DocuSign, the user stops believing anything it tells them. Gate these hard.
That single sentence resolved months of "junk versus completeness" debate. We're generous at the bottom of the funnel and strict at the top, and the two never compete again. Everything from here on is that sentence, implemented.
Chapter 3 - Architecture overview
Chapter 4 - Raw ingestion (Floor 1: the Mailroom)
Every single event from every connector gets stored, exactly as it arrived, forever. We wrap it in a standard label, take a fingerprint of the contents so we never store the same thing twice, hand a copy to the Librarian for indexing, and stop.
That label - the envelope every event gets wrapped in, regardless of source - looks roughly like this:
Zero AI. Zero decisions. Zero judgement.
This floor sounds boring and it is the most important one in the building, for one reason: every floor above it is calculated from it. If we ever get a rule wrong - a filter too aggressive, a category mislabelled, a whole design decision reversed - we haven't lost anything. We re-run it.
That property is what made it safe to be opinionated everywhere else. We can be wrong, and recover, without telling a customer their data is gone.
Chapter 5 - Deterministic processing (Floor 2: the Sorter)
Two jobs, both done by dumb machines. Deliberately, proudly, no AI.
Job A: throw out the obvious junk. Not by keywords - keyword filters are famously unreliable and produce exactly the "why did my important email get eaten" support ticket. By structure, which doesn't lie:
This removes roughly 85% of all volume at zero cost. It is the single cheapest win in the entire system, and it's the fix for the junk records that used to embarrass us in demos.
Crucially: these filters control promotion, never storage. A filtered newsletter is still in the Mailroom, still searchable. We're deciding what gets acted on, not what gets kept.
Job B: work out who people are. When sarah@morphic.io shows up, is she someone we already know? This is a genuinely hard problem, and we solved it with a ladder, cheapest rung first:
Narrow the field. Only compare her against people in the same bucket - same email domain, similar-sounding name. Without this, every new person gets compared against every existing person, and the system dies as the customer grows.
Exact match. Same email, phone, or ID? That's her. About 99% accurate, and free.
Fuzzy match. "Jon Smyth" and "John Smith" are probably the same human. Standard string-similarity scoring, still free.
Only then, ask the AI - for the genuinely ambiguous 2–5%. And we cache the answer forever, so the same confusing pair is never judged twice.
Chapter 6 - The knowledge layer (Floor 3: the Librarian)
For every important thing - a customer, a deal, a person - we keep one living summary page. Not the raw history. A compacted "here's what's going on."
Why a summary instead of reading the raw history every time? Because answering "how's Morphic doing?" by reading 500 messages is slow and expensive, and answering it from one short page is instant and nearly free. Write it once, read it a hundred times.
The page gets rewritten only when something real happens. There are exactly five triggers:
Chapter 7 - The AI boundary (Floor 4: the Detective)
This is the expensive, clever part, and it almost never runs.
When it does run, it doesn't read raw history - it reads the Librarian's summary pages and the map of who-knows-who. If that's not enough, it searches. If that's not enough, it goes and fetches live from the source. Three rungs, cheapest first, same pattern as everywhere else:
Every flashy feature people ask for - pre-meeting briefs, post-meeting intelligence, "which deals have gone quiet", competitor battle cards - turns out to be a question asked of the summary pages. No new pipeline required for any of them. That was the moment we knew the shape was right.
Chapter 8 - Connector routing at scale
Here's a trap we very nearly walked into.
We don't just need Gmail. We need Slack, Calendar, Telegram, phone calls, meeting recorders, Notion, GitHub, Linear, HubSpot, Salesforce, marketing tools, and a long tail of hundreds more.
Trap one: write a custom pipeline per connector. You would never finish. You'd be doing integration work forever and every new source would be a quarter of engineering time.
Trap two: force everything into one rigid shared format. Then every time a source changes its data, everything downstream breaks at once.
The answer we landed on: normalise the label, keep the parcel.
Think of posting a package. Every parcel gets the same address label on the outside. What's inside the box stays exactly as it was. The label is identical for every connector, forever - which customer, which source, their ID for it, a content fingerprint, when it happened, which conversation it belongs to, who was involved, whether it was inbound or outbound, and who's allowed to see it. Then the original contents, untouched.
And each connector gets one configuration row - not code - describing how to treat it:
Adding a new connector is adding a config row, not writing a new pipeline.
Look at what that table kills instantly: two thousand internal Slack messages a day never reach the expensive floor (but stay fully searchable). CI noise never invents a deal. A marketing tool never fabricates five thousand fake contacts.
Chapter 9 - Schema-independent ingestion
This is the one that broke every earlier draft, and it's worth explaining because it's the difference between a sales tool and a platform.
Our CRM doesn't have fixed tables. Customers define their own objects. So a real customer might have Investors, Mentors, Staff, Launch Partners, Vendors, Offshore Partners, and CEO Friends - and no "Companies" or "Deals" anywhere. A hotel has Guests, Properties and Bookings. A school has Students, Schools and Enrolments.
So: an email arrives. Where does it go? A pipeline that hardcodes "Company / Person / Deal" is useless to all of these customers.
The insight
The pipeline must never know what a "Company" is.
But here's the thing we noticed: across every business model we've run through this pipeline so far, whatever they call things, everything reduces to five shapes:
Same pipeline. Same code. Wildly different businesses.
The trick that makes it cheap
AI is used to understand the shape of the business - once. AI is never used to route each individual message.
When someone creates an object called "Launch Partners," we make one AI call: which of the five shapes is this? Answer: it's a group of humans. That answer is saved permanently on the object.
After that, every email that arrives forever is a pure lookup. No AI. No thinking. No cost.
Ten AI calls in a workspace's entire lifetime, instead of five hundred a day. And an admin can override the guess - we don't blindly trust a one-shot classification.
What if several objects share a shape?
Real case: Investors, Mentors, Staff and CEO Friends are all "a single human." An email arrives from sarah@morphic.io. Where does she go?
Same ladder, cheapest rung first:
Does she already exist somewhere? Put her there. Free.
Is there a routing rule? ("Domains on our VC list go to Investors.") Route her. Free.
Is there a default for this shape? Put her there, flagged as unclassified. Free.
None of the above? She waits in a holding area. The AI classifies her later - and only if she ever turns out to matter.
She is never lost, never blocked, and never costs a token to file. If she does become important, she gets moved once. And a human can always correct it.
What if the workspace is completely empty on day one?
This is the safety property that makes everything else possible:
The Mailroom and the Librarian have no schema. Only the CRM does. So the pipeline never blocks and never loses data while the schema is still undefined.
Data flows in from minute one and is immediately searchable. Records with nowhere to go wait in a holding area. Then either the customer picks an industry template, or - after a few days - we make one AI call that reads the aggregate patterns and proposes:
"You seem to talk to hotels and take reservations. Shall I create Guests, Properties and Bookings?"
A human says yes. The objects get created. Everything that was waiting flows in automatically.
That's a CRM that builds itself, and it costs one AI call - not one per message.
And stages are never hardcoded
An early version of the pipeline always created deals at the stage "prospecting." That's a bug for a hotel, where a booking starts at "inquiry," and for a school, where an enrolment starts at "applied."
The fix: read the live stages out of whatever the customer actually configured, and pick from those. Same code, zero hardcoding:
And if we map a shape wrong?
Rebuild it. The summaries are derived from the Mailroom, so changing the mapping and re-running moves the records to the right place. Nothing is lost.
This is also the answer to "what if the customer changes their business model next year?" - it re-analyses everything that already happened, for free, because we never threw anything away.
Chapter 10 - The promotion gate
There's one decision in this whole system that deserves its own section, because it's where trust is won or lost: when do we create a record?
The old answer was per-message and it was terrible. A single email that looked sales-ish would create a deal. That's how you end up with a deal named after a document-signing service, and a user who never trusts the product again.
The new answer is per-conversation, and deterministic. After each new message in a thread, a timer is set. When the conversation goes quiet, we ask a handful of plain, boring questions:
Did both sides actually speak? (Two-way, not a blast)
Is anyone here from outside the company?
Have there been enough exchanges to be real?
Does it look like it's about doing business?
If yes - and only if yes - we create one record, once, for the whole thread. Not one per message. Not three deals for three replies.
Roughly 97% of conversations stop at this gate, having cost nothing, while remaining fully searchable forever. About 3% get real intelligence spent on them.
That ratio is the entire business model.
One more subtlety worth stating: we deliberately did not replace forty cheap checks with one clever AI call per thread. It would have been cheaper than the old design and still the wrong architecture - it spends money on automatically creating records, which is exactly the thing that has to stay conservative. The boring deterministic gate already does that job, and it does it predictably.
Chapter 11 - Human authority and provenance
It moves Morphic to "Negotiation." That's wrong.
The Slack message is the correction surface. Joshua hits [Undo]. The old value comes back, and that field is permanently stamped as human-owned. From then on, the AI never touches that field again.
Every automatically - written field carries three things: who wrote it, how confident it was, and which specific events it was based on. Human edits outrank machine edits, permanently, with an audit trail.
We think this is non-negotiable. An automated system that can't be corrected isn't a product, it's a liability.
The same rule applies one level up, to schema itself: the system may build and manage the schema only while no human has touched it. The moment a person creates or edits an object type, schema ownership hands over to the human, permanently.
Chapter 12 - Twelve stories
This is what it actually looks like in a customer's day.
Story 1 - A new prospect emails you (the one that used to go wrong)
Sarah at Morphic emails Josh about pricing. Josh replies. Sarah replies.
Mailroom: all three messages stored and fingerprinted.
Sorter: no unsubscribe header, not a bot → passes.
becomes a Company. Sarah becomes a Person. An activity is logged for every message, immediately. All deterministic. No AI spent yet, and no deal yet.
Gate: after the thread settles - two-way, external, three exchanges, business-shaped - it fires once.
Record: one deal is created. Its stage is read from the customer's own live configuration. Sarah's budget comment, her migration worry, and Josh's promise to send a pricing doc are mapped onto the deal's actual fields.
Slack: "Created Morphic - Pricing (Discovery). Sarah is champion. Flagged migration concern. [Undo] [Actually it's…]"
Total deals created: 1. Total AI calls for the whole thread: 1.
Story 2 - Internal Slack chatter
Twenty messages in
about the deploy pipeline. Stored, indexed, searchable forever. The source rule says internal Slack never creates records, so the gate short-circuits immediately. Zero AI. Zero records. Zero cost. But ask, three months later, "what did we decide about that deploy change?" - and it's found instantly.
Story 3 - A customer files a GitHub issue (the magic one)
Sarah from Morphic opens an issue in our repo. GitHub never creates CRM records - that's a hard rule. But the Sorter still recognises her GitHub email as the same Sarah. The map now connects Morphic → Sarah → this issue. So the next time Josh asks "what's the status on Morphic?":
"Deal is in Discovery - and heads up, Sarah filed a bug about the import flow two days ago. Still open."
GitHub created nothing. It just made the answer smarter. That's what "bring your whole stack" is supposed to mean.
Story 4 - A hotel takes a booking enquiry (no "deals" anywhere)
maria@gmail.com emails asking about rooms in August.
Free-mail domain → no company created. Correct: she's an individual, not an organisation.
Maria is written to the hotel's Guests object.
Two-way exchange → gate fires.
The pipeline looks for "the thing this business tracks with stages," finds Bookings, reads its real stages, and creates a booking at inquiry.
Slack: "New booking inquiry from Maria, Sep 22–25. [Confirm] [Edit]"
Same pipeline. Same code. The word "deal" appears nowhere.
Story 5 - The deal goes quiet (nothing happens - that's the point)
No event fires at all. Nobody emails. That's precisely the problem. The nightly sweep reads only the summary pages - never raw history - and notices Morphic has been silent for twelve days while sitting in Discovery:
"Morphic's been quiet 12 days. The last thing Sarah asked about was migration effort, and the doc never went out. Want me to draft a follow-up?"
This is the "tap on the shoulder" feature everyone wants, and it costs about a few cents a day for a whole customer, because it reads summaries instead of history.
Story 6 - The AI gets it wrong (human always wins)
It moves Morphic to "Negotiation." That's wrong. The Slack message is the correction surface. Josh hits [Undo]. The old value comes back, and that field is permanently stamped as human-owned. From then on, the AI never touches that field again. Every automatically-written field carries three things: who wrote it, how confident it was, and which specific events it was based on. Human edits outrank machine edits, permanently, with an audit trail. We think this is non-negotiable. An automated system that can't be corrected isn't a product, it's a liability.
Story 7 - A brand-new workspace with zero objects (the CRM builds itself)
Data flows into the Mailroom and the index immediately - nothing blocks, nothing is lost.
Records with no home wait in staging.
One AI call reads the aggregate patterns and proposes a schema: "You talk to hotels and take reservations → create Guests, Properties, Bookings?"
A human accepts. Objects are created, each auto-classified into one of the five shapes.
Everything that was staged flows in automatically.
With one guardrail we insisted on: the system may build and manage the schema only while no human has touched it. The moment a person creates or edits an object type, schema ownership hands over to the human, permanently. That's the same "human wins" rule as Story 6, one level up.
Story 8 - Four objects share a shape
Covered above: existing match → routing rule → default → staging. She's never lost, never blocked, never costs a token to file.
Story 9 - The customer connects HubSpot
HubSpot data was curated by a human, so it's the highest-trust source we have. It skips the significance gate entirely and provisions directly. More importantly, HubSpot contacts merge into the same identity map as Gmail and Slack - so Sarah-from-HubSpot and Sarah-from-Gmail become one Sarah, not three. And a boundary we made explicit: our CRM is always the system of record. An external CRM is a sync target, never a replacement.
Story 10 - A marketing blast
A campaign email with an unsubscribe header gets dropped before it becomes a database row at all. Never reaches the Sorter, never creates anything, costs nothing. Roughly 85% of all volume dies here. Legitimate business mail passes. We verified both directions against real traffic.
Story 11 - Someone creates a custom object
They add "Launch Partners." One AI call classifies its shape and caches the answer forever. An admin can override it. Every future message routes by pure lookup - zero AI, forever. Built-in types skip the AI entirely, because we already know what they are.
Story 12 - A meeting ends
The transcript arrives and its follow-ups are ready. That's a rewrite trigger: every summary page connected to that meeting gets recompacted, so the next "how's Morphic?" already reflects what was said an hour ago.
Chapter 13 - Where this can still break
We wrote this list before shipping, not after.
And the honest "not built yet" list: two-way write-back to external CRMs, and full sync for external CRM records (today we lean on the connector normalising upstream).
Chapter 14 - What it adds up to
Strip away the floors and the archetypes and the gates, and the whole thing is four ideas:
Keep everything, forever, unprocessed. Then no mistake above is permanent, and no filter is a data-loss decision.
Do the cheap thing first and escalate rarely. Rules before models. Exact matches before fuzzy ones. Summaries before raw history. At every single fork, without exception.
Be generous with facts and strict with commitments. Log everything that happened. Create almost nothing automatically. One wrong deal costs more trust than a hundred missing ones.
The human always wins. Every automated write is labelled with who wrote it and what it was based on, every delivery message is also a correction surface, and a human correction is permanent.
The result is a CRM where the records fill themselves in, the quiet deals raise their own hand, and the thing your prospect said on a Tuesday afternoon about the migration timeline is still there - in the deal, in the summary, in the answer - three weeks later when it matters.
But the real thing we built wasn't an AI feature.
It was a system that knows when not to use AI.
Keep the raw data. Make the cheap decisions first. Build context once and reuse it. Escalate only when the system has something worth thinking about. And whenever the machine makes a decision, leave enough context behind for a human to understand, correct, or undo it.
That turned out to be the difference between putting an LLM on top of a CRM and building an ingestion system that can actually become the CRM's memory.
And that's the part I'm most excited about: the better the ingestion layer gets, the less the user has to do.
Chapter 3 - Architecture overview
The easiest way to hold the whole design in your head is to imagine all your company's information as mail arriving at a four-floor building.
The entire trick in one sentence: floors 1, 2 and 3 run on 100% of everything. Floor 4 runs on about 0.3% of everything. That's the cost saving. Everything else is detail.