Orchestrators, workers, memory, and reinforcement learning: the architecture that will separate AI survivors from casualties.
Most AI systems are built like flash floods—impressive force, zero accumulation. Tokens rush through, work gets done, and nothing remains. Tomorrow starts from zero.
The systems that survive look different. They build layers. Each interaction deposits something useful: context about this customer, outcomes from that decision, patterns across thousands of similar requests. Over time, these deposits compress into something valuable—institutional knowledge that makes the next request cheaper, faster, and more accurate than the last.
This is sedimentary intelligence: AI architecture where value accumulates in strata rather than evaporating with each inference.
At Sonoran Capital Investments (SCI), this is the architectural pattern we look for—and the one we think separates survivors from casualties in the coming wave of AI infrastructure consolidation.
The survivors won't be the cleverest prompt engineers. They'll be the ones who built systems that learn, remember, and delegate.
The next generation of AI moats will not be built on model access or prompt cleverness. They will be built on layered architecture that minimizes cognitive friction—both human and machine—while maximizing signal density from operational workflows. The companies that survive will be the ones that treat token efficiency, latency discipline, and graceful degradation as first-class design constraints.
The Orchestration Layer
The default architecture for most AI systems looks like this: take the user's input, stuff it into a context window with all the relevant information you can find, send it to the best model you can afford, and hope for the best.
I call this the "one big model" fallacy.
It feels intuitive. Why wouldn't you want the smartest possible model thinking about every problem? Why wouldn't you give it all the context you have?
Here's why: because intelligence without organization is just expensive noise.
Think about how a well-run company works. You don't have the CEO personally handling customer support tickets, drafting contracts, debugging code, and scheduling meetings. That's not because the CEO couldn't do those things. It's because that architecture doesn't scale, and more importantly, it doesn't leverage specialization.
The pattern that works looks different:
Orchestrator: Routes, decides, delegates. This is a smaller, faster model whose job is traffic control. It classifies intent, assesses urgency, queries the memory layer for relevant context, and decides which specialist should handle this particular task. It doesn't try to be smart about everything. It tries to be smart about one thing: delegation.
Workers: Specialized tasks with focused context. Each worker is right-sized for its job. A triage worker doesn't need a 200K context window. A diagnosis worker might need more context but can use a more capable model because it's only invoked when diagnosis is actually needed. Workers have narrow scope, which means they can have narrow (and cheap) context.
Memory Layer: Persistent state across invocations. This is what transforms a collection of API calls into a system. More on this shortly.
Here's the heretical part: the orchestration overhead is almost always cheaper than context stuffing.
When you stuff context, you're paying for tokens you don't need on every single request. When you orchestrate, you're paying a small routing cost to ensure that expensive context only gets loaded when it's actually required.
In our maintenance dispatch system, switching from "one big prompt" to an orchestrated pipeline reduced our per-request cost by roughly 5x. Latency dropped by 3x. And crucially, we could now reason about the system—we knew which worker handled which task, which made debugging possible.
Your AI shouldn't be a genius doing everything. It should be a competent manager delegating to specialists.
This is not a novel observation. It's how every complex system that actually works has been organized since the beginning of systems thinking. The novelty is that AI makes the delegation boundaries different, not that delegation is suddenly a good idea.
The Memory Problem Nobody Talks About
Here's a question that exposes the maturity of an AI system: what does it know about this customer that it didn't learn in the current request?
For most systems, the honest answer is: nothing.
Every request starts from zero. Every context window is assembled fresh. Every inference is stateless. The model might be brilliant, but it has the institutional memory of a goldfish.
I call these systems "context vampires." They survive by sucking tokens—stuffing the context window with everything that might be relevant because they have no other way to access the past. And they pay for it, literally, on every single request.
The alternative is memory architecture. Real memory. On-disk, queryable, persistent memory.
I think about AI memory in three tiers:
Working Memory: This is the context window. It's expensive, volatile, and limited. Treat it like RAM—use it for the current operation, but don't pretend it's storage.
Episodic Memory: On-disk logs of past interactions. What did this customer ask about last week? What happened when we dispatched to this property before? What did the technician report? This data is cheap to store and cheap to search. It turns every new request into a continuation of a relationship rather than a cold start.
Semantic Memory: Vector stores plus structured knowledge graphs. This is queryable truth—the facts about your domain that don't change request-to-request. Equipment specifications. Property hierarchies. Vendor capabilities. Compliance requirements. This layer lets you ask "what do we know about X?" without stuffing X's entire history into every prompt.
In PropTech—the domain we spend most of our time in—the organization models get specific:
Property hierarchies: Portfolio → property → unit → equipment. When a tenant reports "AC broken," the system should immediately know which unit, which property, which portfolio, what equipment is installed, and when it was last serviced. That's not magic. That's a queryable graph.
Temporal context: Seasonal patterns (HVAC failures spike in July in Phoenix), maintenance history (this unit has had three compressor issues), tenant lifecycle (this is a move-in week, which changes urgency calculations). Time matters. Memory should encode time.
Relationship graphs: Which vendor is certified for this equipment? Which technician has the best resolution rate for this issue type? What's the service history with this vendor? These relationships are the difference between routing work intelligently and routing work randomly.
Here's the economic argument: if your AI can't remember that this HVAC unit failed twice last summer, you're paying for the same diagnosis three times.
Memory isn't just a feature. It's the foundation of unit economics at scale.
The companies that treat memory as an afterthought will find themselves in an uncomfortable position: they'll need ever-larger context windows to approximate what a proper memory layer provides for free. And context windows have a cost curve that doesn't flatten.
Reinforcement Learning: The Secret Weapon
Let me describe two pipelines.
Pipeline A was designed by a talented team. They wrote careful prompts, tuned the routing logic, tested extensively before launch. It works well. It will work exactly as well in six months as it does today, unless someone manually updates it.
Pipeline B was designed by a slightly less talented team. The prompts are okay. The routing logic is okay. But they built a feedback loop: every outcome is logged, scored, and fed back into the system. The pipeline continuously adjusts its behavior based on what actually worked.
In six months, Pipeline B will be significantly better than Pipeline A. Not because of superior initial design, but because it learned from 180 days of production outcomes while Pipeline A stood still.
This is the argument for reinforcement learning in pipeline architecture.
The problem with static prompts is that they encode a hypothesis about what works. Sometimes that hypothesis is right. Often it's wrong in ways you won't discover until production. And even when it's right initially, the world changes—customer behavior shifts, edge cases emerge, upstream data quality fluctuates.
Static systems don't adapt. Learning systems do.
Here's how we think about RL for pipelines:
Reward signals: What does "success" mean for this pipeline? In maintenance dispatch, it might be resolution success (was the issue actually fixed?), time-to-completion (how long from report to resolution?), cost-per-outcome (what did we spend to fix this?), and customer satisfaction (did the tenant feel taken care of?). These are measurable. They can be logged. They can be optimized.
Policy learning: Given these reward signals, which decisions lead to better outcomes? Which worker should handle which task type? How much context is enough for a given decision? When should the system escalate to a human? These are policy questions, and the answers should come from data, not intuition.
Online adaptation: The pipeline improves continuously from production feedback. Not through retraining—that's slow and expensive—but through lighter-weight mechanisms: adjusting routing weights, updating confidence thresholds, reranking prompt variants based on outcome data.
Some practical implementation patterns:
Bandit algorithms for model routing: You have three models that could handle a particular task. A bandit algorithm explores which one works best for which input types, then exploits that knowledge to route future requests optimally. This happens automatically, without human intervention.
Outcome-weighted prompt selection: You have five prompt variants for a particular worker. Instead of A/B testing forever, weight selection toward prompts that historically produced better outcomes for similar inputs. The best prompt wins, dynamically.
Confidence calibration from historical accuracy: The model says it's 90% confident. But historically, when it said 90%, it was actually right 72% of the time. Calibration adjusts for this, which feeds into escalation logic—you know when to trust the system and when to flag for human review.
Here's the line I keep coming back to:
Your pipeline should get cheaper and better over time. If it doesn't, you've built a very expensive static system.
The RL layer is what makes this possible. It's what turns operational data into operational advantage. And it's what separates systems that compound from systems that merely cost.
A technical note: none of this requires the kind of RL infrastructure that trained AlphaGo. We're talking about contextual bandits, Thompson sampling, relatively lightweight online learning. The math is well-understood. The challenge is instrumenting your pipeline to collect the right signals and close the loop.
The Determinism Spectrum
There's a lazy framing I hear constantly: "AI is unpredictable, so you can't use it for anything important" versus "Just use rules if you need predictability."
Both positions miss the point.
The real question isn't "is AI predictable?" It's "how do we design systems that are predictable enough for their context while still capturing the flexibility that makes AI valuable?"
I think about this as a spectrum—what I call the determinism spectrum:
Hard determinism: Cached responses, exact matches. If we've seen this exact input before and the response was validated, just return the cached response. 100% predictable. Zero inference cost.
Soft determinism: Constrained outputs, structured generation. The model must respond in a specific JSON schema. The output must be one of N allowed values. Temperature is zero. You're still using a model, but you've bounded its behavior. Call it 95% predictable.
Guided stochasticity: Temperature control, output validation, retry logic. The model has some freedom, but there are guardrails. Invalid outputs get caught and regenerated. Confidence thresholds trigger human review. Maybe 80% predictable, with explicit handling for the other 20%.
Full stochasticity: Raw LLM, high temperature, minimal constraints. Creative, flexible, expensive, and genuinely unpredictable. Maybe 50% predictable in terms of meeting your actual requirements. Useful for some tasks. Dangerous as a default.
The architecture principle is straightforward: push decisions left on the spectrum whenever possible.
If you can cache it, cache it. If you can constrain it, constrain it. If you can validate it, validate it. Only accept full stochasticity when you've consciously decided the flexibility is worth the unpredictability.
Tools for determinism:
Response caching with semantic similarity: You don't need exact matches. If the current request is semantically similar to a previously validated response, you can often reuse it. This requires a vector store and a similarity threshold, but the infrastructure is commodity at this point.
Structured output schemas: JSON mode, function calling, typed responses. These aren't just conveniences—they're constraints that bound model behavior. A model that must return valid JSON with specific fields is dramatically more predictable than a model that can return arbitrary text.
Validation layers and retry logic: Assume the model will sometimes produce invalid output. Build validation into the pipeline. When validation fails, retry with a modified prompt or escalate. This is basic defensive programming, applied to inference.
Confidence thresholds for human escalation: When the model's confidence falls below a threshold, don't guess—escalate. This turns unpredictability into a handled edge case rather than a silent failure.
The mistake I see most often is treating determinism as a binary: either the system is fully automated or it's fully manual. That's a false choice. The interesting design space is hybrid: automated where confidence is high, human-in-the-loop where it's not, with explicit policies governing the boundary.
PropTech Pipeline Case Study: Maintenance Dispatch Orchestration
Let me make this concrete with a complete example from property operations.
The Problem:
A tenant reports "AC not working." This is one of the most common maintenance requests in residential property management. It sounds simple. It isn't.
To handle this properly, you need to:
- Triage urgency (is this a safety issue? A comfort issue? What's the outside temperature?)
- Check warranty and service history (is this equipment under warranty? Has it been serviced recently?)
- Identify likely cause (is this a refrigerant issue? An electrical issue? A thermostat issue?)
- Dispatch the right technician (who's qualified? Who's available? Who has the best track record with this issue type?)
- Schedule appropriately (tenant availability, technician routing, parts availability)
- Track resolution (was it fixed? First-time fix rate? What was actually wrong?)
A naive approach stuffs all of this into one big prompt. Here's what that looks like:
"You are a maintenance dispatch assistant. Here is the tenant's message. Here is the property information. Here is the equipment inventory. Here is the maintenance history for the past two years. Here is the vendor list. Here is the technician availability calendar. Here are the SLA requirements. Please triage, diagnose, and dispatch."
This prompt might be 15,000 tokens of context before the model even starts thinking. Cost per request: $0.15-0.30. Latency: 8-12 seconds. And the model is trying to do everything at once, which means it does nothing particularly well.
The Architecture:
Here's how we'd structure this as an orchestrated pipeline:
┌─────────────────────────────────────────────────────────────┐
│ ORCHESTRATOR │
│ (Claude Haiku - fast routing, low cost) │
│ - Classifies intent and urgency │
│ - Queries memory for context │
│ - Delegates to specialized workers │
└─────────────────────────────────────────────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ TRIAGE │ │ DIAGNOSIS │ │ DISPATCH │ │ COMMS │
│ WORKER │ │ WORKER │ │ WORKER │ │ WORKER │
│ (Haiku) │ │ (Sonnet) │ │ (Haiku+DB) │ │ (Haiku) │
│ │ │ │ │ │ │ │
│ - Urgency │ │ - Root cause│ │ - Tech match│ │ - Tenant │
│ - Category │ │ - Parts req │ │ - Scheduling│ │ updates │
│ - SLA check │ │ - History │ │ - Routing │ │ - Owner rpts│
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
│ │ │ │
└──────────────┴──────────────┴──────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ MEMORY LAYER │
│ - Vector store: maintenance history, equipment specs │
│ - Knowledge graph: property→unit→equipment relationships │
│ - Outcome logs: what worked, what didn't, why │
│ - RL feedback: routing success rates, cost per resolution │
└─────────────────────────────────────────────────────────────┘How it flows:
- Request arrives. The orchestrator (Haiku, cheap and fast) classifies the intent ("maintenance request"), extracts the category ("HVAC"), and queries the memory layer for property context.
- Orchestrator assesses urgency. Given it's an AC issue, it checks: what's the current temperature at this location? (API call, cached). Is this a vulnerable tenant (elderly, medical condition)? (Memory lookup). This takes milliseconds and determines routing priority.
- Orchestrator dispatches to triage worker. Triage worker confirms urgency classification, checks SLA requirements for this property/owner, and determines if this is a "dispatch now" or "schedule within 24 hours" situation. Focused context, focused task.
- If diagnosis is needed, orchestrator invokes diagnosis worker. This worker uses a more capable model (Sonnet) because root cause analysis is genuinely hard. It gets relevant context from memory: equipment make/model, maintenance history, previous similar issues. It produces a hypothesis about likely cause and required parts.
- Dispatch worker handles technician matching. This is mostly database logic with light AI: query available technicians, filter by certification, rank by historical performance on this issue type, check routing efficiency. The AI layer handles any ambiguity; the database handles the mechanics.
- Comms worker generates appropriate messages: confirmation to tenant, work order to technician, notification to property owner if required. Each message is templated with AI filling in the specifics.
Cost analysis:
- Naive approach (one big prompt): ~$0.15-0.30 per request, 8-12 second latency
- Orchestrated approach: ~$0.03-0.06 per request, 2-4 second latency
That's a 5x cost reduction and 3x latency improvement. At scale—thousands of maintenance requests per day—this is the difference between a viable business and a money pit.
The RL loop:
Here's where it gets interesting. Every resolution generates feedback:
- Did the diagnosis match the actual repair? (Logged when technician closes the work order)
- Was the right technician dispatched? (Measured by first-time fix rate)
- Was the urgency classification correct? (Validated by actual resolution time and tenant feedback)
- What did this cost? (Actual invoice versus predicted cost)
This feedback flows into the memory layer and adjusts system behavior:
- If Diagnosis Worker is consistently wrong about a particular equipment type, its prompts get reweighted or more context gets included for that equipment type.
- If a particular technician has poor first-time fix rates for HVAC but excellent rates for plumbing, the dispatch routing learns this.
- If urgency classification is triggering too many false emergencies, the thresholds get calibrated.
The result: the pipeline improves weekly without code changes. Six months in, it's meaningfully better than it was at launch—not because anyone rewrote it, but because it learned from thousands of outcomes.
What makes this defensible:
The models are commodity. Anyone can call Claude or GPT. What's not commodity:
- The memory layer encoding property hierarchies, equipment lineage, and maintenance history
- The outcome data linking dispatch decisions to resolution quality
- The RL policies tuned on actual operational feedback
- The domain-specific organization models that make context retrieval useful
This is the moat. It's not clever prompts. It's institutionalized operational memory that compounds over time.
The Selection Pressure Is Here
I want to return to where we started: survival.
Yegge's frame is useful because it's honest about what's coming. There will be winners and losers. The losers won't all be bad companies—some will just be companies that made reasonable-sounding architectural decisions that didn't scale, didn't adapt, and couldn't survive cost pressure.
The companies that die:
- Single-model wrappers that compete on prompt quality against every other wrapper
- Stateless systems that pay for context assembly on every request
- Static architectures that can't learn from production outcomes
- "AI features" bolted onto workflows rather than AI-native workflow redesigns
The companies that survive:
- Treat token cost like infrastructure cost: measure it, budget it, optimize it
- Build memory as a competitive moat: proprietary data, organization models, outcome logs
- Use RL to compound improvements: systems that get better from usage, not just from engineering time
- Own the workflow data that trains the system: not just user input, but operational outcomes
- Design for determinism by default, stochasticity by exception: predictable systems that degrade gracefully
The selection pressure is economic. AI inference has a cost. Latency has a cost. Unpredictability has a cost. The systems that survive will be the ones that minimize these costs while maximizing the value they deliver.
In eighteen months, this pressure will be brutal. The question isn't whether your AI is smart. It's whether your pipeline is efficient. It's whether your memory compounds. It's whether your system learns.
Closing
I don't have this figured out. Nobody does. The field is moving too fast for certainty.
But I've become convinced that the architectural patterns described here—orchestration, memory, reinforcement learning, determinism discipline—are not optional flourishes. They're survival requirements.
The companies that treat AI like a magic box will be surprised when the economics don't work. The companies that treat AI like a systems engineering problem will build something that lasts.
At SCI, this is the lens we apply to every vertical AI company we evaluate. Not "is the demo impressive?" but "does this architecture survive at scale?" Not "are the prompts clever?" but "does this system learn from outcomes?" Not "is there AI?" but "is the pipeline built to compound?"
If this helps you build something that survives, steal it.
— jason