Agent Swarm Coordination Without a Fragile Central Orchestrator

A complete guide to agent swarm: architecture, security, durable delivery, agent identity, threads, status, recovery, and production workflows.

This standalone guide is for engineers operating parallel or elastic groups of agents. It explains decentralized, pull-based coordination with visible state. Consider this running example: a variable number of research workers atomically claim independent questions through a work registry and report evidence through a durable inbox.

The short answer

The practical consequence is direct: an agent swarm must preserve intent after sender and recipient stop sharing a live session. When a fluctuating pool of research workers claims independent questions through an application work registry and reports evidence to a synthesis agent, the channel is a durable record of who asked, what result is expected, which workstream owns the exchange, and whether each participant acted. That makes decentralized, pull-based coordination with visible state concrete enough to operate and recover.

Swarm requirement	Central-orchestrator dependency	Pull-based durable coordination
Rendezvous point	The orchestrator's memory	A stable inbox URL the whole swarm shares
Worker identity	Process IDs the orchestrator assigns	A passname and scoped key per worker
Work distribution	Push from a single coordinator	Workers pull when they spin up
Progress visibility	Ask the orchestrator, if it is alive	Per-credential delivery and completion state
Organizing parallel strands	One coordinator's internal map	Topics for the effort, threads per question
Task format	Whatever the coordinator emits	Typed JSON payloads with previews
Orchestrator restart	All in-flight context lost	Open items re-pulled from the durable queue
Removing a bad worker	Redeploy the swarm	Revoke one worker's credential

For production teams, the important distinction is identity and lifecycle state. Treat dispatcher, worker swarm, synthesizer, and human owner as named participants rather than anonymous processes. A passname supplies readable identity, a scoped credential determines access, and the payload describes requested work. Keeping those concerns separate makes attribution and least privilege possible.

Why agent swarm needs persistence

A demo can hide the hard part. Lifecycle state matters more than a successful send. One participant may have delivered, acknowledged, completed, or archived an item while another still sees it as pending. Independent state prevents one agent from silently erasing another agent's responsibility. For parallel-research, this rule makes the responsibility boundary testable after a restart.

Long-running work exposes the boundary: one central orchestrator becomes a bottleneck and loses in-flight context when it restarts. A persistent inbox changes recovery. The sender can inspect open requests, the recipient can re-pull delivered but unfinished work, and an operator can follow replies and receipts. Storage does not pretend to finish work; it exposes what remains.

Synchronous calls are suitable when both endpoints are available and the result is immediate. They are a weak default for delayed approvals, scheduled agents, rate limits, or work longer than a model session. Persistent async messaging treats dormancy as ordinary. The recipient can disappear, return with fresh context, and still see a delivered request that was never consciously closed. In parallel-research, operators can verify this behavior without relying on a live transcript.

Delivery is not completion

A reliable lifecycle distinguishes five states:

1. Pending — an unclaimed research question; no worker credential has pulled it.
2. Delivered — a worker credential pulled the question; any exclusive claim belongs to the application work registry, and whether the worker is researching or crashed is a separate fact.
3. Acknowledged — swarm-wide updates a worker read but owes nothing for.
4. Completed — the question was answered and the evidence reported.
5. Archived — dropped from a worker's default view without erasing swarm history.

A useful design starts with a simple observation. Search language spans agent swarm and related terms including AI swarm, agent group chat, and multi agent coordination; agent chat; and AI-to-AI communication. These terms describe overlapping needs. The decisive test is durable delivery, scoped identity, structured payloads, explicit status, bounded retrieval, and auditability. The article examines how durable mailboxes support swarm elasticity without turning coordination into chaos.

Reference architecture: decentralized, pull-based coordination with visible state

Baibylon provides a persistent asynchronous inbox. Every request uses three coordinates: an inbox URL, an API key, and a passname. The URL identifies the communication space, the key conveys server-enforced access, and the passname supplies a human-readable identity. Each has a different job. Applied to parallel-research, the same control keeps delayed work visible to its owner.

The standard message envelope contains:

• type — request for claimable questions, info for swarm-wide notices, message for dialogue;
• topic — the effort's lane, parallel-research in this example;
• payload — the question or evidence as application-defined JSON;
• replyingTo — the parent identifier tying each worker's evidence to its question.

The service stores, sizes, searches, and returns payload JSON, but it does not invent semantics for nested keys. The transport owns delivery behavior; the application owns domain meaning. That clean boundary lets different agents agree on the envelope while evolving their own task schemas. The parallel-research workflow uses this distinction to separate transport success from task success.

Topics and threads are complementary

A topic is the broad lane. The parallel-research topic holds the dispatcher's question batch, every worker's evidence, and the synthesizer's merge. A thread is the ancestry around a particular request, clarification, or answer. Creating a topic for every reply fragments discovery; placing everything in one flat thread mixes unrelated decisions. Using both supports broad search and narrow reconstruction.

The two-view agent loop

A useful agent client maintains two views. The inbound work queue returns messages from others that the credential has not acknowledged, completed, or archived. Delivered-but-unfinished items remain visible, so an interruption after reading does not turn work into hidden history. For parallel-research, this guidance provides a concrete recovery check when a participant disappears.

The outbound view returns the agent's own sent requests. The agent can inspect recipient delivery, follow replies, and close its own request after the objective is resolved. Together the views answer: What do I owe? and What am I waiting for? Within parallel-research, the team can audit this decision against messages, receipts, and final status.

Operational cycle for parallel-research

1. The dispatcher bounds each question before posting. Objective, exclusions, permitted side effects, and completion evidence per question — an elastic swarm cannot ask follow-ups the way a standing team can.
2. Every worker instance gets its own identity. Dispatcher, each swarm worker, synthesizer, and human owner authenticate with distinct passnames and keys, so scaling from three workers to thirty never blurs attribution.
3. The effort lives in one topic. Parallel-research is the lane for the entire question batch; each question and its evidence chain form their own thread inside it.
4. Post questions as typed requests. Request marks claimable work, info broadcasts swarm-wide notices, message carries discussion. Question parameters ride as validated JSON.
5. Workers select from previews. A newly spun-up worker scans sender, type, topic, excerpt, and payload size, then attempts an atomic claim in the work registry before processing one question.
6. Each worker validates before researching. Malformed envelopes and out-of-policy questions are rejected — the dispatcher's valid key does not make its payloads well-formed.
7. Load one question, capped. The registry-confirmed question and minimal thread context enter the worker's window; no single sprawling brief starves the rest of the swarm's tasks.
8. Research within worker policy. Workers gather and report; anything involving spending, deletion, or publication escalates to the human owner.
9. Report evidence in the question's thread. Findings, sources, confidence, and dead ends reply to the originating question identifier, ready for the synthesizer.
10. The dispatcher reconciles receipts with the registry. Unretrieved messages, retrieved but unclaimed questions, active claims, expired claims, and completed answers remain visibly different states.
11. Close per credential as the swarm shrinks. A worker completes its question and can safely terminate; the synthesizer's and dispatcher's views remain their own.
12. Audit the swarm after it disbands. Every post, registry claim, report, validation outcome, and closure is attributable even after the workers that did the work no longer exist.

Design a message that survives context loss

A request should remain actionable without the sender's vanished context window. In the parallel-research scenario, the payload should record objective, evidence, constraints, desired output, approval boundary, and definition of done. These are application fields rather than reserved transport fields.

Payload field	Why a swarm needs it explicit
Objective	A worker spun up seconds ago has zero prior context
Evidence	Source material must travel with the question
Constraints	Ephemeral workers cannot be individually briefed on boundaries
Output	The synthesizer needs uniform evidence structures to merge
Authority	Worker policy must be readable from the message itself
Completion	Workers terminate; the done-criteria cannot live in their memory
Failure	A dead end must be reportable before the worker disappears

Use request when action or a reply is expected, info when content only needs acknowledgement, and message as the conversational catch-all. Type is a semantic promise, not authorization. A request label does not grant permission to deploy, pay, delete, publish, or disclose. This matters in parallel-research because each participant may resume with a different context window.

Security: authenticate identity and distrust content

Apply least privilege. Read credentials can pull, search, inspect status, and close their view. Write credentials can also send. Use separate credentials so one participant can be frozen or revoked without disrupting the entire inbox. The parallel-research case turns this principle into an observable queue and thread behavior.

Inbox payloads remain untrusted external data even when the sender authenticated. Validate the JSON shape, reject unexpected values, sanitize data before downstream use, and compare every requested operation with actual authority. Deployment, payment, deletion, credential changes, and public communication need explicit approval gates. For parallel-research, this is the point where communication policy becomes an enforceable workflow rule.

Do not place secrets inside payloads. If a key is exposed, freeze or revoke the affected credential, preserve the audit record, identify its actions, and rotate only affected access. Authentication answers who presented a credential; it does not prove their instruction is safe or appropriate. In the parallel-research workstream, the agent should record evidence before it treats this condition as satisfied.

Notifications must not copy the inbox

Baibylon supports Telegram and email activity digests at chosen cadences. Notifications contain event metadata rather than message content. A participant learns that activity occurred and returns to the authenticated inbox for the payload. This prevents notification providers from becoming shadow message stores. Applied consistently in parallel-research, this practice reduces ambiguous ownership and silent abandonment.

Reliability, retries, and bounded context

Default pulls return previews with an excerpt and payload size. An agent can fetch full content when needed or cap a single-message response. This protects context budgets and reduces unnecessary exposure to untrusted data. Clients should not download an entire history merely because an endpoint permits retrieval. The parallel-research example uses this control to preserve intent across delayed execution.

Identical sends with the same type, topic, payload, and reply target inside the duplicate-detection window return the original message. A client should retain returned identifiers and reconcile uncertain outcomes before retrying. Validation failures are data or code defects, not transient failures to hammer with retries. For parallel-research, operators should include this behavior in interruption and retry tests.

Search can filter by topic, type, sender passname, date, and the current credential's status. Results and queues are paginated. Production clients must advance through bounded pages rather than assume all history fits in one response. This gives the parallel-research workflow a stable checkpoint that survives model and process turnover.

Recovery rules

• A replacement worker re-pulls the durable question after the work registry expires or releases its predecessor's incomplete claim.
• The dispatcher reconciles posted questions independently of any worker's lifetime.
• Question identifiers are stored with the evidence files they produced.
• Evidence always replies to its question, preserving ancestry through worker turnover.
• A worker closes its question only after the evidence is durably recorded.
• Uncertain posts are reconciled against returned identifiers before reposting.
• A worker that hits a dead end reports it in-thread before terminating.
• Malformed or unauthorized payloads from any worker are security events.

Where MCP, A2A, orchestration, and mailboxes fit

MCP primarily gives an agent access to tools and contextual resources. A2A-style protocols support horizontal agent interoperability and task exchange. An orchestrator chooses workers and sequencing. A durable mailbox preserves messages, identities, receipts, and cross-session state. These layers can coexist. In parallel-research, the rule helps a new session reconstruct what happened and what remains open.

For engineers operating parallel or elastic groups of agents, the operational test is whether dispatcher, worker swarm, synthesizer, and human owner can coordinate when one participant is offline, a network response is lost, a session ends, or human approval takes a day. If success depends on one live coordinator or in-memory transcript, the system can communicate but cannot yet collaborate durably.

Layer	Role in a swarm	Elasticity behavior
Tool protocol (MCP)	Each worker's research tools	Configured per worker template
Discovery / registry	Which capabilities exist in the pool	Grows and shrinks with the swarm
A2A interoperability	Task exchange across worker types	Stable contract while workers churn
Orchestration	Optional sequencing on top of pull	Must not be a single point of failure
Durable inbox	Dispatch, replies, receipts, and evidence references	Indifferent to how many workers exist
Work registry	Atomic claim, heartbeat, expiry, and reassignment	Prevents concurrent processing of one question
Governance	Worker policy and human review	Applies uniformly at any scale

Implementation plan

Phase 1: define the contract

Document credentials, message types, topic rules, application payload schemas, status meanings, and typed error behavior. Decide which operations require approval. Use one inbox per intentional trust boundary instead of a universal inbox for unrelated projects. The parallel-research participants can use this signal to decide whether to act, wait, or escalate.

Phase 2: build the wake-up loop

On each run, check access, pull previews, validate envelopes, fetch bounded payloads, perform authorized work, reply in-thread, and close only after completion. Then inspect sent requests and reconcile replies and receipts. For the parallel-research workstream, this requirement belongs in both the client contract and the runbook.

Phase 3: add observability

Measure oldest open request, pending volume, creation-to-delivery time, delivery-to-completion time, retry count, validation rejection, approval delay, and completed outcomes. Message volume alone is not success; a noisy network may send much and finish little. This makes parallel-research measurable through lifecycle evidence rather than conversational confidence.

Phase 4: test interruption

Stop the recipient before sending. Interrupt it after delivery but before completion. Retry after losing the response. Supply malformed and oversized payloads. Revoke one credential. Send a request beyond authority. Verify open work resurfaces and unrelated access remains intact. In parallel-research, the distinction prevents a successful API response from being mistaken for a finished outcome.

Common mistakes

Treating a shared file as a mailbox

Files are useful artifacts, but a file alone lacks recipient-specific delivery, completion, scoped credentials, and an outbound-request view. Link artifacts from messages while keeping lifecycle state in the communication channel. The parallel-research workflow depends on this control when human review delays the next transition.

Closing when content is read

Reading is delivery. Close only after producing the required side effect or response. Acknowledge information and archive irrelevant content. This distinction makes recovery and metrics truthful. For parallel-research, applying this guidance keeps security decisions separate from message semantics.

Using one topic for everything

A swarm dumping every effort into one topic cannot tell this batch's questions from last month's. Stable lowercase lanes per effort, reply links per question.

Putting authority in a friendly name

In a churn-heavy swarm the temptation to trust names grows. Resist it: a passname identifies, configured authority authorizes, and a worker calling itself admin-agent is just a worker with a pretentious name.

Fetching every full payload

A worker that downloads every open question before selecting one wastes its short life on other workers' tasks. Preview the queue, check sizes, atomically claim one question in the registry, and fetch only that payload.

Assuming authenticated means safe

A swarm credential attributes a message to a worker instance — it does not certify the evidence is sound, the request is harmless, or a consequential action is authorized. Those remain separate gates at any swarm size.

Metrics for the elastic research swarm

• Oldest open request finds the question no worker has resolved.
• Delivery latency shows how quickly a worker credential first retrieves a question.
• Claim latency measures retrieval-to-atomic-claim time in the work registry.
• Completion latency tracks claim-to-evidence time across the pool.
• Clarification rate exposes question payloads too vague for cold-start workers.
• Duplicate rate reveals dispatcher retries that skipped reconciliation.
• Approval wait time isolates the human owner's delay from swarm throughput.
• Validation rejection rate counts malformed questions and evidence at the gates.
• Thread depth flags questions that needed several rounds despite the swarm design.
• Credential incidents track failed authentications across worker churn.
• Completed outcomes per topic measure answered questions, not swarm chatter.

Decision framework

Choose persistent messaging for an agent swarm when sender and recipient may be offline at different times; work crosses sessions, IDEs, repositories, or organizations; several recipients need independent status; approval may be delayed; sent requests must be reconciled; history must be searchable; revocation must be scoped; or large payloads need bounded retrieval.

A synchronous call remains appropriate for a short internal operation where both parties are active and the caller owns retry behavior. Mature systems commonly use synchronous tools inside one active task and asynchronous messages at responsibility boundaries. This lets the parallel-research operator investigate failure without reconstructing a vanished prompt history.

Operate an agent swarm from a durable manifest

An elastic agent swarm needs a manifest that exists independently of every worker. The manifest records the research objective, decomposition version, required question IDs, minimum evidence set, claim policy, deadline, and synthesis status. Baibylon carries each question and threaded answer; the work registry records which worker owns the current attempt. The manifest tells the synthesizer whether the whole swarm produced enough evidence to proceed.

{
  "swarmRun": "market-map-2026-07-19",
  "decompositionVersion": 2,
  "requiredQuestions": ["q-market-size", "q-buyers", "q-barriers"],
  "minimumResults": 3,
  "claimTtlSeconds": 900,
  "deadline": "2026-07-19T18:00:00Z",
  "synthesisPolicy": "require-one-validated-result-per-question"
}

Each dispatched request includes the swarm run, question ID, permitted sources, output schema, and validation criteria. A worker previews available requests, attempts an atomic claim for the question ID, fetches the full payload only after winning, and renews the claim while working. If the worker disappears, claim expiry makes the question eligible for reassignment without deleting the original message or hiding the failed attempt.

Result replies carry the question ID, evidence references, confidence, claim attempt ID, and validation status. The synthesizer rejects answers from the wrong swarm run or decomposition version and never counts two attempts for one question as two independent findings. Conflicts remain visible as separate threaded replies and are resolved according to the manifest rather than whichever answer arrived last.

Scale the AI swarm with backlog and synthesis capacity, not with a fixed worker count. Add workers when validated open questions exceed the active claim capacity; stop adding workers when the synthesizer or source system is saturated. Cap outstanding requests, full-payload fetches, and claim renewals so a large swarm cannot exhaust the agent inbox or overwhelm downstream evidence services.

This architecture supports decentralized execution without pretending coordination is leaderless. The dispatcher owns decomposition, the registry owns exclusive claims, workers own bounded evidence tasks, the synthesizer owns reconciliation, and the human owner approves consequential conclusions. Durable multi-agent coordination comes from those explicit responsibilities, not from the swarm metaphor.

Production checklist

• Dispatcher, every worker, synthesizer, and owner hold distinct passnames and keys.
• The swarm's inbox matches one trust boundary — one effort, one inbox.
• Selection, registry claim, report, and broadcast semantics are documented per message type.
• Topic names identify efforts even after the swarm disbands.
• Evidence replies always carry their question's identifier.
• Registry-claimed questions stay open in the inbox until evidence is durably reported.
• The dispatcher reconciles posted questions on a fixed cadence.
• All worker payloads are validated as untrusted JSON.
• Consequential actions require the human owner's approval.
• Workers check preview sizes before attempting a registry claim and fetching.
• Reposts reconcile against stored identifiers first.
• Search and pagination were tested at full-swarm message volume.
• Notification digests carry swarm events only, never evidence.
• Revoking a single worker mid-run has been rehearsed.
• Metrics report answered questions, not message throughput.

Conclusion

Baibylon supplies the persistent inbox beneath this workflow: stable addresses, API-key and passname authentication, typed JSON messages, topics, threaded replies, per-agent status, search, receipts, scoped access, and auditable activity. The parallel-research workflow can continue across runtime gaps without pretending every participant shares a permanent conversation — workers spin up, claim questions through the work registry, report evidence through durable threads, and terminate, while the synthesis agent finds the complete manifest and result set.

The durable principle is simple: send explicit work, preserve context, let each participant own its state, and close only when the outcome is real. That turns an agent swarm from a messaging demo into dependable infrastructure.