Emerging Tech17 min readOct 17, 2026

Agentic AI Architecture for Mid-Sized Companies: A Reference Diagram and the Choices Behind It (2026)

Bhavesh Barot - Author

Bhavesh Barot

Founder & CEO

Free assessment

Find Out if an AI Agent Can Take the Task

Name one task your team repeats every day. We tell you whether an agent can do it or whether a tool you can buy already does, and we build a demo on your own sample files before you sign.

Bhavesh, our founder, reads every request and usually replies within 2 to 3 hours.
A man in a white shirt draws empty boxes joined by arrows on a whiteboard with an orange marker while a woman with short silver hair in a dark green sweater points at one box.

"A five-layer reference architecture for agentic AI at a mid-sized company: where work arrives, the agent, the tool layer where permissions live, your ERP and CRM, and the control layer. With the five workflow patterns, when to go multi-agent, how MCP fits, and a worked example from emailed purchase order to draft sales order."

Key Takeaways

  • 1An agentic AI architecture has five layers: where work arrives, the agent, the tool layer, your systems of record, and a control layer of approvals, logs, tests and monitoring.
  • 2Permissions belong in the tool layer and in the target system, not in the prompt. Instructions can be talked around; a permission the ERP refuses cannot.
  • 3Anthropic separates workflows (predefined code paths) from agents (the model directs its own steps) and recommends the simplest solution that works. Most business agents are mostly workflow with one agentic step.
  • 4Start with one agent and a few narrow tools. Split into several agents only when a step needs its own instructions and tests.
  • 5MCP is an open standard for connecting AI applications to external systems. One MCP server in front of your ERP can expose only approved actions to any MCP-capable agent.
  • 6Make every action safe to retry with an idempotency key, or the same order will land in your ERP twice.
  • 7Gartner warns that integrating agents into legacy systems can be technically complex. Budget the effort for integration, not the model.

The short answer

An agentic AI architecture for a mid-sized company has five layers: where work arrives, the agent (model, instructions and the loop that picks the next step), a tool layer that exposes only approved actions, your unchanged ERP and CRM, and a control layer of approvals, logs, tests and monitoring. Most first agents need one agent and a few narrow tools, not a multi-agent system.

We design, build and support custom AI agents, so weigh this page with that in mind. The diagram is ours, drawn from the way we structure builds on ERPs and CRMs. Outside facts come from Anthropic, the Model Context Protocol documentation, OWASP and Gartner, opened on 17 October 2026 and linked where they appear. Search figures are our own DataForSEO measurement for the US on the same day.

About 720 people a month search Google in the US for "agentic ai architecture", and another 260 for a diagram of one (DataForSEO, 17 October 2026). Most of what they find is drawn for a software company building an AI product: vector stores, model routers and agent swarms. A mid-sized manufacturer, distributor or services firm has a different problem. It already runs an ERP, a CRM and a shared inbox, and it wants an agent to do work across them without breaking anything.

This guide draws that architecture. It covers the five layers, the choice between a workflow and an agent, when a second agent is worth it, how MCP fits, a worked example, and the mistakes we see most. If you are setting the rules for your first agent, read it alongside our AI agent governance policy guide.

The reference architecture: five layers

Diagram of an agentic AI architecture in five layers: triggers such as email, portal, EDI, Teams and schedules; the agent with orchestrator, model, instructions and context; a tool layer of MCP servers, API connectors and document tools; systems of record such as ERP, CRM, help desk and files; and a control layer with approval queue, logs, test set, monitoring, secrets manager and off switch.
The path of one request runs top to bottom. The control layer on the right watches the agent and its tools. Tap the diagram to open it full size.
  1. Where work arrives. A shared inbox, a customer portal, an EDI or file drop, a Teams or Slack message, or a schedule. Pick one trigger for the first agent.
  2. The agent. An orchestrator that reads the request and decides the steps, a model (LLM) that does the reasoning, instructions that define the job and its limits, and context such as policies and past cases. Specialist steps, such as reading a messy PDF, can sit beside it.
  3. The tool layer.MCP servers, API connectors and document tools. Each exposes only the actions the job needs, signs in with the agent's own service account, and is where permissions are enforced.
  4. Systems of record. Your ERP (NetSuite, Odoo, SAP Business One and others), CRM (Salesforce, HubSpot), help desk and file stores. They stay as they are and keep their own permission checks.
  5. The control layer. An approval queue, logs of every tool call, a test set of your own past cases, monitoring and alerts, a secrets manager, and an off switch with a named owner.

Workflow or agent: decide how much the model chooses

Anthropic's engineering guide Building effective agents (19 December 2024) gives the cleanest split we know. Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents are systems where LLMs dynamically direct their own processes and tool usage. The same guide recommends finding the simplest solution possible and adding multi-step agentic systems only when simpler ones fall short.

For business work that advice is right. The order of steps in most back-office jobs is known: read the document, match it to a record, check the rules, prepare the action, get approval. The judgement sits inside one or two of those steps. So the reliable design is a workflow you can test, with the model making decisions inside it, rather than a free agent deciding the whole route each time.

The five patterns, with a mid-sized company example for each

Most agentic AI architectures in business combine two or three of these, with agentic AI orchestration deciding which step runs next.

Pattern (Anthropic's name)What it doesExample in a mid-sized company
Prompt chainingFixed steps in sequence, each using the last one's outputRead a supplier invoice, extract lines, match to the PO, draft the bill
RoutingSort the input, then send it to the right handlerShared inbox: new order, order change, invoice query or complaint
ParallelizationRun several checks side by side and combine themCheck credit limit, stock and pricing on a quote at the same time
Orchestrator-workersOne model splits a job and assigns the partsAn RFQ with 40 lines split by product family for pricing
Evaluator-optimizerOne model drafts, another critiques against rulesA customer reply checked against your returns policy before it goes to a person
Seen from behind, a woman with long auburn hair in a cream cardigan points at a monitor showing rows of connected colored blocks while a bald man in a charcoal quarter-zip watches, an orange notebook on the desk.
Most business agents are a fixed sequence of steps with the model deciding inside one or two of them.

One agent or several: when a multi-agent system architecture is worth it

SignalOne agentSplit into several
Number of distinct jobsOne job with a few stepsSeveral jobs owned by different teams
InstructionsFit on a page and test well togetherOne step needs long, specialist instructions of its own
TestingOne test set covers itEach part needs its own test set and pass mark
ToolsA handful, all for the same jobDifferent parts need different systems and permissions
Failure handlingOne place to stop and hand to a personParts can fail independently without stopping the rest

Every hand-off between agents is another place for context to be lost and another thing to test. Start with one. Our guide to multi-agent workflows with LangGraph and n8n on ERP systems covers the larger designs when you reach them.

AI agent orchestration: code, workflow tool or platform

US searches for "ai agent orchestration" run about 590 a month, and "what is ai agent orchestration" rose about 1,500% in a year (DataForSEO, 17 October 2026). Orchestration is the logic that decides what happens next: which tool, in what order, what to do with the result, when to retry and when to stop. You have three ways to build it, and most AI agent orchestration frameworks, platforms and tools fall into one of them.

OptionGood forWatch for
Code (for example LangGraph or plain code on the model APIs)Full control, proper tests, complex rules across systemsNeeds engineers to change it
Workflow tool (for example n8n)Simple, visual flows staff can readGets hard to test as rules grow
Vendor platform (Salesforce Agentforce, Microsoft Copilot Studio)Jobs that stay inside that vendor's product and dataReaching systems the platform does not connect to

Many companies end up with a mix: the platform agent for work inside Salesforce or Microsoft 365, plus a custom tool it can call for the system it cannot reach. We compare the code options in n8n vs LangChain vs CrewAI for enterprise, and the rent or build choice in our AI agent build versus buy guide.

The tool layer: APIs, MCP servers and where permissions live

The Model Context Protocol (MCP) documentation describes MCP as an open-source standard for connecting AI applications to external systems, and compares it to a USB-C port for AI applications. In architecture terms it is a standard shape for the tool layer. You build one MCP server in front of a system, decide exactly which actions it offers, and any MCP-capable client (the docs name Claude and ChatGPT among them) can use those actions and nothing else.

Three rules for the tool layer, taken from OWASP's guidance on Excessive Agency:

  1. Narrow tools. "Create draft sales order" rather than "run any query".
  2. Own credentials. A service account per agent, read-only unless a write action is listed.
  3. Check twice. The tool checks the request, and the target system refuses anything the account is not allowed to do.

We build custom MCP servers and API integrations for this layer; our custom MCP server guide lists the tools an ERP server should expose. See AI integration services for the systems we connect.

Want this diagram drawn for your systems?

Bring the job you want an agent to do and the systems it touches. In a 30-minute call with founder Bhavesh Barot we will sketch the five layers for your stack and point out where the integration effort sits. FactoryJet quotes a fixed price in writing after that call, and we can show working software on your own data before a contract.

Talk to the Founder

Worked example: an emailed purchase order becomes a draft sales order

Here is one request traced through the five layers. We have built purchase and sales order generation on several ERPs, including Odoo, NetSuite, SAP Business One and ERPNext, and this is the shape it takes.

  1. Trigger. A customer emails a PDF purchase order to the orders inbox. The agent picks it up.
  2. Route. The model decides: new order, change to an existing order, or something else. Anything else goes to the orders team untouched.
  3. Read. A document step extracts the customer, PO number, ship-to, lines, quantities and requested dates.
  4. Match. Tools look up the customer and each item in the ERP. Customer part numbers are mapped to your item codes using the cross-reference table.
  5. Check.In parallel: price against the customer's price list, stock against the requested date, and credit status. Any mismatch is flagged, not fixed silently.
  6. Guard against duplicates. The tool builds an idempotency key from the customer and PO number and refuses to create a second order with the same key.
  7. Prepare. The tool creates a draft sales order in the ERP, with the source PDF attached and every flag noted on it.
  8. Approve. The draft lands in the approval queue. A person checks the flags and confirms. Only then does the order release.
  9. Record.Every tool call, the model's decision at each step and the approval are logged against the PO number.

Notice where the model is used: routing, reading and spotting mismatches. The order of steps, the duplicate check and the ERP write are plain code. That is what makes it testable. Our post on RFQ-to-ERP agent architecture in manufacturing goes deeper on the quoting side.

The control layer: what keeps the agent trusted

ControlWhat it doesWithout it
Approval queueHolds actions over a limit for a named personEvery action reaches your systems unchecked
Logs and audit trailRecords every tool call, input, result and approvalNobody can say why an order changed
Test setYour own past cases, re-run before every changeA model update quietly changes behavior
Monitoring and alertsWatches handoffs, approver edits, failures and blocked actionsProblems surface as customer complaints
Secrets managerKeeps keys out of prompts and codeKeys leak into logs and repositories
Off switchLets a named owner stop the agent at onceA bad day becomes a bad week

The policy side of these controls, including who owns each one, is in our AI agent governance guide, which comes with a free policy template. How we run them after launch is on our AI agent monitoring and support page.

Six architecture mistakes we see in mid-sized companies

  1. Admin credentials. The agent connects with an admin login because it was quickest. Give it its own narrow account.
  2. One giant prompt. Five jobs in one set of instructions, so nobody can test any of them. Split the job into steps.
  3. No test set. The agent is judged on a demo, then surprises everyone on real data. Test on your own past cases.
  4. Screen clicking where an API exists. Automation that clicks through the ERP screen breaks on every update. Use the API or an export.
  5. Multi-agent first. Several agents built before one has worked. Start with one.
  6. No retry safety. A timeout triggers a retry and the order is created twice. Add idempotency keys to every write.

Gartner's June 2025 forecast that over 40% of agentic AI projects will be canceled by the end of 2027 notes that integrating agents into legacy systems can be technically complex, often disrupting workflows and requiring costly modifications. All six mistakes above are integration mistakes. None is about the model.

Build on a platform or build custom

If your work...Start withAdd custom work when
Lives in SalesforceSalesforce AgentforceThe job needs your ERP, EDI or a system Salesforce does not reach
Lives in Microsoft 365Microsoft Copilot StudioActions must run in systems outside Microsoft with your own approval rules
Crosses ERP, CRM, inbox and filesA custom agent with MCP servers or API toolsFrom the start
Is mostly one simple flowA workflow tool such as n8nRules grow beyond what the flow can test

We do Salesforce and Agentforce work as well as custom builds, so we are comfortable recommending either. Our comparison of Intercom Fin, Zendesk AI, Agentforce and a custom agent covers the support-desk version of this choice, and Agentforce pricing vs a custom AI agent covers the Salesforce one.

What else ranks for "agentic ai architecture", and how this guide differs

These are the business sites on Google's first page for "agentic ai architecture" in the US when we checked on 17 October 2026 (DataForSEO), with the title Google showed for each. We list them so you can compare; we have not reviewed their services, and the list leaves out forums and video sites.

SitePage title shown in Google
vectorize.ioDesigning Agentic AI Systems, Part 1: Agent Architectures
tredence.comBuilding Agentic AI Architectures: Blueprint for Autonomous ...
salesforce.comWhat Is Agentic Architecture?
onereach.aiAgentic AI Architecture: Building Business-Ready Systems ...
srajdev.comUnderstanding Agentic AI Architecture - Swapan Rajdev
galileo.aiAI Agent Architecture From Patterns to Governance

Several of these describe architectures for building AI products. This guide draws the architecture for a company that already runs an ERP and a CRM, traces one real request through it, and lists the integration mistakes that sink projects.

How FactoryJet builds agentic AI for mid-sized companies

We design, build, implement and support custom AI agents, and you own what we build: the code and repository from day one. AI and hosting accounts can run in your name, at cost. We start from your systems: which ones hold the data, which have usable APIs, and where the agent should stop and hand to a person. Then we show working software on your own data before you sign. FactoryJet quotes the build, and any support you choose, as fixed prices in writing after a short scoping call.

To see the work by system, start with AI agent development, AI agents for ERP systems, NetSuite AI agents, Odoo AI agents and sales order automation. If you are still deciding whether an agent is worth building, our guide to scoping an AI proof of concept that reaches production is the next read.

Sources

All pages opened and read on 17 October 2026.

Architecture call for mid-sized US companies

See your first agent drawn on your own systems, then working on your own data

Tell us the job and the systems behind it: ERP, CRM, inbox, portal. We will tell you which pieces are workflow, which need an agent, what the tool layer should expose and what a person should approve. FactoryJet quotes a fixed price in writing after a short scoping call.

Talk to the FounderSee AI agent development

Frequently Asked Questions

What is agentic AI architecture?
It is the way the parts of an AI agent system fit together: where work arrives, the agent itself (a model, its instructions and the loop that decides the next step), the tools it calls, the business systems those tools reach, and the controls around it such as approvals, logs, tests and monitoring. A good architecture keeps permissions in the tool layer and leaves your ERP and CRM unchanged.
What are the main components of an AI agent?
Six parts cover almost every agent: the model (LLM) that reasons, the instructions that define the job and its limits, the context it reads such as policies and past cases, the tools it can call, the orchestration loop that picks the next step, and the guardrails around it: approval rules, logs and an off switch. Remove any one and the agent either cannot act or cannot be trusted to.
What is the difference between an agentic workflow and an AI agent?
Anthropic draws the line clearly: workflows are systems where LLMs and tools are orchestrated through predefined code paths, while agents are systems where LLMs dynamically direct their own processes and tool usage. A workflow follows steps you wrote. An agent chooses its steps. Many good business systems are mostly workflow with one agentic step where judgement is needed.
What does orchestration mean in AI agents?
Orchestration is the logic that decides what happens next: which tool to call, in what order, what to do with the result, when to retry, and when to stop and hand the item to a person. It can be fixed code, a workflow tool, or the model choosing its own next step. Most reliable business agents mix fixed code for the order of steps with the model making decisions inside them.
What is the best AI agent orchestration framework?
There is no single best one. Code frameworks such as LangGraph suit teams that want full control and testing. Workflow tools such as n8n suit simpler, visual flows. Vendor platforms such as Salesforce Agentforce or Microsoft Copilot Studio suit work that stays inside those products. Anthropic recommends starting with LLM APIs directly and adding frameworks only when they earn their place.
Do I need a multi-agent system?
Probably not for your first agent. One agent with a few well-scoped tools handles most single jobs, such as turning emailed purchase orders into draft sales orders. Split into several agents only when one step is complex enough to need its own instructions and tests, or when separate teams own separate parts. More agents means more hand-offs to test and monitor.
Is multi-agent the same as agentic AI?
No. Agentic AI means AI that plans and takes actions through tools. Multi-agent is one way to build it, with several agents each doing part of the work and passing results between them. A single agent is still agentic AI. Multi-agent designs suit larger jobs; they are not a requirement.
What are agentic AI design patterns?
Anthropic names five common workflow patterns: prompt chaining (fixed steps in sequence), routing (sort the input, then send it to the right handler), parallelization (run checks side by side), orchestrator-workers (one model splits a job and assigns parts), and evaluator-optimizer (one model drafts, another critiques). Business agents usually combine routing and prompt chaining with one approval step.
What is MCP and why does it matter for AI agents?
MCP, the Model Context Protocol, is an open-source standard for connecting AI applications to external systems. Its own documentation compares it to a USB-C port for AI applications. For a company it means you can build one MCP server in front of your ERP or CRM that exposes only approved actions, and then any MCP-capable agent, including Claude and ChatGPT, can use it within those limits.
How do AI agents connect to an ERP like NetSuite, Odoo or SAP Business One?
Through the ERP's own API, wrapped in a tool or MCP server that exposes only the actions the agent needs, such as find a customer, check stock or create a draft sales order. The agent signs in with its own service account, read-only by default. Where the ERP has no usable API, the tool reads exports or files instead. Screen-clicking automation is the last resort.
Can AI replace an ERP?
No. The ERP remains the system of record for orders, stock, invoices and the general ledger, with its own controls and audit history. AI agents sit in front of it and do the reading, matching and data entry around it: turning emails and PDFs into ERP records, answering questions from ERP data, and flagging exceptions for a person.
Where should AI agent permissions be enforced?
In two places: the tool layer, which exposes only the actions the job needs, and the target system, which refuses anything the agent's service account is not allowed to do. Instructions in the prompt are not a permission control, because they can be talked around. OWASP calls the target-system check complete mediation.
Do I need a vector database or RAG for an AI agent?
Only if the agent must search a large body of text, such as hundreds of policy documents or product manuals. RAG (retrieval-augmented generation) fetches the relevant passages and gives them to the model. Many business agents need structured lookups instead: find the order, check the price list. Those are better served by direct queries through a tool than by a vector search.
What is agentic RAG?
Agentic RAG is retrieval where the agent decides what to search for, reads the results, and searches again if the first results are not enough, instead of running one fixed search. It helps when questions are vague or need several sources. It also costs more steps per answer, so use it where a single search often misses.
Which LLM should a business agent use?
Choose per step, not per company. A larger model suits the step that needs judgement, such as deciding whether an email is a new order or a change to an old one. A smaller, faster model often handles extraction and formatting. Keep the model behind your own tool layer so you can switch providers when a model is retired or a better one appears.
How do you stop an AI agent from creating duplicate records?
Give every action an idempotency key, a unique ID built from the source document such as the customer PO number and date, and have the tool check for it before creating anything. Retries happen: a timeout, a model re-run, a second email with the same attachment. Without that check the same order lands in the ERP twice.
What does an agentic AI architecture diagram usually show?
At minimum: the triggers where work arrives, the agent and its model, the tools or MCP servers, the systems of record, and the controls (approval queue, logs, test set, monitoring, secrets, off switch). Ours on this page draws those five layers and the path one request takes through them.
Should I build AI agents on Salesforce Agentforce or Microsoft Copilot Studio?
Use the platform your work already lives in when the job stays inside it. If cases and customer records live in Salesforce, Agentforce builds its actions from your flows, Apex and APIs. If the job crosses systems the platform cannot reach, or needs approval rules spanning your ERP and CRM, a custom agent or a custom tool the platform can call is the better fit.
What is the hardest part of agentic AI architecture?
Integration with existing systems, not the model. Gartner notes that integrating agents into legacy systems can be technically complex and often requires costly modifications. Mapping exactly which fields an action needs, handling records that do not match, and making every action safe to retry takes more work than writing the instructions.
How do you test an agentic AI system?
Build a test set from your own past cases, including ones the agent should refuse or hand to a person, and agree the pass mark before testing. Run it before go-live and again after any change to the model, instructions or tools. Add tests for hidden instructions in documents, permission refusals and cases just over each limit.
Where should a mid-sized company host its AI agents?
In a cloud account the company owns, close to where its other systems run, with the model provider account also in the company's name where possible. That keeps data, keys and logs under your control and avoids being locked to one vendor. Agents that run on a schedule need a server or serverless job; chat-triggered ones can run inside the tool where staff ask.
What skills does a team need to build and run agentic AI?
Integration engineering (APIs, data mapping, error handling) matters more than machine learning. You also need someone who can write and test instructions against real cases, someone who understands the business process well enough to set limits, and an owner who reviews results. Many mid-sized companies keep the process owner in-house and hire the engineering.
How do you build AI agent orchestration?
Write down the steps of the job first, then decide which steps are fixed and which need the model's judgement. Put the fixed order in code or a workflow tool, call the model only where a decision is needed, give each step its own tool with narrow permissions, add a stop-and-hand-off rule, and log every step. Test the whole flow on past cases before going live.
What are AI agent orchestration tools?
They are the software that runs the steps of an agent: code frameworks such as LangGraph, workflow tools such as n8n, and vendor platforms such as Salesforce Agentforce or Microsoft Copilot Studio. They differ in how much control and testing they give you and which systems they reach. Pick based on where your work lives and who will maintain it.
Bhavesh Barot - Founder & CEO
Written by

Bhavesh Barot

Founder & CEO

Founder & CEO of FactoryJet, an ecommerce and AI services company that has served 500+ businesses since 2014. Writes about ecommerce builds, AI agents, and AI search.

View Profile LinkedIn 10+ yrs exp.
Free quote
Founder replies in 24h