AI Center of Excellence vs Distributed: 2026 Guide

If you run Group Strategy, the CIO office, or the transformation function at a multi-business-unit enterprise — a marketplace group of the Ricardo + IAZI + Homegate shape, a multi-vertical group like Enel or Eni, or any 2,000–20,000-employee company with three or more P&Ls — you've probably already had the meeting. One side wants a central AI center of excellence. The other side wants each business unit to own its own AI capability. Both sides cite the same McKinsey deck. Neither side is wrong, and that's exactly the problem.

This guide makes the argument that the AI center of excellence vs distributed debate is a false binary, walks through what each model actually owns, and lays out the Hub-and-Spoke architecture that most serious enterprises end up at — regardless of which side they started on.

The CoE vs Distributed Debate Is a False Binary

The question gets framed as a choice: do we build a central AI center of excellence that owns AI for the whole group, or do we federate the capability so each business unit ships against its own roadmap? Phrased that way, the debate is unresolvable, because both positions are defending something real.

The pro-CoE camp is defending scale economics, security baseline, and pattern reuse. They've watched three BUs sign three contracts with three model providers, build three eval harnesses, and ship three slightly different versions of the same fraud-detection workflow. They want to stop the waste. They're right.

The pro-distributed camp is defending domain context, delivery speed, and accountability to the BU P&L. They've watched a central team write a 60-page "Group AI Standards" document that nobody in the operating BU has time to read, while a junior product manager in one of the verticals ships a working assistant in three weeks using a credit card and ChatGPT Enterprise. They want to keep that velocity. They're also right.

The honest answer almost every serious enterprise lands on — IBM, JPMorgan, BMW, and most of the publicly documented mid-cap groups — is Hub-and-Spoke. A central AI CoE owns the platform, the standards, the reusable patterns library, and the cross-BU coordination. Embedded BU practitioners own domain context, customer-facing deployment, and delivery against business outcomes. The interesting work isn't picking a side. It's deciding, workflow type by workflow type, where to draw the centralisation line.

Get that line right and you compound: every pattern shipped in one BU becomes a reusable asset for the others. Get it wrong in either direction and you get the worst of both worlds — a CoE that's a bottleneck, BUs that are isolated, and a patterns library that nobody curates.

What an AI Center of Excellence Actually Owns

A well-scoped AI center of excellence is not "the team that does the AI." It's the team that owns the conditions under which AI can be done well, repeatedly, across business units. Concretely, that breaks into five domains.

1. The platform layer

This is the hard infrastructure floor. A serious CoE owns:

• An LLM gateway — one managed entry point for all model calls (OpenAI, Anthropic, Mistral, Llama, in-house). Centralised key management, rate limiting, cost attribution per BU, model fallback, and audit logging.
• Retrieval infrastructure — a shared vector store stack, document ingestion pipelines, chunking standards, embedding model selection, and the eval scaffolding to test retrieval quality.
• An evaluation harness — a reusable framework for offline evals, golden datasets, regression tests on prompt changes, and A/B infrastructure for shadow-launching new models.
• Guardrails and safety tooling — PII redaction, jailbreak detection, prompt injection defences, output policy filters. One implementation, audited once, used everywhere.
• Observability — latency, token cost, hallucination rate, user feedback signals piped into one dashboard the CoE and BU practitioners both see.

None of this is BU-specific. Building it five times is the canonical failure mode an AI center of excellence exists to prevent.

2. Standards and patterns

Standards are how the CoE turns one good decision into many. The high-leverage ones:

• Prompt patterns — a curated library of system prompts, few-shot structures, and chain-of-thought templates that have passed eval gates.
• Evaluation criteria — minimum bar for accuracy, faithfulness, safety, and latency before any AI-touched workflow goes to a customer.
• Security baseline — what data classes can hit which model providers, when human-in-the-loop is mandatory, how to handle EU AI Act high-risk classification.
• Procurement playbook — vetted vendor list, contract templates with the model-provider clauses already negotiated, DPAs reusable across BUs.

3. The reusable patterns library (the Ricardo lift-and-shift)

This is the highest-leverage thing a CoE does and the most chronically under-resourced. When one BU solves a problem — fraud detection on listings, content moderation at scale, semantic search across a long-tail catalogue, agent-assisted customer care triage — the CoE's job is to abstract the pattern into a reusable kit and ship it into the next BU that needs it. We come back to this in its own section because it's the argument.

4. Vendor and contract relationships

Group-level contracts with hyperscalers and model providers. One enterprise agreement with OpenAI, Anthropic, AWS Bedrock, or Azure OpenAI almost always beats five BU-level contracts on price, on data-handling clauses, and on the legal review burden. The AI CoE is also the natural home for the relationships with specialist providers (eval tooling, observability, guardrail vendors) that no single BU has enough volume to negotiate well on its own.

5. Cross-BU coordination and capability building

Internal community of practice, monthly demos across BUs, shared Slack channels, a job rotation programme for AI engineers, an upskilling curriculum for BU product managers and analysts. The CoE is the connective tissue. Without it, every BU re-discovers the same lessons six months apart.

What Distributed/Federated AI Owns

The mirror question is what the BU practitioners must own — not because the CoE doesn't want to, but because centralising it actively destroys value. The distributed AI model enterprise argument is strongest in five areas.

• Domain context. The CoE does not know that the Homegate listings team has a peculiar duplicate-detection edge case driven by a 2008 data-migration artefact. The CoE does not know that the customer care team at Ricardo has a category of refund disputes that doesn't exist on any other platform. AI workflows built without that context degrade fast.
• Customer-facing deployment. The BU owns the customer relationship, the brand voice, the support escalation path. AI-powered features that touch a customer must be shipped by the team that owns the customer, not by a central group that ships once and disappears.
• Regulatory specifics per market. A Swiss marketplace operating in CH + DE + AT has three slightly different consumer-protection regimes. An energy group operating across IT + ES + RO has three different regulator postures on AI in customer-facing channels. The CoE owns the baseline; the BU owns the market-specific overlay.
• BU-specific data integration. The connectors into each BU's CRM, ERP, ticketing, listings DB, and partner APIs live in the BU. The CoE provides the retrieval primitives; the BU plumbs them into its actual systems.
• Measurement against BU P&L. The only honest measure of an AI workflow is what it did to the unit economics of the BU that owns it. That number lives in the BU finance function, not in a group dashboard.

Centralising any of these is the failure mode that produces the "central CoE that nobody in the BU respects." If the BU can't move without group sign-off on a workflow change, the workflow change doesn't happen.

The Hub-and-Spoke Architecture in Practice

The model most multi-BU enterprises converge on — once they've burnt themselves on either pure extreme — is Hub-and-Spoke. A small central AI center of excellence (the hub) plus embedded practitioners inside each BU (the spokes). For a group of roughly 2,000 employees and 4–8 business units, the staffing reference looks like this.

Role	Location	Headcount	Primary remit
Head of AI CoE	Hub (Group)	1	Owns the platform, standards, patterns library; chairs cross-BU AI council
Platform engineers	Hub	3–5	LLM gateway, retrieval infra, eval harness, observability
Applied AI / patterns engineers	Hub	2–4	Build and harden reusable kits from BU successes; ship them into next BUs
AI security & governance lead	Hub	1	EU AI Act compliance, model-risk policy, audit posture
Enablement / community lead	Hub	1	Upskilling curriculum, community of practice, demo cadence
BU AI lead	Spoke (per BU)	1 per BU	Translates BU roadmap into AI workflow backlog; primary interface to the CoE
BU AI engineers / product	Spoke (per BU)	1–2 per BU	Build and ship BU-specific workflows on top of the CoE platform
Dotted-line analysts	Spoke (per BU)	As needed	Existing analysts upskilled, not net new hires — usage support and adoption

That's a central hub of 8–15 FTE and a spoke footprint of 2–3 dedicated AI practitioners per BU. Total: 20–40 FTE for a 2,000-person enterprise across 4–8 BUs. The hub reports to a Chief Data Officer, CTO, or directly to the COO depending on org. The BU spokes report to the BU leadership with a dotted line to the head of the AI center of excellence. That dotted line is what makes the model work — it gives the CoE eyes and ears in every BU without taking accountability for the BU's outcomes.

The Patterns-Library Insight (Ricardo Lift-and-Shift)

This is the argument that converts CoE-vs-distributed sceptics in both directions. The single highest-leverage thing a central AI center of excellence can do is run a patterns library — and the lift-and-shift between BUs of a marketplace group is the textbook case.

Picture a Swiss-style marketplace group with three verticals: a horizontal consumer marketplace (Ricardo shape), a real-estate vertical (Homegate shape), and a property-valuation data business (IAZI shape). Each BU has its own AI roadmap. Each runs into recognisable problems:

• The horizontal marketplace builds a fraud-detection model for suspicious listings. It works.
• The real-estate vertical has a near-identical problem: fake listings, brokers gaming the duplicate filter. Different domain, same pattern.
• The valuation business has a content-quality classifier that scores incoming data submissions. Functionally similar to the listings classifier.

In a pure distributed model, each BU rebuilds the pattern from scratch — new vendor selection, new eval harness, new prompt iteration, new failure mode discovery. Eighteen months of duplicated work. In a pure CoE model, the CoE builds "Group Fraud AI" centrally, ships it, and discovers it works for none of the three BUs perfectly because it was built without enough domain context.

In a Hub-and-Spoke model with a working patterns library, the sequence is different:

1. The horizontal marketplace ships its fraud-detection workflow first, owning it end-to-end. The CoE provides the platform (LLM gateway, evals, observability) but the BU owns delivery.
2. Once it's working in production, the CoE's applied-AI engineers abstract the pattern. They strip the BU-specific bits (the listings schema, the marketplace-specific category taxonomy) and document the reusable kit: the architecture, the prompt patterns, the eval rubric, the failure modes seen in production, the recommended human-in-the-loop thresholds.
3. When the real-estate vertical raises its own fraud problem six months later, the conversation does not start with "let's evaluate vendors." It starts with the patterns library entry. The real-estate BU forks the kit, layers its domain-specific data and category logic, and ships in weeks instead of quarters.
4. The patterns library entry gets updated with whatever the real-estate BU learned. The valuation business inherits both bodies of knowledge when it gets there.

This is the lift-and-shift compounding pattern. Each pattern shipped becomes cheaper for the next BU. A serious AI CoE measures itself partially on this metric: how many patterns are in the library, how many BU implementations are derived from a library entry rather than greenfield, and what the lead-time-to-second-BU is for each pattern. Most enterprises do not measure this. The ones that do compound much faster.

Hub-and-Spoke Anti-Patterns

The architecture is correct on paper and easy to mis-execute in practice. The five failure modes we see repeatedly:

• The CoE becomes a bottleneck. Every BU has to file a ticket and wait for the CoE to approve, provision, or build. The CoE has 12 people and a queue of 80 requests. BU velocity drops below what it was before the CoE existed. Fix: the platform must be self-service for the 80% of work BUs do routinely; the CoE only gates the genuinely novel or risky 20%.
• BU practitioners are isolated from each other. The CoE talks to each BU bilaterally; BUs never talk to each other. The patterns library is starved of feedback. Fix: a monthly cross-BU demo with mandatory attendance from each spoke; a shared Slack channel actively run by the CoE enablement lead; rotations between BU teams.
• No clear ownership of the patterns library. Everyone agrees the patterns library is important; no specific person's bonus depends on it. It rots. Fix: a named applied-AI lead inside the CoE whose primary KPI is patterns-shipped-and-reused, not models-deployed.
• Standards become bureaucracy. The CoE writes 60 pages of standards that BUs cannot operationalise. Compliance is performative. Fix: every standard must ship with a reference implementation. If the CoE cannot show you what compliance looks like in code, it has not finished writing the standard.
• No escalation path for BU-CoE conflicts. A BU wants to ship something the CoE's security lead is uncomfortable with. There is no defined arbitration mechanism. Things stall or get bypassed politically. Fix: a quarterly AI council (Group CTO, COO or CFO, BU MDs, Head of CoE) with explicit decision rights to resolve conflicts within two weeks.

How to Decide What Goes Central, What Goes BU

The honest answer to "where exactly do we draw the line" is workflow-by-workflow. We use a four-dimension decision matrix when scoping the operating model for a client. Score each candidate workflow type on each dimension; the result tells you whether it belongs in the hub, the spoke, or a deliberate hybrid.

Dimension	Score high → centralise (hub)	Score high → distribute (spoke)
Cross-BU recurrence frequency	Workflow recognisable in 3+ BUs (fraud detection, content moderation, search relevance, eval harness, RAG infra)	Workflow specific to one BU's market or product (energy-tariff optimisation, property valuation model)
Regulatory variation across markets	Same regulatory class across all BUs (e.g. internal productivity tools, group-wide GDPR)	Heavy market-specific regulation (consumer protection in CH vs DE vs IT; sector regulators)
Integration complexity	Generic integration surface (LLM gateway, observability, eval tooling)	Deep integration into BU-specific systems (BU CRM, BU listings DB, BU ERP)
Business-specific judgement	Decisions defensible by generic principles (security, cost, model selection)	Decisions require BU domain judgement (when to escalate to a human, brand voice, customer-tier policy)

Workflows that score high on dimensions 1–2 and low on 3–4 belong in the hub. Workflows that score the opposite belong in the spoke. The genuinely hard cases are the ones that score mixed — those are usually best solved as a "kit": the hub ships a reusable component, the spoke wires it into the BU. The decision matrix is not magic. It's a forcing function to make the conversation specific instead of ideological.

Sequencing: Distributed → CoE → Hub-and-Spoke (or Vice Versa)

Most enterprises arrive at Hub-and-Spoke by walking through one of two predictable failure paths. Knowing which path you're on is half the battle.

Path A: Started distributed, overcorrected to pure CoE

This is the more common path. The enterprise let BUs experiment freely for 18–24 months. Some good things shipped. Some embarrassing things shipped. Shadow AI usage was rampant. Costs spiralled because every BU bought its own LLM contract. A board-level incident (a hallucination that made the press, a data leak through an unsanctioned tool) triggered the panic. The CIO office stood up a central AI center of excellence with the explicit mandate to "get this under control." Six months later, BU velocity has collapsed, the CoE has a 90-day approval queue, and the head of the most profitable BU is openly hostile. The fix is to keep the CoE for platform + standards + patterns, then aggressively re-distribute delivery authority back to the BUs with the platform as a self-service substrate.

Path B: Started with a pure CoE, no BU traction

Less common but more painful when it happens. The enterprise hired a Chief AI Officer and a 20-person central team before any BU had asked for one. The CoE built a beautiful platform that nobody uses, because no BU practitioner was involved in the requirements. The BUs are quietly using whatever they used before. The fix is to embed CoE engineers inside each BU on rotation, force the CoE to deliver visible value into one BU at a time, and instrument adoption (gateway calls per BU, patterns reused per BU) as the CoE's primary KPI.

The settled state

Both paths converge on Hub-and-Spoke if you let them. The honest sequencing playbook for a 2,000–5,000-employee enterprise:

• Months 0–3: name the head of the CoE, do the workflow inventory across BUs, pick one BU as the lighthouse for the first patterns-library entry.
• Months 3–9: stand up the platform (LLM gateway, eval harness, observability) and ship the first reusable pattern with the lighthouse BU.
• Months 9–18: embed spokes into 2–3 more BUs; lift-and-shift the first pattern to a second BU; run the first cross-BU demo cadence.
• Months 18+: the patterns library has 5–10 entries, each used by 2+ BUs. At that point the operating model is self-evidently working and the political debate ends.

Cost Reference

Order-of-magnitude reference for a 2,000–5,000-employee enterprise across 4–8 business units. Numbers are fully loaded run-rates including people, platform, and vendor contracts. Excludes Phase-I-style discovery engagements.

Model	Annual run-rate	Headcount	Strengths / failure mode
Pure central CoE	€2–5M / year	15–30 FTE central, ~0 in BUs	Strong governance and platform; high failure-to-adopt risk if BUs were not co-design partners
Pure distributed	€3–8M / year (varies wildly)	0 central, 2–5 per BU	High BU velocity; high total cost because of duplicated tooling, vendor contracts, and patterns; weak group security posture
Hub-and-Spoke (recommended)	€1.5–4M central + €300K–1M per BU	8–15 central + 2–3 per BU	Compounds via patterns library; requires explicit governance; failure mode is bottlenecking
Outsourced ("we'll let a partner own it")	€1–3M / year in fees	0 internal, 5–15 partner	Fast start; expensive at steady state; institutional knowledge never lands inside the enterprise

The honest read: pure distributed is almost always the most expensive model on a total-cost basis even though it looks cheapest line by line, because nobody is amortising the platform or the patterns work. Pure CoE looks disciplined on paper and underperforms in practice if the BUs were not part of the design. Hub-and-Spoke costs roughly the same as a pure CoE for the central layer plus a thin spoke footprint per BU — and it's the only one of the four models that actually compounds.

SUPALABS First-Party Data

FAQ

Do we need a Chief AI Officer to run an AI center of excellence?

Almost never as a Phase-zero prerequisite. The head of the AI center of excellence is usually a Director-or-VP-level role reporting to the CTO, CDO, or COO — not a peer to those roles. Most mid-cap and large enterprises already have the talent distributed across existing teams (Head of Data Engineering, Group Security & Trust, existing ML leadership) and the right move is to elevate and connect what exists rather than build a parallel C-suite function. A Chief AI Officer becomes appropriate when AI is genuinely a board-level strategic lever and the CEO wants a single accountable executive — usually 18–24 months into a serious programme, not before.

How many people does an AI CoE actually need?

For a 2,000–5,000-employee enterprise with 4–8 business units, the central hub typically runs 8–15 FTE: a Head of CoE, 3–5 platform engineers, 2–4 applied-AI/patterns engineers, an AI security & governance lead, and an enablement lead. On top of that, you want 2–3 dedicated AI practitioners embedded in each BU (the spokes) reporting into BU leadership with a dotted line to the CoE. Below that staffing line, the CoE cannot keep up with BU demand and the model degrades into a bottleneck. Significantly above that line and the CoE drifts toward becoming a parallel IT function, which is its own failure mode.

What's the difference between an AI CoE and a Data CoE we already have?

A Data CoE typically owns the data platform, governance, and analytics tooling. An AI center of excellence owns the AI-specific layer that sits on top of (and depends on) the data platform: the LLM gateway, evaluation harness, prompt and pattern libraries, AI-specific guardrails and observability, and EU AI Act compliance posture. In most enterprises the two are run as sister functions under the same CDO or CTO, sharing infrastructure where it makes sense (vector stores often live in the data platform; the AI-specific eval and gateway layer usually does not). If you already have a strong Data CoE, you are roughly six months ahead in standing up an effective AI CoE.

Is the distributed AI model enterprise approach ever the right answer on its own?

Rarely, and only in two situations. First, very small portfolios — an enterprise with one or two business units where the overhead of a separate hub is not justified; here the "CoE" is effectively a small platform team inside the single dominant BU. Second, holding-company structures where business units are deliberately operated at arm's length, often with different ownership stakes, regulatory regimes, or eventual divestiture in mind; here you might run a very thin group standards function and let each company own its AI stack outright. For the typical multi-BU operating group, pure distributed is the most expensive model on a total-cost basis and exposes the group to security and reputational risk that compounds quietly.

How do we measure whether the CoE is actually working?

Four KPIs separate a working AI CoE from a vanity one. (1) Patterns-library cadence: how many reusable patterns have shipped, and how many BU implementations are derived from them rather than greenfield. (2) Lift-and-shift lead time: when the second BU adopts an existing pattern, how long until production. (3) Platform adoption: percentage of LLM calls in the group routed through the gateway, percentage of AI-touched workflows passing the standard eval harness before production. (4) BU satisfaction: a quarterly NPS-style score from BU AI leads on the CoE's responsiveness and usefulness. Metrics like "models deployed" or "use cases identified" measure activity, not value, and reward exactly the wrong behaviour.

How does this fit with the "AI center of excellence vs federated" debate I keep reading about?

The AI center of excellence vs federated framing is essentially the same debate as CoE vs distributed, just with different vocabulary — "federated" usually emphasises that each BU has equal standing and there is no single central authority. In practice, the same Hub-and-Spoke answer wins, because pure federation suffers the same compounding cost and security problems as pure distribution, and pure centralisation suffers the same adoption problems. The interesting decision is not central-vs-federated as a slogan but the per-workflow line-drawing we describe in the decision matrix. Treat anyone who answers "centralised" or "federated" without asking which workflow type you mean as someone who hasn't done this work before.

Sources & References