AI Center of Excellence – how to build from scratch in 90 days (leadership guide)

AI CoE isn't for every organization at every moment. Triggers justifying its creation: (1) 3+ successful AI pilots in portfolio (proof of concept that AI has potential); (2) plans for 5+ parallel AI projects in next 12 months; (3) mounting governance concerns (EU AI Act compliance, multiple shadow AI, data security issues); (4) executive sponsorship for AI as strategic priority.

Without these triggers CoE becomes premature optimization. An organization with 1 AI pilot establishing a 10-person CoE likely wastes budget. Better path: 1-2 successful pilots → AI Steering Committee → CoE.

Realistic organizational scale for CoE: 200+ people (small CoE 3-5 people) or 500+ people (typical CoE 8-12 people). Smaller organizations are more efficient with 1-2 dedicated AI engineers reporting to CTO without formal CoE.

First and most important decision: whether CoE is centralized (all AI capabilities located in CoE, departments delegate work) or federated (small core CoE + AI champions in each department).

Centralized model: CoE is a mini-software house in the organization. Finance department wants AI in invoicing → delegates to CoE → CoE builds. Pro: architectural coherence, scale economies. Con: CoE bottleneck, weak ownership in departments, 'AI ghetto' risk.

Federated model: CoE 5-8 people delivers shared services (governance, infrastructure, training). Each department has 1-2 'AI champions' — department employees who learn AI tooling and lead AI projects in their department. Pro: faster decisions, ownership, distributed expertise. Con: quality variability, more coordination.

In 80% of cases federated wins for organizations 200-2000 people. Centralized makes sense only in very specific scenarios (highly regulated industries, very early AI maturity, extreme need for consistency).

Typical federated-model CoE has 5-12 people with defined roles. Each role has its own responsibility and typical candidate profile.

Head of AI / AI Director (1): CoE leader, reports to CEO or CTO. Senior with 10+ years experience, mix of technical and business. Responsible for AI strategy, AI Roadmap, executive communication.

AI Architect (1-2): responsible for architecture decisions, technology choices, integration patterns. Senior engineer with deep AI expertise (LLMs, agents, RAG, MLOps).

AI Governance Lead (1): responsible for AI policy, AI Act compliance, risk management. Mix of legal + technical understanding. Increasingly important role from 2026.

AI Engineers (2-4): responsible for shared infrastructure (RAG platform, agent framework, observability stack). Senior to mid-level engineers, not juniors.

AI Product Manager (1): responsible for project prioritization, CoE resource allocation, KPI tracking. Mix of product + AI understanding.

Plus: 1-2 people on enablement (trainings, internal docs, community building) depending on scale.

AI Steering Committee is a decision-making group with executive sponsors overseeing strategic AI directions in the organization. Typical composition: CEO or COO (sponsor), CTO, CFO, Head of AI, 2-3 business unit leaders. Cadence: monthly 90-minute meetings.

Steering Committee responsibility: quarterly AI Roadmap approval, large project authorization (above EUR 125k), major architecture change decisions, governance reviews, escalation resolution.

Without Steering Committee, CoE falls into 'consultancy without authority' trap — does work but decisions are blocked or revisited. Steering Committee gives CoE political cover for difficult decisions (decline projects, reject vendors, enforce governance).

Most common AI CoE failure: without clear charter, mandate and KPIs it's dissolved after 12 months as 'not delivering value'. These 3 elements must be established in the first 90 days.

Charter: 1-page document answering 'what CoE does, what it doesn't'. Does: AI strategy, shared infrastructure, governance, enablement. Doesn't: no 'magic AI solutions' for departments — departments own own projects. Without this boundary CoE is flooded with impossible requests.

Mandate: explicit authority from Steering Committee. CoE can: block non-compliant projects, require governance reviews before deployment, allocate shared resources. Without mandate decisions are questioned.

KPIs: 5-7 metrics measuring real CoE value. Top KPIs: (1) Number of AI projects in production (target +X/year); (2) Time-to-pilot for new AI use cases; (3) AI cost as % of business revenue; (4) Number of trained employees; (5) Governance audit findings (zero tolerance for AI Act gaps). KPIs reported quarterly to Steering Committee.

Practical 90-day CoE setup roadmap.

Days 1-30: foundations. Steering Committee setup (member recruitment + first meeting). Head of AI hiring decision (internal promote vs external hire). CoE charter draft + iteration. Initial AI Roadmap mapping (which projects CoE leads, which supports). AI governance framework draft.

Days 31-60: capability building. Recruitment of key roles (AI Architect, AI Governance Lead). Technology stack selection for shared infrastructure. AI Project Review process pilot for 2-3 existing projects. Training plan for AI champions in departments.

Days 61-90: operationalization. Full team operating. First quarterly Steering Committee meeting. AI Project Review process active for all new projects. First wave AI champions trained (5-10 people from different departments). KPI tracking dashboard active.

After 90 days: review with Steering Committee. Adjustment based on learnings. Plan for quarter 2.

Having led dozens of CoE setups we see repeating mistakes. Awareness = avoidance.

Mistake #1: too large team at start. CoE 15+ people in first year is a recipe for failure. Start small (5-8), scale based on demonstrated value.

Mistake #2: technical-only leadership. Head of AI with purely technical background and no business understanding can't engage Steering Committee. Technical + business mix is critical.

Mistake #3: no quick wins in first 90 days. CoE without visible early wins loses momentum. Plan 2-3 small but visible quick wins (e.g. internal AI policy launch, first cross-functional AI training).

Mistake #4: ignoring change management. CoE without champion network in departments is isolated. Invest in champion recruitment and training from day 30.

Mistake #5: governance as afterthought. Adding governance after 6 months = retrofit chaos. Build in governance from day 1.