AI for CTO – 8 architectural decisions for AI deployment in B2B (2026)

The first and most important decision. Affects cost, time-to-value, compliance and long-term growth path. In 2026 cloud-first is the default choice for most projects — fastest start, best tooling (Azure OpenAI, AWS Bedrock, Google Vertex), low CapEx.

Hybrid (cloud + partial on-prem) wins when you have regulated sensitive data (medical, financial, legal) that cannot leave your infrastructure but you want to use cloud compute for less sensitive workflows. Typically 30-40% higher TCO than pure cloud, but sometimes no alternative.

Pure on-prem (own GPUs, own infra, open-weight models) wins rarely: when industry compliance forbids cloud (defense, some fintech), when you have extreme volume (above 10M queries/month and cloud LLM cost becomes a problem), or when you have unique domain data you want to strictly protect.

Picking an LLM isn't a lifetime decision — you typically swap models every 12-18 months due to pace of progress. But the architecture around it must enable that swap. Meaning: abstraction via router (LiteLLM, LangChain, your own layer) INSTEAD of direct OpenAI SDK calls in 30 places.

Practical 2026 recommendation: GPT-4o or GPT-5 as primary for complex tasks, GPT-4o mini or Haiku 4.5 for simpler ones (classification, short replies). Claude 3.7 Sonnet for tasks needing long context or strong reasoning. Open-source (Llama 3.3, Qwen 2.5) as fallback and for sensitive data.

Antipattern: picking one model for the whole stack. Most organizations in 2026 use 3-5 different models in parallel, each for a different task type. Multi-model architecture is the standard today, not the exception.

If your project uses RAG (and most enterprise AI projects do), vector DB choice is fundamental. Four main options in 2026: Pinecone (managed SaaS, fastest start, expensive at scale), Weaviate (open-source + cloud, good balance), pgvector (PostgreSQL extension, for teams already on PG who want minimal operational complexity), Azure AI Search (for Microsoft ecosystem firms).

Most common mistake: picking Pinecone for everything. Great for MVP, but at 10M+ embeddings SaaS subscription costs quickly exceed self-hosted Weaviate TCO. Second trap: under-estimating embedding costs — embedding generation alone for 1M documents is ~EUR 700-2000, regeneration on model change is the same cost again.

Practical pattern: start with pgvector (if you have PostgreSQL) or Pinecone (if not). Migrate to self-hosted Weaviate after crossing 5-10M embeddings or budget >EUR 25k/year on SaaS subscription.

The most often skipped decision early on. Without observability, AI in production is a black box: you don't know when the model hallucinated, what costs per request were, why specific queries failed, where performance bottlenecks are. Adding observability after a year of production requires redesigning a major part of the system.

2026 stack: LangSmith (from LangChain, good for LangChain stacks), Helicone (open-source, model-agnostic), Datadog APM for traditional metrics, Grafana + OpenTelemetry for custom dashboards. Minimum viable observability: every LLM call logged with (input, output, latency, cost, model version, user_id, session_id).

Critical metrics to track from day 1: latency p50/p95/p99, cost per request, model accuracy (where measurable), hallucination rate (where detectable), token usage trend. Without these metrics you won't optimize quality or cost.

AI in production touches 3 security areas: input data protection (PII, sensitive data), model protection (if you have fine-tuned, IP), protection against prompt injection and data exfiltration. Each requires a separate architectural decision.

Input data: PII detection + redaction BEFORE sending to LLM (Presidio, own regex), optionally tokenizing sensitive data. Most GDPR-related AI incidents involve PII leakage to a public LLM prompt.

Prompt injection: especially in agents that have tool/data access. Standard defensive patterns: separation of system prompt from user input, output validation, sandbox for tool execution, monitoring anomalous behavior. Without these protections an agent with DB access is a business risk.

Choosing LLM provider, vector DB, cloud platform — each creates lock-in. Full escape from lock-in is unrealistic (always something), but you can consciously manage its level.

Low lock-in pattern: LLM router abstraction, vector DB in an API layer (not raw SDK), own observability infrastructure. Provider swap is then 2-4 weeks of team work instead of 6 months of refactoring.

High lock-in pattern (acceptable): deep integration with one ecosystem (Microsoft + Azure + Copilot), when it brings measurable benefits (quality, cost, support). This is NOT a mistake if consciously chosen.

The strongest predictor of AI project success isn't budget or technology — it's the competency of the IT team that has to maintain it. AI deployment without 2-3 seniors capable of maintaining AI in production almost always fails.

Required 2026 competencies: AI Engineer (LLMs, agents, prompt engineering, evaluation), DevOps with AI flavor (deploying, monitoring, cost optimization), Data Engineer (RAG, embeddings, vector DBs). These roles are expensive (EUR 50-85k annually in Poland for a senior) and hard to find.

Three strategies: external hiring (slow, expensive), upskilling your own team (6-12 months, less certain), consulting partnership (fastest, but creates external lock-in). Most organizations use a combination of all three.

The last decision: host open-source models in your own infrastructure or use managed APIs (OpenAI, Anthropic, Azure OpenAI). Most organizations pick managed APIs — fastest start and predictable costs.

Self-hosting open-weight models (Llama 3.3 70B, Qwen 2.5, Mistral Large) wins in 3 scenarios: very high volume (above 50M tokens/month, where API cost becomes significant), sensitive data requiring data residency, or IP strategy (you want to own your use case models).

Realistic Llama 70B self-hosting cost in production is EUR 12-38k annually (GPU compute, devops time, monitoring). Below 30M tokens/month managed APIs are almost always cheaper. Above 100M tokens/month self-hosting starts to be significantly more economical.