AI for CTOs: 8 architecture decisions that will determine the cost and security of your AI rollout

For most mid-sized companies, the cloud is a sensible starting point. Services such as Microsoft Foundry (formerly Azure AI Foundry), Amazon Bedrock or Google Vertex AI provide access to current models without buying hardware. You pay for usage, and the provider is responsible for availability and updates.

A hybrid model means you process some data in-house and some in the cloud. It makes sense if you have data that cannot leave the company for legal or contractual reasons, such as medical records. It costs more, because you maintain two environments and the rules for data flows between them.

Running exclusively on your own servers is a rare choice. The justification may be a ban on cloud processing in a client contract, a very large volume of queries, or a strategy to protect know-how. Before making such a decision, check whether the requirement really rules out the cloud, or only specifies the region where data is processed.

An AI model is not a choice for years. Vendors release new versions and retire old ones every few months, and prices and quality change faster than in most areas of IT. Your architecture must allow for this.

The simplest rule: the application should not call a specific vendor’s model directly from many places in the code. It is better to build a single intermediate layer that receives the request and passes it to the chosen model. Swapping the model then means changing the configuration and testing, not rebuilding the application.

It also pays to match the model to the task. Simple tasks, such as classifying emails or short answers, can be handled by smaller, cheaper models. Keep large models for analysing long documents and difficult questions. Whenever you change the model, run the same set of test questions and compare the answers.

Most business AI solutions work on the RAG model: before answering, the system searches for relevant passages in company documents and passes them to the model. It is like an employee who checks the right manual before answering a customer. The quality of answers therefore depends largely on the quality of the search.

This kind of search needs a database that finds passages similar in meaning, not just by keyword. The options are described in the table below. In a company that works on Microsoft 365 and Azure, Azure AI Search is the natural choice. If you already have a PostgreSQL database, the pgvector extension lets you start without a new system.

A typical mistake is treating the choice of database as the most important decision. Well-split documents, current versions and permissions carried over from the source system matter more. Without them, any database will return an outdated or unauthorised passage.

An AI solution without monitoring is a black box. You do not know what a single query costs, which questions end with a wrong answer, or why this month’s cost has doubled. Adding monitoring after a year usually means rebuilding a large part of the system.

The minimum is to log every model call with the few pieces of information listed below. On top of that comes a set of test questions with correct answers, run after every change. To collect the data, you can use tools you already have, such as Azure Monitor, or specialised AI monitoring tools.

Remember that query logs often contain personal data and company secrets. Set a retention period for them and limit access to a small group of people.

AI changes three areas of security. The first is the data sent to the model: personal data and company secrets must be detected and, where necessary, removed or masked before sending. The second is permissions: an AI assistant will find in seconds a document that was previously hidden only because nobody knew about it.

The third area is prompt injection: hiding an instruction in an email or document that AI may treat as a command. The risk grows when an AI agent has access to systems and can change or send something on its own. The basic safeguards are separating instructions from external content, checking outputs and requiring human approval before irreversible actions.

If your company is an essential or important entity under the KSC Act, AI solutions fall within the scope of its risk management. Include them in the information security management system that must be in place by 3 April 2027.

Every choice – of model provider, knowledge base or cloud – creates some dependence. You cannot avoid it entirely, but you can decide its level deliberately. A simple test: how long would it take to move the solution to a different model provider? If the answer is “several months”, the dependence is high.

High dependence is not always a mistake. A company that works in Microsoft 365 often deliberately chooses Copilot and Copilot Studio, because it gets ready-made integration with permissions, Teams and SharePoint. What matters is that this is a decision, not an accident.

Whatever you choose, make sure that the model instructions, test sets and configuration belong to the company and are stored outside the vendor’s tool. They are the most valuable output of the project.

An AI solution has to be maintained: the model updated, instructions improved, wrong answers dealt with and costs kept in check. A rollout without people who can do this ends with a tool that nobody uses after a few months.

A mid-sized company rarely needs a separate AI team. One or two people from the IT department who understand how models work, system integration and monitoring are usually enough. Add an external partner for the build phase and for larger changes.

You have three routes, which are most often combined: hiring, training your own people and working with a partner. When working with a partner, include knowledge and documentation transfer in the contract. Without it, you will swap one dependence for another.

An API is the link through which your application uses a model running at the provider. You pay for usage, measured in tokens, i.e. chunks of text. The alternative is to run an open-weight model, which can be downloaded and installed on your own hardware.

When comparing costs, set two figures side by side. The API cost is the monthly number of tokens multiplied by the price in the provider’s price list. The cost of your own model is servers with graphics cards, electricity, administrators’ time, monitoring and updates. You bear this second cost even when nobody is sending queries.

At the query volumes typical of a mid-sized company, an API is usually cheaper and simpler. Your own model makes sense when data cannot leave the company, or when the volume is very large and steady. Remember too that open models also age, and you have to replace them yourself.