AI Engineering & Custom AI Solutions

When do you need custom AI engineering rather than a Copilot licence?

When the answer has to come from data Copilot cannot see, or the AI has to live inside something you built. Microsoft 365 Copilot is very good within its boundary: content in Exchange, SharePoint, OneDrive and Teams, reached as the signed-in user, inside the applications people already use.

If your question is 'summarise this thread' or 'draft this from those documents', buy the licence and stop reading. Custom engineering starts where that boundary does: a case management system, a product catalogue, an equipment history, a decade of PDFs in a share nobody has indexed, or a customer-facing feature in your own product.

The second trigger is control. A licence gives you the model, the interface and the behaviour Microsoft chose. An engineered system lets you decide what the model may see, how the answer is grounded, what it does when it does not know, what gets logged, where inference runs and what the whole thing costs per query.

Most organisations need the licence and not the engineering. The ones that need the engineering usually know why, because they have already tried to solve the problem with a general tool and hit the same wall twice.

What does a RAG pipeline actually consist of?

Five stages, and the interesting work is in the first two rather than the model. Ingestion collects the source material and normalises it, which in practice means dealing with PDFs, scanned documents, spreadsheets, database records and pages that were never written to be machine-read.

Chunking splits that content into passages small enough to retrieve precisely and large enough to still make sense on their own, carrying the metadata that says where each passage came from. Indexing turns those passages into a searchable form, typically a vector index, often combined with keyword search because the two fail in different places.

Then retrieval selects the passages relevant to a question, and generation asks the model to answer using only those passages, with citations back to the source. Around all of that sits the part people forget: permission filtering, so a user only ever retrieves passages they are entitled to see, and a refusal path, so that when retrieval finds nothing useful the system says so rather than improvising. A pipeline without those two is a demonstration, not a product.

Where do RAG projects usually go wrong?

In retrieval, almost every time, and it gets blamed on the model. If the right passage never reaches the model, no amount of prompt engineering or model upgrading will produce a correct answer.

The second failure is stale content. An index built once at launch drifts from reality with every document that changes, and nobody notices, because a confidently wrong answer from a superseded policy is indistinguishable from a right one.

The third is permissions, where the pipeline reads everything with a single service identity and hands answers to whoever asks: an oversharing incident with extra steps. The fourth is having no way to tell whether a change made things better, which is why evaluation belongs in the build rather than after it.

How do you evaluate an AI feature before it goes live?

With a fixed test set that you write before you tune anything. That means a collection of real questions people actually ask, each with the passage that should be retrieved and a note on what a good answer contains, including the questions where the correct behaviour is to refuse. It is unglamorous work and it is the difference between engineering and guessing, because without it every change becomes a matter of opinion about whether the last few answers felt better.

Evaluate retrieval and generation separately, because they fail separately. Retrieval is measured on whether the right passage came back at all and how far down the list it was; generation is measured on whether the answer is supported by the retrieved passages, whether it cites them, and whether it declines when it should.

Add adversarial cases deliberately: questions the source material does not answer, questions that invite the model to speculate, and attempts to talk it out of its instructions. Then run the same set after every change, so improvement is demonstrated rather than asserted.

When is a small language model the right choice?

When the task is narrow, the volume is high, and the data should not leave your control. A large general-purpose model is a good default precisely because it handles anything, but most production tasks are not anything: classifying an enquiry, extracting fields from a form, rewriting text into a house style, routing a ticket.

For work with a tight, well-defined shape, a smaller model can meet the quality bar at a fraction of the cost and latency, and can run somewhere you choose rather than somewhere you send data to.

The trade is capability and effort. Smaller models are less forgiving of vague prompting, weaker at multi-step reasoning, and more likely to need real evaluation work and sometimes fine-tuning to reach the standard a large model reaches out of the box.

The sensible pattern is usually mixed rather than either or: prototype against a capable general model to establish what good looks like, then move the high-volume, well-defined steps to something smaller once you have a test set that proves the smaller model still passes.

Where should the model actually run?

There are three broad options and the decision is mostly about data, not performance. A hosted commercial API is fastest to build against and gives you the strongest models, at the cost of sending content to a third party under their terms.

A model hosted in your own Azure subscription keeps inference inside a boundary you control and inside your existing commercial relationship, which matters when a client contract or a regulator asks where processing happens. Self-hosting on infrastructure you own goes furthest on control and is the most work to run, and only makes sense when there is a concrete reason it has to be that way.

The practical advice is to decide the data position first and let it constrain the options, rather than picking a platform and then arguing about the data afterwards. Write down what categories of information the system will handle, what your own contracts and policies say about where they may be processed, and what you would have to tell a client if they asked.

That usually eliminates one option immediately and makes the remaining choice straightforward. Build the integration so the model is a replaceable component either way, because this part of the market moves faster than your application will.

How do you keep the cost of a production AI feature under control?

Sending an entire document when three retrieved passages would do is the single most common source of avoidable spend, and it usually also makes the answer worse, because relevant content gets diluted.

Instrument cost per request and per feature from day one, not after the first bill, and treat a rising cost per answer as a design signal rather than a billing problem.

What does governance look like for a custom AI application?

Human review is not a checkbox, it is a named responsibility with the authority to overrule the system.

Where the output influences a decision about a person, the bar rises and the question stops being technical. UK data protection law places real constraints on automated decision-making, and the emerging European rules turn on how a system is used rather than how it was built, so the same model can sit in a low-risk and a high-risk application.

The practical response is to record the purpose, the data, the human checkpoint and the review date at design time. Retrofitting that record after launch is considerably harder than writing it while the decisions are still being made.

How does this fit with the Microsoft stack you already run?

Usually as an extension of it rather than an alternative to it. Identity is the first integration point: an internal AI tool should authenticate through Entra ID like everything else, so access is governed by the same groups, the same Conditional Access policies and the same joiner and leaver process, rather than by a separate user list somebody maintains by hand. That single decision removes most of the access-review problems custom applications otherwise create.

Beyond identity, the pieces fit where you already have them: Azure for hosting and for keeping inference inside your own subscription, your existing Microsoft 365 content as a source where it is genuinely the right source, and Power Platform for the workflow around the AI rather than the AI itself.

The boundary worth respecting is that Microsoft 365 Copilot and a custom system solve different problems, and building a custom answer engine over content Copilot already handles well is expensive duplication. We would rather point you at the licence.

What does an engagement look like, and where does our scope end?

That conversation frequently ends with a recommendation not to build anything, because the problem turns out to be a search or a data quality problem wearing an AI costume, and solving it that way is cheaper and more reliable.

Where a build is right, the shape is a narrow prototype against real data and a real test set, an honest assessment of whether it clears the bar, then production hardening: permission filtering, evaluation in the pipeline, logging, cost controls, monitoring and a documented handover.

Our scope is the engineering and the governance around it. We are not a data cleansing service, we will not claim an accuracy figure we cannot demonstrate on your own test set, and where the requirement is really Copilot adoption and permissions readiness, that is our AI and Copilot service rather than this one.