Generalist systems can write, classify, extract and reason on many subjects. This does not mean that they are familiar with the company's internal rules, vocabulary, contractual exceptions or the level of caution expected by its teams. The answer may seem convincing and yet remain unusable in a regulated context.
The temptation then consists in asking for a "proprietary model". The expression reassures. However, it combines four very different assets: a trained model, data held by the company, an evaluation method and an application system. In many projects, lasting value is less found in the weight of the model than in the cleaned corpus, annotated examples, decision rules and business feedback loop.
So the right goal is not to own the biggest model. It is to build the smallest system capable of making the right decision, at the right cost, with a known level of error.
1. The figures to keep in mind
Stanford's AI Index 2026 points out how much the model layer remains concentrated: more than 90% of notable AI models in 2025 were from industry. An SME or mid-sized company can rarely compete realistically on foundational training. Specialisation, orchestration and business data offer a much more accessible opportunity.
The price of capability is changing very quickly. The AI Index 2025 estimated that the inference cost of a system at a level comparable to GPT‐3.5 had dropped more than 280 times between November 2022 and October 2024. This decline changes the economic calculation: an architecture frozen around an expensive model can become obsolete before even being amortised.
Inference benchmarks show the same acceleration. MLCommons received 17,457 results from 23 organisations for MLPerf Inference v5.0. Six months later, version v5.1 included submissions from 27 participants and the best systems delivered gains of up to 50% in some scenarios compared to v5.0. Buying a theoretical performance without a routing or replacement mechanism therefore creates an expensive dependency.
Internal evaluations must also remain modest. A case study published on Hugging Face compared several variants of RAG on only 156 queries: the naïve system was 70.5% on the selected metric, against 63.8% for a so-called contextual variant; dense retrieval reached 69.2%, 63.5% for hybrid search. This result does not condemn any technique. It shows that a method deemed superior can degrade a real case if the data, parameters or measurement are not adapted.
Finally, an average score is not enough. The RTEB benchmark requires a minimum of 1,000 documents and 50 queries per dataset, with a coverage of 20 languages. This scale gives an order of magnitude: validating a retrieval system on ten questions chosen by the project team produces a demonstration, not proof.
2. The data balance sheet: the asset before the model
Before talking about architecture, Logiks recommends a data audit. It is one page long and answers nine questions.
| Dimension | Question of decision | Expected evidence |
|---|---|---|
| Use case | What decision or production needs to be improved? | current process and target outcome |
| Population | Which documents, objects or users are concerned? | volumes and segmentation |
| Signal | What information makes it possible to produce the correct answer? | positive examples and counter-examples |
| Ground truth | Who knows how to say a result is correct? | rule, expert or observed outcome |
| Coverage | Are rare and expensive cases present? | distribution by scenario |
| Freshness | How quickly does knowledge change? | update frequency |
| Rights | Can data be used for this purpose? | legal basis, licence, contract |
| Sensitivity | What data should not come out of a perimeter? | classification and location |
| Reversibility | Can we export the corpus, the labels and the tests? | formats, versions and documentation |
This assessment avoids a frequent error: training a model to compensate for a documentary problem. If the procedures contradict each other, if the reference answers do not exist or if the experts do not agree, the model will reproduce uncertainty. It can even make it more difficult to see behind fluent wording.
The useful dataset is not necessarily massive. It must above all represent decisions that matter. A thousand redundant examples of a simple case can be less valuable than fifty counter-examples covering the boundaries of the rule.
3. Decision tree: from standard to training
Customization is decided by steps. Move to the next level only when the previous one cannot satisfy a measured requirement.
3.1. Tier 1 — Use a standard model
This option is appropriate when the task is generic, the context is in the query and the error is easily verifiable. Reformulating a text, proposing title variants, or turning notes into a record generally does not warrant training.
The real question is about control: what data is sent, how long are they stored, and does the supplier use them to improve its services? These responses fall under contract and configuration, not a commercial promise.
3.2. Tier 2 — Structure prompting and tools
A robust prompt describes the role, entry, exit, constraints and refusal criteria. It's not just a long set-up. Validators, calculators, rule engines and API calls must support what requires deterministic accuracy.
Example: In order to establish a commercial proposal, the model can draft the argument. Marginal calculation, availability of resources and discount eligibility remain under the control of deterministic services. Language is probabilistic. The financial rule should not be left to the model.
3.3. Tier 3 — Add knowledge retrieval, or RAG
The RAG provides the model with extracts recovered from a corpus. It is appropriate when knowledge often changes, needs to be cited or cannot be reliably integrated into weights. An HR policy, technical catalogue or contract basis belongs to this category.
However, the complete system includes much more than a vector base: ingestion, chunking, metadata, access rights, lexical and semantic search, reranking, citations, versioning and measurement. A false answer may come from the model, but also from an absent document, from improper extraction, from a permissions filter or from a poorly classified passage.
3.4. Tier 4 — Adapt the model
Fine-tuning becomes relevant when the defect is stable and repetitive: specific format, terminology, classification style, sequence of actions or expected behaviour. It can reduce the length of prompts and improve regularity. It does not provide a good knowledge base for facts that change every week.
This includes supervised adjustment from input-output examples, light LoRA-type adaptation and preference training. The choice depends on the supplier, the open or closed model, the volume of examples and the possibility to retrain. In all cases, the evaluation dataset must remain separate from the training dataset.
3.5. Tier 5 — Training a specialised model
This level is defended when the domain has an inaccessible signal to existing models, when the volume of use justifies the investment, or when latency, embedded and sovereignty constraints impose it. It concerns industrial vision, time series, sensor detection or certain scientific vocabularies more naturally than a simple editorial assistant.
This requires funding for data retrieval, training, infrastructure, assessments, safety, maintenance and future replacement. A high-performance prototype is only the first asset. The ability to reproduce it is another.
4. Designing a dataset that resists reality
A good dataset reflects real-world conditions, including its uncomfortable areas. Its construction can follow six layers.
The population. First, the target universe is defined: country, languages, client types, product versions, channels and periods. Without this scope, an accuracy of 95% says nothing.
Sampling. Frequent cases ensure volume. Rare cases provide security. In a documentary check, the corpus must include incomplete documents, degraded scans, layout changes and legal exceptions, even if they represent few documents.
Annotation check. Two experts must be able to read the same rule and produce the same label. Examples, exclusions and an escalation procedure are added. The rate of agreement between annotators reveals the ambiguities of the task; high disagreement is not corrected with more GPUs.
The provenance. Each item retains its source, date, license, transformation and version. This traceability facilitates deletion, correction or incident analysis. It becomes essential when personal data or copyrighted content are concerned.
Separation. Training, validation and testing datasets must not share duplicates or almost identical variants. For temporal data, the test is located after training. For clients, one can separate by account to measure generalization.
The red-team dataset. It gathers hostile, ambiguous or out-of-range entries: prompt injection, contradictory document, unexpected language, missing data, prohibited instruction. We don't expect production to discover these scenarios.
5. Three-stage evaluation
A business AI system is not valid with a single note. It is measured at three levels.
The first concerns the component: extraction accuracy, search recall, calibration of a score, compliance with a JSON scheme, latency and cost. These metrics help the technical team locate a defect.
The second concerns the task. Is an answer useful, complete, circumscribed and in accordance with the rules? A panel of experts scores blinded cases. The criteria must be observable. "Good quality" is not a criterion; "identifies the three incompatible clauses and cites their location" is one.
The third concerns the job result: processing time, recovery rate, income, risk avoided or satisfaction. Winning five precision points does not create any value if the user must always read each output. Conversely, a less dramatic system can remove data entry and reduce the time to one day.
Thresholds depend on the cost of errors. For an inspirational tool, a false positive has little consequence. To block a payment or direct a patient, tolerance changes dramatically. Thus, false positives, false negatives, abstentions and cases transmitted to a human are documented separately.
6. Practical case: specialise a pre-sales assistant
A B2B company wants to respond more quickly to its tenders. It has 8,000 documents, but only 1,200 are still valid. Historical responses sometimes contain outdated commitments.
The first reflex would be to fine-tune a model on the winning files. It would be dangerous: the old formulations could be stored without respecting the current catalogue.
Rather, the chosen architecture begins with a versioned documentary database. Documents are segmented by product, country, date of effect and level of confidentiality. A hybrid search retrieves passages, then a model produces a response with citations. A rule engine prohibits promises about SLAs, prices and certifications missing from the repository.
After several weeks, the assessments show a consistent weakness: the model respects the facts, but the structure of the compliance matrices is not good. A slight adjustment is then carried out on 600 reviewed examples. The content remains in the RAG; fine-tuning learns the format and sequence of reasoning.
The project creates four reusable assets: a clean corpus, a set of 350 test questions, a risk taxonomy and an error history. The model can change. These assets remain.
7. Costs: reason in terms of total cost per decision
Token price does not capture the economics of the system. The total cost shall include at least:
- collection, cleaning and data rights;
- Annotation and arbitration of disagreements;
- experimentation, training and storage;
- Inference, research, tools and logs;
- evaluation in each version;
- human supervision and the handling of errors;
- safety, compliance and response to incidents;
- migration to another model or supplier.
This sum is then reduced to the number of decisions that are actually useful. A cheap option that requires 40% of outputs to be reworked is more expensive than a well-routed premium service. Conversely, using the most capable model for every query wastes budget if 70% can be handled by a light option.
A mature architecture practices routing: light model for simple tasks, more capable model for complex cases, deterministic rules for calculations and human for exceptions. It also sets a budget per file. Without a ceiling, apparent quality can mask a silent drift.
8. Ownership, confidentiality and reversibility
"Your data remains your data" does not answer all the questions. It is necessary to distinguish source data, annotations, embeddings, prompts, adapted weights, logs and outputs.
The contract specifies who can use them, where they are hosted, how long they are kept, how they are deleted and what is exportable. For a closed model, access to the service does not amount to weight ownership. For an open model, the licence may impose conditions on use or redistribution.
The GDPR applies as soon as personal data enters the cycle. minimisation, purpose, retention periods, data-subject rights and security must be integrated. A pseudonymization reduces certain risks without automatically making the data anonymous.
Reversibility is being tested. Once a year, the team should be able to export the necessary corpus, metadata, evaluation dataset, configurations and logs, and then run a sample on an alternative. A never-exercised exit plan remains a hypothesis.
9. Signals that training is premature
Several symptoms invite you to go back:
- The team does not know how to formulate the business metric;
- the experts do not agree on the reference answers;
- the corpus contains several undated versions;
- the demonstration is based on less than fifty selected cases;
- The main problem is research or access rights;
- knowledge changes faster than the retraining cycle;
- the cost of an error was not estimated;
- no owner is designated to maintain the dataset.
In this context, training adds a layer of opacity. Priority work is to clarify the decision, clean up knowledge and build evaluation.
10. Road map in twelve weeks
Weeks 1 and 2: scoping. The team defines the decision, the current process, the cost of errors and the data allowed. It selects fifty representative cases.
Weeks 3 and 4: simple reference. Baseline is based on off-the-shelf capabilities and minimal architecture. Quality, latency, cost and abstention rates are measured.
Weeks 5 and 6 — Data. The corpus is cleaned, versioned and documented. The evaluation dataset reaches several hundred cases, with critical exceptions.
Weeks 7 and 8: comparison. Prompt, RAG, tools and possible fine tuning are tested on the same dataset. The results are analysed by segment, not only on average.
Weeks 9 and 10: Pilot. Real users work within a limited perimeter. Corrections, circumventions and refusals are recorded.
Weeks 11 and 12: Decision. The committee compares value, risk, total cost and reversibility. It decides to deploy, adapt, suspend or redirect.
The main deliverable is not a model. This is reproducible evidence that the system improves a decision without shifting the risk elsewhere.
11. Pre-investment reality check
Four verdicts are enough. Standard. The available capacity meets the need. Enriched. Knowledge must be found in a corpus. Adapted. Stable behaviour deserves adjustment. specialised. The signal, volume and constraints justify training from scratch.
There is no virtue in moving up these levels. The choice must remain reversible. A mature team can also go back to a simpler solution as the market progresses, costs fall or the need is less differentiated than announced.
The final comparison is based on the same set of cases, under the same conditions, with the full cost and human review included. It reports failures. It shows the fragile segments. It preserves exit options. The committee can then decide without being impressed by the fluidity of the demonstration, because the decision is based on a measured boundary between what the system accomplishes, what it refuses and what an operator still needs to secure.
12. Frequently Asked Questions
12.1. How many examples does fine-tuning require?
There are no universal numbers. A few hundred coherent examples can improve a narrow format; several thousand remain insufficient if labels contradict each other. A learning curve, built at 100, 300, 1,000 and more examples, shows whether the gain justifies the collection.
12.2. Does RAG prevent hallucinations?
No. It can reduce factual errors and improve citations, but the model can ignore a passage, misinterpret or combine incompatible sources. The system must measure retrieval, check citations and know how to abstain.
12.3. Does an open-source model guarantee sovereignty?
No. Sovereignty also depends on hosting, components, skills, licences, equipment and the ability to maintain the system. Downloadable weights improve control, without removing all dependencies.
12.4. Should every interaction be retained to improve the model?
No. retention must meet a purpose, duration and level of sensitivity. A sample of cleaned and annotated cases is often better than a lake of logs containing unnecessary data.
12.5. What is the best protection against obsolescence?
Separate assets. The corpus, rules, tests and orchestration layer must be able to survive the replacement of the model. This reduces the cost of migration and retains the knowledge gained.
13. What Logiks recommends
Start by forming the evaluation dataset before choosing the model. Test the simplest system. specialise only when a stable, large and measured defect justifies it. Protect the business dataset, rules and test cases above all: they are the ones who turn a technology available on the market into an operational advantage.
14. Main sources
- Stanford Institute for Human-Centered AI, AI Index Report 2026, chapters Research & Development and Economy: https://hai.stanford.edu/ai-index/2026-ai-index-report
- Stanford Institute for Human-Centered AI, AI Index Report 2025: https://hai.stanford.edu/ai-index/2025-ai-index-report
- MLCommons, MLPerf Inference v5.0 Results: https://mlcommons.org/2025/04/mlperf-inference-v5-0-results/
- MLCommons, MLPerf Inference v5.1 Results: https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
- Hugging Face, Charles Azam, feedback from RAG: https://huggingface.co/blog/charles-azam/rag
- Hugging Face, Retrieval Embedding Benchmark (RTEB): https://huggingface.co/blog/rteb
- NIST, AI Risk Management Framework, Measure function: https://airc.nist.gov/airmf-resources/airmf/5-sec-core/
- CNIL, recommendations for the development of the systems: https://www.cnil.fr/fr/developpement-des-systemes-dia-les-recommandations-de-la-cnil-pour-respecter-le-rgpd