The Hidden Engine of Generative AI: From Raw Data to Intelligent Systems
GENERATIVE AI · THE COMPLETE DATA JOURNEY
The Hidden Engine of Generative AI
From Raw Data to Intelligent Systems
Discover the preparation, learning, retrieval, and engineering behind a useful AI response.
A generative AI application can summarize a report or explain an unfamiliar codebase in seconds. The visible response is only the final step of a much larger process.
Behind it are decisions about data sources, text extraction, numerical representations, model learning, retrieval, and evaluation. Weaknesses in any of these stages can undermine the result.
This guide follows that journey from raw data to a deployed system. It focuses on language applications while explaining which parts train a model and which parts supply information without changing it.
Generative AI has a model pipeline and an application pipeline
Building a foundation model and building an application around one are different projects. A team can create a useful AI assistant without training a large model from scratch.
| Journey | Main stages | Result |
|---|---|---|
| Model development | Prepare training data, train, adapt, and evaluate. | Model weights and associated configuration. |
| Application development | Connect data, retrieve context, call a model, and validate. | A service that performs a defined task. |
Important distinction: Adding documents to a retrieval index does not automatically train the language model. It makes information available for future requests.
Raw data needs context, ownership, and purpose
Source material may include technical manuals, approved support conversations, code repositories, or product documentation. Availability alone does not make a dataset suitable.
Define the task first. A deployment assistant needs current runbooks and service information. Unrelated documents add processing work without necessarily improving its answers.
Record where information came from
Preserve source identifiers, document versions, update dates, and access restrictions. This provenance supports corrections, deletion, debugging, and trustworthy citations.
Check that the material is approved for its intended use. Minimize unnecessary personal information and secrets before sending content to external processing services.
A practical quality question
“If the system uses this document to answer a question, can we identify its owner, verify its currency, and confirm that the caller is allowed to see it?”
Clean and structure data without removing its meaning
Raw files contain more than useful content. PDFs may repeat page headers. HTML may include navigation. Scanned documents require OCR, which can misread symbols and numbers.
- Extract: Recover text and relevant structure.
- Normalize: Repair encoding and formatting problems.
- Deduplicate: Identify repeated or near-identical material.
- Validate: Inspect extraction quality and missing sections.
- Version: Record the dataset and processing configuration.
Cleaning should preserve meaningful details. Removing punctuation indiscriminately can damage code. Flattening a table can separate a value from its heading and change its apparent meaning.
Common mistake: Splitting duplicated documents across training and test sets can make evaluation look better than performance on genuinely unfamiliar data.
Tokenization and embeddings serve different purposes
Tokenization creates model inputs
A tokenizer divides text into units and maps them to token IDs. A token may represent a word, part of a word, punctuation, or another sequence, depending on the tokenizer.
Token count affects context capacity and, for many hosted services, usage cost. Do not assume one word equals one token.
Embeddings support numerical representation
Inside a language model, token IDs are mapped into learned representations. A retrieval embedding model can separately represent an entire passage or query as a vector.
Those passage vectors support similarity search. They are not compressed documents that can reliably be decoded back into the original wording.
Tokenization prepares sequences for a model. Retrieval embeddings help compare content. Neither operation, by itself, establishes whether a statement is true.
How training turns examples into model behaviour
Pretraining learns broad patterns
In a common autoregressive language-model setup, training predicts subsequent tokens from preceding context. A loss function measures prediction error.
Backpropagation computes gradients, and an optimizer updates model parameters. Repetition across many examples can produce useful language and task capabilities.
Post-training shapes responses
Supervised fine-tuning uses selected examples to adapt a pretrained model. Preference-based methods can further influence which responses it favours.
These stages can improve instruction-following, but they do not guarantee factual correctness or appropriate behaviour in every situation.
Do application teams need to train?
Not always. Start by evaluating an existing model with clear instructions. Consider adaptation when evidence shows a persistent task gap that better prompting or retrieval does not resolve.
RAG gives an application access to external information
Retrieval-augmented generation retrieves relevant material and supplies it with the question. This is useful when answers depend on private documentation or information that changes.
Prepare a retrieval index
Split documents into meaningful passages, attach source and permission metadata, create embeddings, and index them. Preserve headings so passages remain understandable.
Retrieve during the request
Authenticate the caller, restrict retrieval to permitted material, search for relevant passages, and supply selected evidence to the model. Return source references based on retrieved records.
Use compatible document and query embeddings. A model change generally requires re-embedding the indexed passages; matching dimensions alone does not guarantee compatibility.
More retrieved text is not always better. Irrelevant passages consume context and can distract from the evidence needed to answer.
What happens between a question and a response?
During inference, the application assembles instructions, the question, and any allowed context. An autoregressive model produces output tokens using its trained parameters.
The surrounding application must handle authentication, timeouts, validation, and failures. These responsibilities do not disappear because a model generates the response.
| Component | Responsibility |
|---|---|
| API layer | Validate requests and enforce identity. |
| Retrieval layer | Find relevant, permitted evidence. |
| Model integration | Manage calls, budgets, and response handling. |
| Tool layer | Execute approved operations with validated arguments. |
| Monitoring | Track quality, cost, latency, and errors. |
A Java implementation can use Spring Boot for the API and Spring AI for model and retrieval integration. Document ingestion can run separately from interactive requests.
Evaluation reveals where the system actually fails
Test representative questions with known expectations, including unanswerable requests and restricted documents. Evaluate retrieval separately from answer generation.
- Retrieval: Did the necessary evidence appear?
- Answer quality: Is the response correct and supported?
- Sources: Do references point to the right documents?
- Operations: What are latency and cost per completed task?
If a policy table was extracted incorrectly, a larger model may simply explain the wrong values more convincingly. Trace failures to their originating stage before changing the model.
Treat user feedback as a signal to investigate. Review and curate it before using it for training; not every correction is accurate or appropriate to retain.
Maintenance is part of intelligence in practice
Documents change, access rights expire, and model updates alter behaviour. Track versions of prompts, models, data, and retrieval settings so regressions can be investigated.
Re-index changed sources and remove obsolete passages. Re-run evaluation before deployment, and retain a rollback path. A model does not automatically discover every change in your business.
Your next practical step
Build a small documentation assistant with approved sources and a fixed question set. Inspect extraction, retrieval, and answers separately. Record failures before optimizing.
Comments
Post a Comment