Imagine you're building a project.

You're deep into the code when suddenly—you hit an error.

You stare at it for a few minutes, try a couple of things, and eventually give up.

So you do what most of us do now.

You copy-paste the code and the error into ChatGPT.

A few seconds later, you get an explanation of what went wrong, along with the corrected code.

You paste it into your project.

Boom. It works.

Problem solved.

You get that little dopamine hit. You feel productive. You move on to the next task.

And honestly, why wouldn't you?

It saved you time. You didn't have to spend an hour going through documentation or Stack Overflow. You got the answer almost instantly.

Now imagine doing this for every problem you encounter.

Every error.

Every unfamiliar concept.

Every piece of code you don't understand.

Every architectural decision you're unsure about.

And imagine doing it consistently for an entire year.

At the end of that year, you might have:

• Multiple projects you've built.

• Thousands of lines of code you've written.

• Experience with several frameworks and technologies.

• A GitHub profile that looks pretty impressive.

But there's a catch.

How much of it do you actually understand?

You may have built ten projects, but struggle to build the eleventh without AI.

You may have fixed hundreds of bugs, but don't know how to debug one from first principles.

You may have worked with technologies for a year, but still don't understand what is happening under the hood.

You have a lot of output, but surprisingly little experience.

And the uncomfortable part?

I am also a victim of this.

I've caught myself doing exactly this—getting stuck, asking AI for the solution, applying it, seeing the problem disappear, and moving on.

It feels like progress.

But sometimes, you're not actually learning.

You're just removing obstacles from your path before your brain gets the chance to figure them out.

And that's where I think we need to start asking a more important question:

Are we using AI to become better developers, or are we using AI to avoid becoming better developers?

The Solution: Don't Stop Using AI. Start Using It Differently.

So, what's the solution?

Stop using AI?

Absolutely not.

In a world where we want to build faster, ship faster, and solve problems faster, completely avoiding AI doesn't make much sense.

The real question is:

How do we use AI to become better developers while still using it to move faster?

I think it starts with understanding what kind of problem we're asking AI to solve.

1. Problems I've solved before

These are problems where I already understand the concept and don't have a knowledge gap.

For example, I've created Spring Boot entity classes so many times that I don't need to manually write them anymore. I can give AI my table structure and let it generate the entity class.

That's a good use of AI.

I'm not outsourcing my understanding. I'm outsourcing repetitive work.

The important part is that I can look at the generated code and understand what it does.

If you know the problem, let AI save you time.

2. Problems I understand, but have a knowledge gap

This is where AI can become a really good teacher.

When I was building a RAG system, I understood how RAG worked. But my documents contained tables and diagrams, so traditional text-based retrieval wasn't enough.

I needed to understand concepts like multimodal retrieval, multi-vector embeddings, and models like ColPali and ColQwen.

I could have asked ChatGPT for the solution and blindly implemented whatever it suggested.

The problem would probably have been solved.

But I wouldn't have learned much.

Instead, I first used AI to understand the concepts I was missing. Once I understood the different approaches and their trade-offs, I could make an informed decision about what I actually needed.

Identify the knowledge gap. Fill it. Make the decision. Then use AI to implement that decision.

That's very different from asking AI to make the decision for you.

Don't use AI to jump over your knowledge gaps. Use it to close them.

3. Problems I know nothing about

This is where blindly asking AI for code can be dangerous.

If you don't understand the problem, how will you know whether the solution AI gave you is actually a good one?

Start with the concepts.

Understand what you're trying to solve, what the important components are, what approaches exist, and how they fit together.

Build a mental model first.

AI can help tremendously with this. Ask questions, challenge the explanations, explore examples, and gradually build your understanding.

Once you have enough understanding, make the architectural or technical decision yourself.

Then ask AI to help you implement it.

The distinction is subtle but extremely important:

Don't ask AI, "What should I build?" when you don't understand the problem.

Ask AI, "I've decided to build X because of Y. Help me implement it."

In the first case, AI is making the decision for you.

In the second, you're making the decision and AI is helping you execute it.

And there's one rule that applies to all three categories:

You should understand the code you're shipping.

You don't have to type every line yourself.

You don't have to memorize every API.

But when AI gives you a piece of code, you should have a reasonable understanding of what it does, why it's there, and what could go wrong.

The ideal workflow, in my opinion, is:

Understand → Identify the knowledge gap → Learn → Make the decision → Ask AI to implement → Review and validate.

Because the real test isn't when the code works.

The real test is when it breaks.

That's when you'll find out whether you actually learned something—or whether you simply copied a solution.

Let AI write more code for you. Just don't let it make all the decisions for you.

Why Does the Solution Work?

For those who are into strength training, you might be familiar with the concept of "time under tension."

The basic idea is that the longer your muscles are under meaningful tension, the stronger the stimulus for them to adapt and grow.

I think something similar happens with our brain.

The more time we spend thinking about a problem before reaching for the answer, the more our brain has to work through it.

We try different approaches. We break the problem down. We connect it to concepts we already know. We question our assumptions. We try to build a mental model of what's actually happening.

And even if we don't solve the problem completely, we've already done something valuable.

We've made our brain work.

When we immediately ask AI for the solution, we skip a lot of that process. We get the answer, implement it, and move on.

But when we spend some time struggling with the problem first and then use AI to help us, something different happens.

The explanation AI gives us now has something to attach to.

We can connect the new concept to something we already understand.

And this is where the compounding effect starts.

Every problem we genuinely think through adds another piece to our mental model. Those pieces start connecting with each other, making it easier to understand and solve the next problem.

The more you learn, the more you have to build upon.

So maybe we shouldn't try to eliminate the struggle from software development.

We should use AI to make that struggle more productive.

Let your brain spend some time under tension.

Then let AI help you go further, faster.

AI should reduce the time it takes to solve a problem—not the thinking required to learn from it.

And Yes, I Used AI to Write This Article

There is a little irony in writing an article about using AI correctly—with AI.

But that's exactly the point.

I didn't open ChatGPT and say:

"Write me an article about how AI can make or break a developer's career."

Instead, I had the idea first.

I decided what I wanted to say.

I thought about the examples.

I structured the argument.

I wrote down my experiences and the points I wanted to convey.

And then I used AI to do what it's really good at:

refine, challenge, restructure, and enhance what I had already thought through.

In fact, this entire article is an example of the approach I've been talking about.

I'll leave the link to this conversation here: Chat Link

You can see the rough ideas I started with and how AI helped turn them into something more readable and structured.

The important thing is that the thinking happened before the prompting.

I didn't outsource the idea.

I outsourced the polishing.

And I think that's a pretty good way to use AI—not just for writing, but for software development too.

Think first. Build the mental model. Make the decision. Then let AI make you faster.

Because the developers who benefit the most from AI won't necessarily be the ones who ask AI the most questions.

They'll be the ones who know which questions to ask—and why they're asking them.

Evaluation is not just about checking whether a system gives the correct answer. It helps us identify where the system is failing, especially when the system consists of multiple components working together.

If the final answer is incorrect, the evaluation should help us answer: Did the retrieval fail? Was the SQL generated incorrectly? Did execution fail? Was the result misunderstood? Or was the final response generated incorrectly?

Context: SQL AI Assistant

I built a LangGraph-powered SQL AI Assistant that allows non-technical users to interact with a PostgreSQL database using natural language.

For example:

"Show the number of conversations created on each day from July 28 to August 1, 2026."

Instead of writing SQL, the user simply asks the question, and the assistant handles the rest.

SQL AI Assistant Workflow:

For this question, the assistant might generate:

SELECT created_date::date AS conversation_date, 
       COUNT(conversation_id) AS conversations_count
FROM LIVE_CHAT_DETAILS
WHERE created_date::date BETWEEN '2026-07-28' AND '2026-08-01'
GROUP BY created_date::date
ORDER BY conversation_date;

and return:

"Between July 28 and August 1, 2026, there were 10 conversations created each day."

At first glance, this looks simple. But there are multiple places where the system can fail—retrieval, SQL generation, validation, execution, result interpretation, or final answer generation.

Before Writing a Single Line of Code: Defining What to Evaluate

Before creating the golden dataset or writing any evaluation code, I first identified the different components of the SQL AI Assistant that could independently fail. The goal was to make the evaluation diagnostic rather than just produce a single accuracy number.

Retrieval

I started with the context retrieval layer:

• Precision — Are the tables retrieved actually relevant to the user's question?

• Recall — Did the retriever find all the tables required to answer the question?

SQL Generation & Execution

For the SQL pipeline, I wanted to separate logical correctness from execution:

• SQL Query Correctness — Does the generated SQL correctly represent the user's intent?

• SQL Result Correctness — Does the executed query produce the expected result?

• Execution Success — Did the generated SQL execute successfully?

Final Answer

Finally, I evaluated what the user actually sees:

• Answer Correctness — Does the final response correctly answer the user's question and convey the expected information?

The next decision was how to evaluate these metrics. Some can be measured deterministically, such as whether a query executed successfully or whether the expected tables were retrieved. For semantic evaluations like SQL correctness and answer correctness, I decided to use an LLM-as-a-Judge approach.

This gave me a clear evaluation framework.

With the evaluation criteria defined upfront, I could now move on to building the golden dataset around these metrics.

Building the Golden Dataset

Once I had defined what to evaluate, the next step was creating a Golden Dataset.

I didn't want the dataset to contain similar questions phrased in different ways. Instead, I divided the test cases into different categories based on the type of reasoning or SQL operation required.

The initial dataset covered seven categories:

Each test case also contains the information needed to evaluate the complete pipeline:

{
    "id": "TC-026",
    "question": "How many terminated conversations were created on July 28, 2026?",
    "category": "multi_condition",
    "difficulty": "medium",
    "expected_tables": [
      "LIVE_CHAT_DETAILS"
    ],
    "expected_behavior": "execute",
    "expected_sql": "SELECT COUNT(*) FROM LIVE_CHAT_DETAILS WHERE status = 'Terminate' AND created_date >= '2026-07-28 00:00:00' AND created_date < '2026-07-29 00:00:00';",
    "expected_result": {
      "type": "scalar",
      "value": 2
    },
    "expected_answer": "There were 2 terminated conversations created on July 28, 2026."
  }

For the first version, I decided that 30 test cases were enough to establish an initial baseline. The objective wasn't to build a huge dataset immediately, but to create a diverse set of questions that could expose different failure modes. Based on the evaluation results, the dataset can then be expanded around the areas where the assistant performs poorly.

The Evaluation Report

I won't make this article lengthy by walking through every line of the evaluation code. If you want to explore the implementation, you can find the complete evaluation pipeline here:

GitHub Repository: GitHub repository link

Instead, I want to show what the evaluation actually produces.

For every test case, the final report captures the complete evaluation trace across retrieval, SQL generation, execution, and final answer generation:

test_case_id
question

retrieved_tables
expected_tables
precision
recall

generated_query
expected_query
query_correct
query_failure_category
query_reasoning

actual_query_result
expected_result
result_success
result_failure_category
result_reasoning

generated_final_answer
expected_final_answer
is_final_answer_correct
final_answer_score
final_answer_failure_category
final_answer_reasoning

This gives me more than a simple accuracy score. For every failed test case, I can see what failed, where it failed, and why the evaluator considered it a failure.

Evaluation Results

I also maintain the complete test-case-level results for easier analysis and tracking:

Google Sheet: Google Sheet link

This report becomes the foundation for the next step: using the evaluation results to identify the biggest gaps in the SQL AI Assistant.

What I Learned From the Initial Evaluation

The first evaluation run gave me a few useful insights into where the system is working and where it needs improvement.

Read-Only SQL Validation Is Working

The assistant successfully rejects queries that are not read-only, such as DELETE, UPDATE, or other destructive operations.

This confirms that the SQL validation layer is working as intended and is preventing unsafe queries from reaching the database.

Retrieval Is One of the Current Bottlenecks

I identified a retrieval gap around two tables with very similar business context:

LIVE_CHAT_DETAILS

LIVE_CHAT_CONV_AUDIT

Because both contain conversation-related information, the retriever can sometimes select the wrong table or fail to distinguish which one is required for a particular question.

This then affects the next stage:

Ambiguous Context
      ↓
Incorrect Table Retrieval
      ↓
Incorrect SQL

Final Answer Generation Has Not Been the Bottleneck

One interesting observation from the evaluation was that when the SQL is correct and produces the expected result, the final answer is consistently correct.

This suggests that, for the current dataset, improving the final-answer generation isn't the priority. The bigger opportunity is earlier in the pipeline:

Improve retrieval → improve SQL generation → improve overall correctness.

What's Next?

Based on these observations, my next steps are:

1. Improve the metadata/context descriptions for LIVE_CHAT_DETAILS and LIVE_CHAT_CONV_AUDIT so their differences are clearer to the retriever.

2. Expand the Golden Dataset with more questions specifically designed to distinguish between these two tables.

3. Run the evaluation again and check whether retrieval precision/recall and SQL correctness improve.

This is exactly where evaluation becomes useful: instead of randomly changing prompts or agents, I now have evidence pointing to the specific component and failure pattern I need to improve.

I was building a SQL AI Assistant for the live chat flow of a customer chatbot.

The goal was simple: instead of manually querying the database, business users could ask questions about the conversations and agents in natural language.

For example:

• How many conversations happened on a particular date?

• How many agents were active during a particular time period?

Behind the scenes, the assistant had to understand the user's question, identify the relevant database information, generate the SQL query, validate it, execute it, and return the result.

The first version of the assistant was working, but there was one major problem:

It was taking around 40 seconds to answer a question.

For an interactive assistant, 40 seconds is simply too slow.

At first, I thought the LLM itself was the bottleneck. But after looking at the execution flow and measuring the individual steps, I realized that the problem was not just the model.

The architecture was making the model do too much work.

That led me to rethink how my SQL agent was retrieving database context and interacting with tools.

In this article, I'll walk through the original architecture, where the latency was coming from, what I changed, and how the same type of queries went from roughly 40 seconds to around 8 seconds.

What I Was Building

The goal of the assistant was to let users query the live-chat data using natural language, without needing to know the underlying database schema or write SQL.

For example, a user could ask:

how many chat were ended by the customer on 1st Aug 2026?

The assistant would first understand the question and generate the corresponding SQL:

SELECT COUNT(*) AS chats_ended_by_customer
FROM LIVE_CHAT_CONV_AUDIT\nWHERE status = 'Terminate' AND created_date = '2026-08-01'

The database returns:

4

And instead of exposing the SQL result directly, the assistant converts it into a natural-language response:

On 1st August 2026, 4 chats were ended by the customer.

Version 1 — My initial architecture and the problems

My first approach was to make the SQL assistant as autonomous as possible.

The generate_agent was responsible for generating SQL, but before doing that, it could call multiple database tools:

get_tables()
get_schema()
get_distinct_values()

The critic_agent also had access to similar tools so that it could independently inspect the database while validating the generated SQL.

This gave me a graph that looked like this:

At first, this seemed like a good idea.

The agent could:

Figure out what it needs → call the appropriate tools → generate SQL → critique it → fix it → execute it.

The agent was doing what it didn't need to do

Schema retrieval was becoming part of the agent's reasoning loop.

For a simple question, the flow could become:

User Question
      ↓
generate_agent
      ↓
get_tables()
      ↓
generate_agent
      ↓
get_schema()
      ↓
generate_agent
      ↓
get_distinct_values()
      ↓
generate_agent
      ↓
Generate SQL
      ↓
critic_agent
      ↓
More tool calls...

Every tool call meant more latency, and every additional LLM/tool interaction added to the overall cost.

What started as a simple task:

Understand the question and generate SQL.

had turned into:

Let an autonomous agent discover the database, retrieve its schema, reason about it, generate SQL, critique itself, retrieve more information, and then execute the query.

I had over-engineered a relatively simple problem.

The goal wasn't to make the agent autonomous at every step.

The goal was to make it efficient at the steps that actually require reasoning.

And that realization led me to completely rethink the architecture.

The New Approach — Give Every Node One Job

The biggest change wasn't a better prompt or a different model.

I changed the architecture.

Instead of making the agents autonomous and allowing them to call tools whenever they wanted, I separated the workflow into clearly defined steps.

Each node now has a specific responsibility

1. Context Retriever

The retriever identifies the relevant database context required for the user's question.

Instead of asking the LLM to repeatedly discover the database using tools like get_tables() and get_schema(), the context is prepared before SQL generation.

2. SQL Generator

This is where the LLM actually generates the SQL.

It receives:

User Question + Relevant Database Context

and produces the SQL query.

The LLM doesn't need to figure out how to retrieve the schema anymore.

3. SQL Validator

This handles deterministic checks such as:

• Is the SQL valid?

• Is it read-only?

• Does it reference valid tables/columns?

• Does it follow the application's SQL safety rules?

There is no need to use an LLM for these checks.

4. SQL Executor

Once the query passes validation, it is executed against the database.

Validated SQL
      ↓
Database
      ↓
Query Result

Again, this is deterministic work.

5. SQL Critic

The critic is triggered only when SQL execution fails.

It receives the database error along with the question and generated SQL, reasons about the problem, and generates a corrected SQL query.

SQL Executor
     ↓
Database Error
     ↓
SQL Critic
     ↓
Corrected SQL
     ↓
SQL Validator

This keeps the LLM out of the normal execution path and uses it only when reasoning is actually needed.

6. Final Response

Once the SQL is considered correct and the result is available, the assistant converts the result into a concise natural-language response for the user.

For example:

On 1st August 2026, 4 chats were ended by the customer.

The key idea

The redesign wasn't about removing LLMs.

It was about using LLMs only where reasoning is required.

Instead of:

❌ LLM → Tool → LLM → Tool → LLM → Tool

the workflow became:

✅ Retrieve → Generate → Validate → Execute → Critique → Respond

Every node has a clear responsibility, which made the system faster, easier to debug, and easier to scale.

Why Does the New Architecture Scale Better?

The first version worked with my current database, but I started thinking about what would happen as the number of tables increased.

Today:

7 tables

Tomorrow:

100 tables

With the old approach, the agent could repeatedly discover and retrieve schema information through tools.

That means more:

• tool calls

• LLM reasoning

• tokens

• latency

As the database grows, the amount of information the LLM has to deal with grows as well.

The new architecture separates database context retrieval from SQL generation.

Instead of giving the LLM the entire database:

100 Tables
   ↓
Context
   ↓
LLM

the system can retrieve only what is relevant:

100 Tables
    ↓
Context Retriever
    ↓
2-3 Relevant Tables
    ↓
SQL Generator

This means the SQL generator doesn't care whether the database has 7 tables or 100 tables. It receives the context it needs for the current question.

And because the context retriever is a separate component, I can evolve it independently — from a simple metadata-based LLM retriever today to a semantic or hybrid retriever later.

The goal wasn't just to make the current system faster. It was to make sure adding more tables didn't automatically mean adding more context, more tokens, and more agent reasoning.

What I Learned

This project changed how I think about building AI systems.

1. Don't use an LLM for deterministic tasks

An LLM is powerful because it can reason, but that doesn't mean it should handle every task.

If something can be done reliably with code, I'll prefer code.

Before:

"Let the agent figure out which tools it needs and retrieve the schema."

Now I think:

"Can I solve this deterministically? If yes, why make the LLM do it?"

This is something I'll carry into the next AI system I build.

2. Design for the system you will have, not just the system you have today

My database had only a few tables when I started.

It would have been easy to say:

"The current schema is small, so I'll just send everything to the LLM."

But what happens when it becomes 20, 50, or 100 tables?

Scaling shouldn't be an afterthought.

I learned to ask early:

• What happens when the data grows?

• What happens when the number of users grows?

• What happens when the context grows?

• Which part of my architecture will become the bottleneck?

A solution that works today isn't necessarily a solution that scales tomorrow.

Code & Implementation

I've committed the implementation of this version of the SQL AI Assistant to GitHub.

If you want to see how the architecture translates into code, you can explore the repository here:

GitHub Repository

The repository includes the LangGraph workflow, context retrieval, SQL generation, validation, execution, and error-handling flow discussed in this article.

What's Next?

This is only the first iteration of the SQL AI Assistant. My next focus areas are:

1. Stronger Guardrails

Adding stronger controls for:

• Read-only access

• Unsafe SQL prevention

• Query complexity limits

• Table/column access control

2. Proper Evaluation

Build an evaluation dataset with:

User Question → Expected SQL → Expected Result

Measure SQL correctness, answer accuracy, execution success, and latency.

3. Better Context Retrieval

As the database grows, explore:

Metadata Retrieval
       ↓
Semantic Retrieval
       ↓
Hybrid Retrieval

to improve table and column selection.

The goal is to move from a working SQL AI Assistant to one that is reliable, measurable, and scalable.

Over the past few weeks, I built a multimodal Retrieval-Augmented Generation (RAG) system for a due diligence application that analyzes company documents such as annual reports, financial statements, and investor presentations.

Before adding visual understanding, I had already built a fairly mature text retrieval pipeline. The pipeline extracted structured text directly from PDFs, performed hierarchical chunking to preserve document context, indexed the chunks in Qdrant, and combined semantic vector search with BM25 using hybrid retrieval. For text-heavy questions, the system consistently returned relevant context and produced reliable answers.

The limitations only became apparent when the documents relied on visual information.

Annual reports and investor presentations are filled with financial tables, bar charts, pie charts, organizational diagrams, process flows, and complex page layouts. While these pages often contain text, simply retrieving the extracted text isn't always enough to understand the information being presented. The meaning frequently depends on the visual structure of the page.

This article isn't another "How to Build a Multimodal RAG" tutorial. Instead, it's a walkthrough of the engineering decisions, implementation challenges, architectural trade-offs, and lessons I learned while extending an already capable text RAG system into a multimodal one.

When My Text RAG Reached Its Limits

My text RAG pipeline was already performing well. With hierarchical chunking and hybrid retrieval, it could accurately answer questions such as "Who is the CEO?" or "What are the company's plans for the next quarter?" because the required information existed as text in the document.

The problem started when the answers depended on charts, graphs, or tables.

Questions like "What was the Q2 traction?" or "What is the ARR projection till 2030?" weren't answered reliably, even though the information existed in the PDF.

Initially, I thought the retrieval pipeline was failing. After inspecting the retrieved chunks, I realized the opposite—the hybrid search was retrieving the correct content.

The real issue was the document representation. The parser extracted the labels and numbers from charts, but the visual relationships between them were lost during text extraction and chunking. As a result, the LLM had the right data but not the context needed to interpret it correctly, leading to inaccurate answers.

That was when I realized that improving retrieval wasn't enough—I needed a way to preserve and retrieve the visual understanding of the document itself.

Figure 2a - Page from the PDF

Figure 2b - Wrong answer from text-only RAG.

Finding the Right Approach

Once I realized that my text pipeline wasn't enough, I started exploring the world of Multimodal RAG. That's when I encountered a flood of new concepts—captioning models, CLIP, unified embedding spaces, multivector retrieval, ColPali, and ColQwen2. Every article seemed to recommend a different approach, making the decision far from straightforward.

Option 1: Captioning Every Page

The first idea was to generate a caption for every PDF page and index those captions alongside the document text.

Although simple, I quickly ruled it out. Captioning models summarize what an image is about, but they don't preserve the detailed information inside it. A caption like "A bar chart showing revenue growth" is unlikely to help retrieve the page for a query like "What is the ARR projection for 2028?" The quality of retrieval would depend entirely on how descriptive the generated caption was.

Option 2: CLIP

Next, I looked at CLIP, which embeds both text and images into a shared vector space. It seemed promising because it allowed text queries to retrieve images directly.

However, my use case involved dense document pages filled with charts, tables, and small text—not natural images. More importantly, CLIP represents an entire page using a single embedding vector, which felt insufficient for capturing all the information present in a complex document page.

That was when I came across the idea of multivector embeddings. Instead of representing a page with one vector, it could be represented by hundreds of vectors, preserving much richer information. This eventually led me to ColPali and ColQwen2, which became the foundation of my visual retrieval pipeline.

Building the Visual Pipeline

Once I understood why multivector embeddings were a better fit for document retrieval, I designed a separate visual pipeline instead of modifying my existing text pipeline.

The pipeline starts by rendering every page of the PDF as an image. Each page image is then passed to ColQwen2, which generates a multivector embedding representing the visual content of that page. These embeddings are stored in a dedicated Qdrant collection, completely independent of the text embeddings.

Keeping the text and visual pipelines separate turned out to be an important architectural decision. The text pipeline continues to excel at retrieving textual information using hierarchical chunking and hybrid search, while the visual pipeline specializes in retrieving pages whose meaning depends on charts, tables, diagrams, and other visual elements.

At query time, both pipelines work together—the text pipeline retrieves relevant text chunks, while the visual pipeline retrieves the most relevant document pages. These results are then passed to a multimodal LLM, allowing it to reason over both textual and visual context before generating the final answer.

Running the Visual Pipeline

Designing the visual pipeline was only half the challenge. The next hurdle was running ColQwen2, a relatively large model that wasn't practical to run on my local machine. Instead of forcing everything into a single application, I decided to separate model inference from the main application.

The pipeline begins by using PyMuPDF (fitz) to render every page of the PDF into an image. These page images are then sent to a FastAPI server hosting ColQwen2, which generates multivector embeddings for each page.

Since I didn't have access to a dedicated GPU locally, I deployed the FastAPI server on Google Colab and exposed it to my local application using ngrok. This allowed my application to generate embeddings through simple HTTP requests, while all the heavy computation happened remotely.

The generated multivector embeddings are then stored in Qdrant, where they can later be retrieved using query embeddings generated by the same model.

Separating the embedding service from the main application turned out to be one of the best architectural decisions of the project. It kept the application lightweight, made development possible without local GPU resources, and allowed me to replace or upgrade the embedding model in the future with minimal changes to the rest of the system.

Tech Stack

• PyMuPDF (fitz) – Render PDF pages as images

• ColQwen2 – Generate multivector embeddings

• FastAPI – Expose the embedding model as an API

• Google Colab – Free GPU for model inference

• ngrok – Connect the local application to the Colab API

• Qdrant – Store and retrieve multivector embeddings

• OpenAI GPT-4o – Reason over retrieved text and page image

Retrieving Both Text and Visual Context

Once the ingestion pipeline was complete, the retrieval process became surprisingly simple.

Whenever a user asks a question, the query is sent to both the text pipeline and the visual pipeline simultaneously.

The text pipeline performs hybrid retrieval over hierarchically chunked content and returns the most relevant text passages. In parallel, the visual pipeline generates a query embedding using ColQwen2 and performs multivector search in Qdrant to retrieve the most relevant document pages.

These two sources of information are then combined and sent to GPT-4o.

This design turned out to be particularly effective because each pipeline specializes in what it understands best. The text pipeline excels at retrieving semantic information from paragraphs, while the visual pipeline retrieves pages whose meaning depends on charts, tables, diagrams, and layout.

An important design decision was that the visual pipeline does not return extracted text—it returns the original page image. This allows GPT-4o to reason directly over the visual content instead of relying on potentially incomplete text extraction.

For example, a query such as:

"What is the ARR projection till 2030?"

retrieves:

• Text chunks describing financial projections.

• The original page containing the ARR chart.

GPT-4o then combines both textual and visual context to generate the final answer.

Final Answer

Complete source Code

If you're interested in building a similar Multimodal RAG system or experimenting with multivector retrieval, I've open-sourced the complete implementation used in this article.

📂 GitHub Repository

The complete source code, including both the text and visual pipelines, is available here:

🔗 GitHub: <GitHub Repository Link>

📓 Google Colab Notebook

Since ColQwen2 requires a GPU, I also created a Google Colab notebook that hosts the model using FastAPI and exposes it through ngrok. You can use it directly or adapt it to your own setup.

🔗 Google Colab: <Colab Link>

Final Thoughts

Building this project completely changed how I think about RAG systems.

Initially, I believed that retrieval quality was the bottleneck. In reality, the bigger challenge was how the document was represented. A retriever can only search over the information it has been given, and if visual context is lost during preprocessing, no amount of retrieval optimization can recover it.

This project also reinforced another lesson: there is rarely a single "correct" architecture. Exploring captioning models, CLIP, and multivector retrieval helped me understand the trade-offs of each approach before arriving at a solution that fit my use case.

The current system already combines hybrid text retrieval with visual page retrieval to answer questions grounded in both text and images. There is still plenty of room for improvement—better reranking, golden dataset evaluation metrics, and optimized indexing are all on my roadmap—but this implementation provides a solid foundation for building document-aware Multimodal RAG systems.

If you found this article useful, I'd love to hear your thoughts or discuss different approaches to Multimodal RAG. Feel free to connect with me or open an issue on GitHub.