How to Choose a RAG Development Company: A Buyer's Checklist

Every demo looks good

Choosing a RAG development company is harder than it looks, because every demo looks good. Ten clean PDFs, a handful of rehearsed questions and a fluent answer on screen prove very little about how a system will behave with 50,000 documents, strict access rules and a team that relies on it every day.

The differences only show up later: when the archive turns out to be half scanned, when two departments must not see each other's files, or when nobody can say whether last week's change made the answers better or worse.

I run a RAG development company, so I am not neutral. I have written this checklist to be useful whoever you hire. These are the questions I would ask if I were the buyer, and the answers that should reassure you.

Proof

1. Can they show RAG systems in production, with numbers?

Ask for systems that real users rely on, not prototypes, and ask what was measured and how. Good answers name the scale, the response time and the method used to measure quality.

For reference, these are the numbers I can point to on Datagreffe Justice: 50,000+ court decisions indexed, cited answers in under 2 seconds end to end, and a 15% retrieval improvement from fusing three search methods.

2. Who will actually do the work?

Many buyers meet senior people during the sales process and junior people during delivery. Ask who will architect the system, who will write the code and who you will speak to every week. Ask to meet them before you sign. A small senior team usually serves you better than a large team with a thin layer of experience on top.

Approach

3. How will they measure answer quality?

This is the most important question on the list. A credible team will propose a golden dataset of real questions with expected answers and sources, built with your domain experts and run automatically on every change.

Ask which metrics they track. Look for retrieval accuracy, faithfulness to sources, refusal behavior when the answer isn't in the documents, latency and cost. On Datagreffe Justice, 200 question-and-answer pairs validated with a lawyer gated every architecture change.

4. Why that retrieval design, for your data?

Be cautious of a vendor who proposes a complete architecture before seeing your documents. The right design depends on how your users search. Exact references and codes call for keyword search, paraphrased questions call for semantic search, and filters such as dates, regions or document types call for metadata search. Many serious systems need a hybrid of all three.

A good answer explains the trade-offs and commits to testing them on your data, including chunk sizes, rather than copying a tutorial default.

5. How will they handle your real documents?

Ask specifically about scanned pages, tables, multiple languages and duplicate versions. If a large share of your archive is scanned, the plan needs an OCR stage, and the vendor should explain how they will check its quality.

6. How do answers cite sources and handle uncertainty?

Every answer should cite the passage it came from, and the system should say so when the documents don't contain an answer. Ask to see how a citation is displayed, and what happens when retrieval confidence is low.

Security and compliance

7. Where does your data go?

Ask which model providers will receive your data, in which regions and under which terms. Ask whether the system can run in your own cloud, or on-premise with open-weight models if documents must not leave your environment.

8. How is access control enforced?

Permissions must be applied at retrieval time, down to the individual document. If they are only applied in the interface, a document a user is not allowed to see can still shape the answer the model writes. Ask how the vendor will test this.

9. How will they handle personal data?

If your documents contain personal data, ask whether it will be detected and removed before indexing, and how detection accuracy will be measured. Filtering personal data out of answers after the fact is not the same thing. On Datagreffe Justice, anonymization runs before indexing, and the engine reached 98% PII detection accuracy, with 100% data-protection and OHADA compliance.

Commercials and ownership

10. Who owns the code, prompts and evaluation data?

You should. Confirm in the contract that source code, prompts, evaluation datasets and documentation are delivered to your repositories and owned by you. Without the evaluation set in particular, nobody can safely change the system later, whoever maintains it.

11. How is the work priced and staged?

Look for a structure that limits your risk: a short discovery phase, a pilot with agreed success criteria, then a production build on fixed-price milestones. Open-ended billing with no milestones or defined deliverables makes it hard to know what you are paying for.

Ask about running costs too. Model usage, vector database and search hosting, and monitoring continue every month after launch. A good vendor will estimate them at your expected volume and explain how they will keep them down, for example with prompt caching, which cut input-token costs by about 90% on one of our production backends.

12. What happens after launch?

Ask what monitoring will be in place for answer quality, latency and token cost, what documentation and runbooks you will receive, and how support works. A system that only its builders understand is a risk to your business.

Red flags

• Accuracy promises made before anyone has looked at your documents
• No evaluation plan, or informal testing with your team as the only quality step
• Fine-tuning proposed as the default way to teach a model knowledge that changes
• Security and access control treated as a later phase
• No clear explanation of why a retrieval approach was chosen
• No handover plan, or a proprietary platform you can't leave
• Delivery staff you never get to meet

What a strong proposal includes

By the end of the selection process, a strong proposal should cover:

• The specific questions the system must answer, and for whom
• The proposed retrieval design, and how alternatives will be tested
• How quality will be measured, including who builds and owns the evaluation set
• Security, access control and personal-data handling
• Milestones, deliverables and acceptance criteria for each stage
• Estimated running costs at your expected volume
• What you receive at handover: code, prompts, evaluation data, documentation and runbooks

If a proposal leaves any of these out, ask for them before you compare prices. Proposals that look cheaper often leave out evaluation and security, which are exactly the parts that decide whether a system can go live.

Run a paid pilot, not a beauty contest

Slide decks and demos are poor predictors of delivery. A short paid pilot on your own data is a far better one. Choose one document collection and one user group, write 30 to 50 real questions with the answers you would accept, agree on success criteria upfront, and allow three to six weeks.

At the end, you will have measured results on your own documents and a clear sense of how the team works, communicates and handles problems. Both matter as much as the code.

How we answer these questions

If you are evaluating Retrieva Labs, here are our answers in brief:

• I personally lead the architecture of every engagement, and you work directly with senior engineers.
• Every build includes an evaluation suite built from real questions in your domain.
• Personal data is anonymized before indexing, access is enforced at query time, and we can deploy in your own cloud or on-premise.
• You own all the source code, prompts, evaluation datasets and documentation we create for you.
• Defined builds run on fixed-price milestones. Projects typically start from $5,000, and every engagement is scoped after a free strategy call.
• We sign an NDA before discovery whenever you need one.

Our RAG development services page describes how we build, and the Datagreffe Justice case study shows a production platform end to end. If you already have a RAG system that underperforms, our AI strategy and audit services cover evaluation and remediation. When you are ready, start a conversation.