Suites / Gujarati questions, English documents

Gujarati questions, English documents

The same 12 English articles, every question asked in Gujarati. Lexical search cannot match across languages, so Grape's first LLM call rewrites the question in the documents' language. Vector RAG uses a large multilingual embedding model (multilingual-e5-large), which embeds both languages in one space.

3 runs / 12 files, 0.6 MB / model haiku, thinking off / Vector RAG embedder intfloat/multilingual-e5-large. Raw results: 2026-09-30 16:04 2026-09-30 16:10 2026-09-30 16:16

Grape

21.7 (21 to 23) / 23

Off-topic refused: 5 / 5
Input tokens per question: 1,604 (1,566 to 1,660)
LLM calls per question: 1.86 (1.82 to 1.89)
Cost per question: $0.00234 ($0.00230 to $0.00239)
Time per question: 7.4 s (7.3 s to 7.5 s)
Index time: 47 ms

Vector RAG

21 / 23

Off-topic refused: 5 / 5
Input tokens per question: 1,375
LLM calls per question: 1.00
Cost per question: $0.00199 ($0.00198 to $0.00200)
Time per question: 5.1 s (5.0 s to 5.3 s)
Embedding time: 12.5 min

Questions

right every run some runs never. Left half Grape, right half Vector RAG.