Grape benchmark

Suites / Gujarati documents

Gujarati documents

Gujarati questions on 12 Gujarati Wikipedia articles (Ahmedabad, Gir, Narmada, ...). Vector RAG uses multilingual-e5-large.

3 runs / 12 files, 0.5 MB / model haiku, thinking off / Vector RAG embedder intfloat/multilingual-e5-large. Raw results: 2026-09-30 16:45 2026-09-30 16:50 2026-09-30 16:54

Grape

19.7 (19 to 20) / 23

Off-topic refused
5 / 5
Input tokens per question
1,684 (1,675 to 1,693)
LLM calls per question
1.25
Cost per question
$0.00221 ($0.00216 to $0.00223)
Time per question
5.1 s (5.0 s to 5.2 s)
Index time
29 ms
Vector RAG

22 / 23

Off-topic refused
5 / 5
Input tokens per question
4,137
LLM calls per question
1.00
Cost per question
$0.00404 ($0.00273 to $0.00664)
Time per question
4.4 s (4.4 s to 4.4 s)
Embedding time
5.1 min
Questions

right every run some runs never. Left half Grape, right half Vector RAG.

  1. 01
  2. 02
  3. 03
  4. 04
  5. 05
  6. 06
  7. 07
  8. 08
  9. 09
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28

j / k or arrow keys