{
  "corpus": "large",
  "provider": "claude",
  "model": "haiku",
  "cost_basis": "Claude API list price reported by the claude CLI (haiku, thinking off), grape server rerank + gate, 5 passages",
  "embed": "BAAI/bge-small-en-v1.5",
  "meta": {
    "date": "2026-09-30 16:57",
    "questions": "questions_large.json",
    "embed_key": "bge-small",
    "modes": [
      "grape",
      "rag"
    ],
    "corpus_files": 300,
    "corpus_bytes": 18379509,
    "chunks": 22672,
    "settings": {
      "limit": 8,
      "snippet": 600,
      "top_k": 5,
      "chunk_chars": 1000
    },
    "thinking": false,
    "rerank": null,
    "gate": false,
    "server_rerank": true,
    "server_limit": 5,
    "judge": true,
    "grape_version": "0.1.0",
    "bench_commit": null
  },
  "setup_seconds": {
    "grape": 1.1422215059992595,
    "grape_update": 1.0494582280007307,
    "rag": 2778.2464952149894
  },
  "rows": [
    {
      "q": "Which river is the German town of Aalen situated on?",
      "type": "fact",
      "expect": [
        "Kocher"
      ],
      "grape": {
        "answer": "The German town of Aalen is situated on the river Kocher, located at the upper reaches of this river at the foot of the Swabian Jura.",
        "in": 1422,
        "out": 40,
        "cost": 0.0016220000000000002,
        "calls": 1,
        "route": "direct",
        "coverage": 0.76,
        "seconds": 3.82,
        "api_seconds": 0.96,
        "correct": true
      },
      "rag": {
        "answer": "Based on the excerpts provided, Aalen is situated on the **river Kocher**. The text states: \"Aalen is situated on the upper reaches of the river Kocher, at the foot of the Swabian Jura.\"",
        "in": 1726,
        "out": 58,
        "cost": 0.002016,
        "calls": 1,
        "seconds": 2.96,
        "api_seconds": 1.15,
        "correct": true
      }
    },
    {
      "q": "What was the Acorn Electron nicknamed inside Acorn?",
      "type": "fact",
      "expect": [
        "Elk"
      ],
      "grape": {
        "answer": "The Acorn Electron was nicknamed the \"Elk\" inside Acorn. This nickname was announced publicly for perhaps the first time at the machine's launch at the Acorn User Exhibition in August 1983.",
        "in": 1464,
        "out": 49,
        "cost": 0.001709,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.17,
        "api_seconds": 1.03,
        "correct": true
      },
      "rag": {
        "answer": "The Acorn Electron was nicknamed the \"Elk\" inside Acorn and beyond.",
        "in": 1327,
        "out": 23,
        "cost": 0.0014420000000000001,
        "calls": 1,
        "seconds": 2.82,
        "api_seconds": 1.01,
        "correct": true
      }
    },
    {
      "q": "What was the launch price of the Acorn Electron?",
      "type": "number",
      "expect": [
        "199"
      ],
      "grape": {
        "answer": "The launch price of the Acorn Electron was £199, as it was introduced on 25 August 1983 at that price.",
        "in": 1513,
        "out": 34,
        "cost": 0.001683,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.19,
        "api_seconds": 0.99,
        "correct": true
      },
      "rag": {
        "answer": "The launch price of the Acorn Electron was £199, which was announced when the machine was introduced on 25 August 1983.",
        "in": 1346,
        "out": 34,
        "cost": 0.001516,
        "calls": 1,
        "seconds": 2.68,
        "api_seconds": 0.92,
        "correct": true
      }
    },
    {
      "q": "How many times did the Apollo 8 crew orbit the Moon?",
      "type": "number",
      "expect": [
        [
          "ten",
          "10"
        ]
      ],
      "grape": {
        "answer": "According to the excerpts, the Apollo 8 crew orbited the Moon **ten times** during their mission. The crew orbited the Moon ten times over the course of twenty hours before departing safely back to Earth.",
        "in": 1472,
        "out": 49,
        "cost": 0.001717,
        "calls": 1,
        "route": "direct",
        "coverage": 0.94,
        "seconds": 3.38,
        "api_seconds": 1.03,
        "correct": true
      },
      "rag": {
        "answer": "The Apollo 8 crew orbited the Moon **ten times**. According to the excerpts, \"The crew orbited the Moon ten times without landing\" and they \"made ten lunar orbits in 20 hours\" during their mission.",
        "in": 1363,
        "out": 54,
        "cost": 0.0016330000000000001,
        "calls": 1,
        "seconds": 2.92,
        "api_seconds": 1.11,
        "correct": true
      }
    },
    {
      "q": "Who were the three astronauts of Apollo 8?",
      "type": "fact",
      "expect": [
        "Borman",
        "Lovell",
        "Anders"
      ],
      "grape": {
        "answer": "The three astronauts of Apollo 8 were Frank Borman, James Lovell, and William Anders, who were the first humans to witness and photograph the far side of the Moon and an Earthrise.",
        "in": 1294,
        "out": 49,
        "cost": 0.001539,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.29,
        "api_seconds": 1.09,
        "correct": true
      },
      "rag": {
        "answer": "The three astronauts of Apollo 8 were Frank Borman, James Lovell, and William Anders, who were the first humans to witness and photograph the far side of the Moon and an Earthrise.",
        "in": 1363,
        "out": 49,
        "cost": 0.001608,
        "calls": 1,
        "seconds": 2.82,
        "api_seconds": 1.0,
        "correct": true
      }
    },
    {
      "q": "In what year did the Boeing 767 prototype first fly?",
      "type": "number",
      "expect": [
        "1981"
      ],
      "grape": {
        "answer": "According to the excerpts, the Boeing 767 prototype first flew in **1981**—specifically on September 26, 1981.",
        "in": 1439,
        "out": 33,
        "cost": 0.001604,
        "calls": 1,
        "route": "direct",
        "coverage": 0.81,
        "seconds": 3.17,
        "api_seconds": 0.87,
        "correct": true
      },
      "rag": {
        "answer": "The Boeing 767 prototype first flew in **1981**. According to the excerpts, \"the prototype first flew on September 26, 1981.\"",
        "in": 1508,
        "out": 37,
        "cost": 0.001693,
        "calls": 1,
        "seconds": 3.0,
        "api_seconds": 1.11,
        "correct": true
      }
    },
    {
      "q": "Which Boeing airliner was the company's first wide-body jet with only two engines?",
      "type": "paraphrase",
      "expect": [
        "767"
      ],
      "grape": {
        "answer": "Based on the excerpts, the **Boeing 767** was the company's first wide-body jet with only two engines. The excerpt states: \"The 767 is Boeing's first wide-body twinjet, powered by General Electric CF6, Rolls-Royce RB211, or Pratt & Whitney JT9D turbofans.\"",
        "in": 1295,
        "out": 80,
        "cost": 0.001695,
        "calls": 1,
        "route": "direct",
        "coverage": 0.86,
        "seconds": 3.64,
        "api_seconds": 1.25,
        "correct": true
      },
      "rag": {
        "answer": "Based on the excerpts provided, the **Boeing 767** was Boeing's first wide-body jet with only two engines. The document explicitly states: \"The 767 is Boeing's first wide-body twinjet, powered by General Electric CF6, Rolls-Royce RB211, or Pratt & Whitney JT9D turbofans.\"",
        "in": 1658,
        "out": 81,
        "cost": 0.002063,
        "calls": 1,
        "seconds": 3.35,
        "api_seconds": 1.39,
        "correct": true
      }
    },
    {
      "q": "What is the official estimate of the Ainu population of Japan?",
      "type": "number",
      "expect": [
        [
          "25,000",
          "25000"
        ]
      ],
      "grape": {
        "answer": "According to the excerpts, official estimates place the total Ainu population of Japan at 25,000, though unofficial estimates place the population at 200,000 or higher due to assimilation and many individuals of Ainu descent lacking knowledge of their ancestry.",
        "in": 1227,
        "out": 59,
        "cost": 0.001522,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.75,
        "api_seconds": 1.29,
        "correct": true
      },
      "rag": {
        "answer": "According to the excerpts, the official estimate places the total Ainu population of Japan at 25,000. However, unofficial estimates place the total population at 200,000 or higher, as many individuals of Ainu descent have assimilated into Japanese society without knowledge of their ancestry.",
        "in": 1409,
        "out": 65,
        "cost": 0.001734,
        "calls": 1,
        "seconds": 2.96,
        "api_seconds": 1.2,
        "correct": true
      }
    },
    {
      "q": "In which Mexican state is Acapulco?",
      "type": "fact",
      "expect": [
        "Guerrero"
      ],
      "grape": {
        "answer": "Acapulco is located in the state of **Guerrero** on the Pacific Coast of Mexico. According to the excerpts, \"Acapulco de Juárez is a city and major seaport in the state of Guerrero on the Pacific Coast of Mexico.\"",
        "in": 1571,
        "out": 65,
        "cost": 0.001896,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.29,
        "api_seconds": 1.13,
        "correct": true
      },
      "rag": {
        "answer": "Acapulco is located in the state of **Guerrero**, Mexico. According to the excerpts, it is \"a city and major seaport in the state of Guerrero on the Pacific Coast of Mexico\" and \"the most populated city in the state of Guerrero.\"",
        "in": 1367,
        "out": 65,
        "cost": 0.001692,
        "calls": 1,
        "seconds": 3.04,
        "api_seconds": 1.17,
        "correct": true
      }
    },
    {
      "q": "How many KNVB Cups has AFC Ajax won?",
      "type": "number",
      "expect": [
        [
          "20",
          "twenty"
        ]
      ],
      "grape": {
        "answer": "According to the excerpts, AFC Ajax has won **20 KNVB Cups**. This is stated in the first excerpt which notes that Ajax is \"the most successful club in the Netherlands, with 36 [championships] and 20 KNVB Cups.\"",
        "in": 1545,
        "out": 63,
        "cost": 0.0018599999999999999,
        "calls": 1,
        "route": "direct",
        "coverage": 0.96,
        "seconds": 3.38,
        "api_seconds": 1.2,
        "correct": true
      },
      "rag": {
        "answer": "According to the excerpts, AFC Ajax has won 20 KNVB Cups, making it the most successful club in the Netherlands along with 36 league titles.",
        "in": 1273,
        "out": 40,
        "cost": 0.0014730000000000001,
        "calls": 1,
        "seconds": 3.09,
        "api_seconds": 1.26,
        "correct": true
      }
    },
    {
      "q": "Who was the city of Ann Arbor named after?",
      "type": "paraphrase",
      "expect": [
        [
          "wives",
          "wife"
        ]
      ],
      "grape": {
        "answer": "Ann Arbor was named after the wives of the village's founders, both named Ann, and the stands of bur oak trees they found at the site of the town.",
        "in": 1391,
        "out": 39,
        "cost": 0.0015860000000000002,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.36,
        "api_seconds": 0.89,
        "correct": true
      },
      "rag": {
        "answer": "Ann Arbor was named after the wives of the village's founders John Allen and Elisha Walker Rumsey, both of whom were named Ann, and for the stands of bur oak trees found at the site.",
        "in": 1452,
        "out": 49,
        "cost": 0.001697,
        "calls": 1,
        "seconds": 2.95,
        "api_seconds": 1.12,
        "correct": true
      }
    },
    {
      "q": "What population did the 2020 census record for Ann Arbor?",
      "type": "number",
      "expect": [
        [
          "123,851",
          "123851"
        ]
      ],
      "grape": {
        "answer": "According to the 2020 census, Ann Arbor's population was 123,851, making it the fifth-largest city in Michigan.",
        "in": 1375,
        "out": 34,
        "cost": 0.001545,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.3,
        "api_seconds": 0.93,
        "correct": true
      },
      "rag": {
        "answer": "According to the excerpts, the 2020 U.S. Census recorded Ann Arbor's population to be 123,851, making it the fifth-largest city in Michigan.",
        "in": 1511,
        "out": 43,
        "cost": 0.001726,
        "calls": 1,
        "seconds": 2.91,
        "api_seconds": 1.04,
        "correct": true
      }
    },
    {
      "q": "In what year was Antoine Lavoisier guillotined?",
      "type": "number",
      "expect": [
        "1794"
      ],
      "grape": {
        "answer": "According to the excerpts, Antoine Lavoisier was guillotined in **1794**. Specifically, he was convicted and guillotined on 8 May 1794 in Paris at the age of 50.",
        "in": 1354,
        "out": 52,
        "cost": 0.001614,
        "calls": 1,
        "route": "direct",
        "coverage": 0.96,
        "seconds": 3.33,
        "api_seconds": 1.08,
        "correct": true
      },
      "rag": {
        "answer": "Antoine Lavoisier was guillotined in 1794. According to the excerpts, he \"was convicted and guillotined on 8 May 1794 in Paris, at the age of 50.\"",
        "in": 1158,
        "out": 50,
        "cost": 0.001408,
        "calls": 1,
        "seconds": 2.83,
        "api_seconds": 1.04,
        "correct": true
      }
    },
    {
      "q": "Which element did Lavoisier predict would exist, in 1787?",
      "type": "fact",
      "expect": [
        "silicon"
      ],
      "grape": {
        "answer": "According to the excerpts, Lavoisier predicted the existence of **silicon** in 1787. The text states: \"He predicted the existence of silicon (1787) and discovered that, although matter may change its form or shape, its mass always remains the same.\"",
        "in": 1470,
        "out": 60,
        "cost": 0.0017699999999999999,
        "calls": 1,
        "route": "direct",
        "coverage": 0.65,
        "seconds": 3.44,
        "api_seconds": 1.15,
        "correct": true
      },
      "rag": {
        "answer": "According to the excerpts, Lavoisier predicted the existence of **silicon** in 1787.",
        "in": 1540,
        "out": 25,
        "cost": 0.0016649999999999998,
        "calls": 1,
        "seconds": 2.86,
        "api_seconds": 0.98,
        "correct": true
      }
    },
    {
      "q": "Who developed the process that makes ammonia from nitrogen in the air?",
      "type": "fact",
      "expect": [
        "Haber",
        "Bosch"
      ],
      "grape": {
        "answer": "Fritz Haber and Carl Bosch developed the Haber–Bosch process to produce ammonia from nitrogen in the air in 1909 and patented it in 1910.",
        "in": 1286,
        "out": 45,
        "cost": 0.001511,
        "calls": 1,
        "route": "direct",
        "coverage": 0.87,
        "seconds": 3.17,
        "api_seconds": 0.95,
        "correct": true
      },
      "rag": {
        "answer": "The Haber–Bosch process, which produces ammonia from nitrogen in the air, was developed by **Fritz Haber and Carl Bosch in 1909** and patented in 1910.",
        "in": 1517,
        "out": 49,
        "cost": 0.001762,
        "calls": 1,
        "seconds": 2.97,
        "api_seconds": 1.07,
        "correct": true
      }
    },
    {
      "q": "What share of industrially produced ammonia goes into fertilisers?",
      "type": "number",
      "expect": [
        "70"
      ],
      "grape": {
        "answer": "Around 70% of industrially produced ammonia is used to make fertilisers in various forms and composition, such as urea and diammonium phosphate.",
        "in": 1262,
        "out": 37,
        "cost": 0.0014470000000000002,
        "calls": 1,
        "route": "direct",
        "coverage": 0.74,
        "seconds": 3.04,
        "api_seconds": 0.88,
        "correct": true
      },
      "rag": {
        "answer": "According to the excerpts, approximately **70% of industrially produced ammonia is used to make fertilisers** in various forms and composition, such as urea and diammonium phosphate, with ammonia also being applied directly into the soil.",
        "in": 1269,
        "out": 55,
        "cost": 0.001544,
        "calls": 1,
        "seconds": 3.0,
        "api_seconds": 1.17,
        "correct": true
      }
    },
    {
      "q": "What is the capital of Brunei?",
      "type": "fact",
      "expect": [
        "Bandar Seri Begawan"
      ],
      "grape": {
        "answer": "The capital of Brunei is Bandar Seri Begawan, which is also the country's largest city and home to approximately 100,000 residents.",
        "in": 1675,
        "out": 39,
        "cost": 0.0018700000000000001,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.3,
        "api_seconds": 1.03,
        "correct": true
      },
      "rag": {
        "answer": "The capital of Brunei is **Bandar Seri Begawan**, which is located in the Brunei-Muara District and is home to approximately 100,000-150,000 residents, making it the country's largest city.",
        "in": 1406,
        "out": 58,
        "cost": 0.001696,
        "calls": 1,
        "seconds": 3.34,
        "api_seconds": 1.63,
        "correct": true
      }
    },
    {
      "q": "What is the name of the animal boundary Alfred Russel Wallace found in the Malay Archipelago?",
      "type": "paraphrase",
      "expect": [
        "Wallace Line"
      ],
      "grape": {
        "answer": "The animal boundary Alfred Russel Wallace found in the Malay Archipelago is called the **Wallace Line**. It is a faunal boundary that separates the Indonesian archipelago into two distinct parts: a western portion with animals largely of Asian origin, and an eastern portion where the fauna reflect Australasia.",
        "in": 1505,
        "out": 69,
        "cost": 0.00185,
        "calls": 1,
        "route": "direct",
        "coverage": 0.82,
        "seconds": 3.5,
        "api_seconds": 1.23,
        "correct": true
      },
      "rag": {
        "answer": "The animal boundary Alfred Russel Wallace found in the Malay Archipelago is called the **Wallace Line**, which is a faunal boundary that separates the Indonesian archipelago into two distinct parts: a western portion with fauna of Asian origin and an eastern portion with fauna reflecting Australasia.",
        "in": 1259,
        "out": 65,
        "cost": 0.001584,
        "calls": 1,
        "seconds": 3.37,
        "api_seconds": 1.57,
        "correct": true
      }
    },
    {
      "q": "At which battle did Alfred the Great defeat the Vikings in 878?",
      "type": "fact",
      "expect": [
        "Edington"
      ],
      "grape": {
        "answer": "Alfred the Great defeated the Vikings at the **Battle of Edington in 878**. According to the excerpts, he won this decisive victory and subsequently made an agreement with the Vikings that divided England between Anglo-Saxon territory and the Viking-ruled Danelaw.",
        "in": 1618,
        "out": 59,
        "cost": 0.0019130000000000002,
        "calls": 1,
        "route": "direct",
        "coverage": 0.9,
        "seconds": 3.38,
        "api_seconds": 1.13,
        "correct": true
      },
      "rag": {
        "answer": "Alfred the Great defeated the Vikings at the **Battle of Edington in 878**. Following this decisive victory, he made an agreement with the Vikings that divided England between Anglo-Saxon territory and the Viking-ruled Danelaw.",
        "in": 1405,
        "out": 52,
        "cost": 0.001665,
        "calls": 1,
        "seconds": 3.08,
        "api_seconds": 1.28,
        "correct": true
      }
    },
    {
      "q": "What is the largest city in Andhra Pradesh?",
      "type": "fact",
      "expect": [
        "Visakhapatnam"
      ],
      "grape": {
        "answer": "Based on the excerpts provided, the largest city in Andhra Pradesh is Visa… (the text appears to be cut off in the excerpt), as stated in the first excerpt which mentions \"the largest city being Visa…\"\n\nHowever, since the city name is incomplete in the provided excerpts, I cannot give you the full answer with certainty from these documents alone.",
        "in": 1513,
        "out": 81,
        "cost": 0.001918,
        "calls": 1,
        "route": "direct",
        "coverage": 1.0,
        "seconds": 3.76,
        "api_seconds": 1.53,
        "correct": false,
        "judge_cost": 0.0008060000000000001,
        "judged": true
      },
      "rag": {
        "answer": "Based on the excerpts provided, the largest city in Andhra Pradesh is **Visakhapatnam**.",
        "in": 1325,
        "out": 27,
        "cost": 0.00146,
        "calls": 1,
        "seconds": 3.09,
        "api_seconds": 1.23,
        "correct": true
      }
    },
    {
      "q": "Which earlier online service does AOL trace its history to?",
      "type": "fact",
      "expect": [
        "PlayNET"
      ],
      "grape": {
        "answer": "AOL traces its history to an online service known as PlayNET, which licensed its software to Quantum Link (Q-Link), that went online in November 1985.",
        "in": 1443,
        "out": 40,
        "cost": 0.0016430000000000001,
        "calls": 1,
        "route": "direct",
        "coverage": 0.83,
        "seconds": 3.18,
        "api_seconds": 0.9,
        "correct": true
      },
      "rag": {
        "answer": "According to the excerpts, AOL traces its history to an online service known as **PlayNET**. PlayNET licensed its software to Quantum Link (Q-Link), which went online in November 1985 and was eventually renamed as America Online in 1989.",
        "in": 1517,
        "out": 62,
        "cost": 0.0018269999999999998,
        "calls": 1,
        "seconds": 3.06,
        "api_seconds": 1.13,
        "correct": true
      }
    },
    {
      "q": "Which atomic numbers does the actinide series span?",
      "type": "number",
      "expect": [
        "89",
        "102"
      ],
      "grape": {
        "answer": "The actinide series spans atomic numbers from 89 to 102, encompassing the 14 metallic chemical elements from actinium through nobelium.",
        "in": 1393,
        "out": 36,
        "cost": 0.001573,
        "calls": 1,
        "route": "direct",
        "coverage": 0.85,
        "seconds": 3.28,
        "api_seconds": 0.9,
        "correct": true
      },
      "rag": {
        "answer": "According to the excerpts, the actinide series encompasses the 14 metallic chemical elements with atomic numbers from 89 to 102, spanning from actinium through nobelium.",
        "in": 1316,
        "out": 42,
        "cost": 0.0015259999999999998,
        "calls": 1,
        "seconds": 2.74,
        "api_seconds": 1.02,
        "correct": true
      }
    },
    {
      "q": "Who lived longer, Antoine Lavoisier or Alfred Russel Wallace?",
      "type": "cross",
      "expect": [
        "Wallace"
      ],
      "grape": {
        "answer": "Alfred Russel Wallace lived longer. Lavoisier lived from 1743 to 1794 (51 years), while Wallace lived from 1823 to 1913 (90 years).",
        "in": 2075,
        "out": 79,
        "cost": 0.00247,
        "calls": 2,
        "route": "route-search",
        "coverage": 0.42,
        "searched": [
          [
            "Antoine Lavoisier lifespan birth death",
            "Alfred Russel Wallace lifespan birth death"
          ]
        ],
        "seconds": 6.28,
        "api_seconds": 1.9,
        "correct": true
      },
      "rag": {
        "answer": "Alfred Russel Wallace lived longer. According to the excerpts, Antoine Lavoisier lived from 26 August 1743 to 8 May 1794 (50 years), while Alfred Russel Wallace lived from 8 January 1823 to 7 November 1913 (90 years).",
        "in": 1324,
        "out": 76,
        "cost": 0.0017040000000000002,
        "calls": 1,
        "seconds": 3.02,
        "api_seconds": 1.27,
        "correct": true
      }
    },
    {
      "q": "What will the weather be in London tomorrow?",
      "type": "out_of_scope",
      "expect": "NOT FOUND",
      "grape": {
        "answer": "Not in the documents.",
        "in": 1409,
        "out": 8,
        "cost": 0.0014490000000000002,
        "calls": 1,
        "route": "direct",
        "coverage": 0.63,
        "seconds": 2.94,
        "api_seconds": 0.76,
        "correct": true
      },
      "rag": {
        "answer": "Not in the documents.\n\nThe excerpts provided contain climate and weather information about Athens, Atlanta, Albania, and Aberdeen, but do not include any information about London's weather forecast for tomorrow.",
        "in": 992,
        "out": 42,
        "cost": 0.001202,
        "calls": 1,
        "seconds": 2.83,
        "api_seconds": 1.01,
        "correct": true
      }
    },
    {
      "q": "Write a haiku about autumn.",
      "type": "out_of_scope",
      "expect": "NOT FOUND",
      "grape": {
        "answer": "I appreciate your request, but I need to let you know:\n\n**Not in the documents.**\n\nThe excerpts provided discuss haiku as a poetic form and autumn as a season, but they don't contain an actual haiku about autumn that I could share with you. I'd be happy to write an original haiku for you instead if you'd like!",
        "in": 1134,
        "out": 78,
        "cost": 0.001524,
        "calls": 1,
        "route": "direct",
        "coverage": 0.63,
        "seconds": 3.57,
        "api_seconds": 1.51,
        "correct": true
      },
      "rag": {
        "answer": "Not in the documents.\n\nThe excerpts provided contain information about English language variations, mourning customs, colors, and linguistic terms, but none of them contain content suitable for writing a haiku about autumn.",
        "in": 1500,
        "out": 45,
        "cost": 0.001725,
        "calls": 1,
        "seconds": 2.95,
        "api_seconds": 1.18,
        "correct": true
      }
    },
    {
      "q": "Translate 'good morning' into German.",
      "type": "out_of_scope",
      "expect": "NOT FOUND",
      "grape": {
        "answer": "Not in the documents.\n\nThe excerpts provided do not contain information about German translations or language instruction. While one excerpt mentions \"Good morning\" in an English advertising slogan, there is no German translation provided in these documents.",
        "in": 1562,
        "out": 49,
        "cost": 0.001807,
        "calls": 1,
        "route": "direct",
        "coverage": 0.71,
        "seconds": 3.5,
        "api_seconds": 1.31,
        "correct": true
      },
      "rag": {
        "answer": "Not in the documents.\n\nThe excerpts provided do not contain a translation of \"good morning\" into German. While they include some German text and references to German language and culture, none of them provide this specific translation.",
        "in": 1174,
        "out": 48,
        "cost": 0.001414,
        "calls": 1,
        "seconds": 3.1,
        "api_seconds": 1.32,
        "correct": true
      }
    },
    {
      "q": "What is 144 divided by 12?",
      "type": "out_of_scope",
      "expect": "NOT FOUND",
      "grape": {
        "answer": "Not in the documents.",
        "in": 0,
        "out": 0,
        "cost": 0,
        "calls": 0,
        "route": "refused",
        "coverage": 0.0,
        "seconds": 0.42,
        "api_seconds": 0.0,
        "correct": true
      },
      "rag": {
        "answer": "Not in the documents.",
        "in": 1580,
        "out": 8,
        "cost": 0.00162,
        "calls": 1,
        "seconds": 2.65,
        "api_seconds": 0.73,
        "correct": true
      }
    },
    {
      "q": "How do I install Python on Windows?",
      "type": "out_of_scope",
      "expect": "NOT FOUND",
      "grape": {
        "answer": "Not in the documents.\n\nThe excerpts provided discuss Python only as a mythological serpent from Greek mythology and as a word in linguistics/zoology, not as a programming language or its installation process.",
        "in": 991,
        "out": 45,
        "cost": 0.0012159999999999999,
        "calls": 1,
        "route": "direct",
        "coverage": 0.53,
        "seconds": 3.57,
        "api_seconds": 1.39,
        "correct": true
      },
      "rag": {
        "answer": "Not in the documents.\n\nThe excerpts provided contain information about Windows operating system history, Bluetooth compatibility, and other topics, but do not include any instructions on how to install Python on Windows.",
        "in": 1251,
        "out": 43,
        "cost": 0.0014659999999999999,
        "calls": 1,
        "seconds": 2.97,
        "api_seconds": 1.16,
        "correct": true
      }
    },
    {
      "q": "Who won the 2018 Wimbledon men's singles final?",
      "type": "out_of_scope",
      "expect": "NOT FOUND",
      "grape": {
        "answer": "Not in the documents.\n\nThe excerpts provided are about Andre Agassi's Wimbledon performances in the 1990s and early 2000s, and British Airways' sponsorship roles, but they do not contain information about who won the 2018 Wimbledon men's singles final.",
        "in": 1502,
        "out": 70,
        "cost": 0.001852,
        "calls": 1,
        "route": "direct",
        "coverage": 0.69,
        "seconds": 3.74,
        "api_seconds": 1.3,
        "correct": true
      },
      "rag": {
        "answer": "Not in the documents.\n\nThe excerpts provided are about Andre Agassi's career and his matches against Pete Sampras, but they do not contain information about the 2018 Wimbledon men's singles final.",
        "in": 1516,
        "out": 52,
        "cost": 0.001776,
        "calls": 1,
        "seconds": 2.94,
        "api_seconds": 1.2,
        "correct": true
      }
    }
  ]
}