AeroQuery
Ask questions about Canadian aviation regulations. Powered by RAG over the TC AIM.
For demonstration purposes only. AI-generated answers can be incomplete or wrong. Always cross-verify every response against the official Transport Canada AIM and other authoritative sources before relying on it. This tool is provided “as is” and assumes no responsibility or liability for any decisions made or outcomes resulting from its use.
Eval Benchmark (RAGAS v0.4)
50 questions · 6 metrics · 3 models · Judged by GPT-5.4-mini via Azure AI Foundry
| Metric | GPT-5.4-mini | GPT-3.5-turbo | Llama 3.2 1B (WebLLM) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Bare | RAG | +Rerank | Bare | RAG | +Rerank | Bare | RAG | +Rerank | |
| Factual Correctness | 0.32 | 0.40 | 0.43↑34% | 0.23 | 0.44↑91% | 0.44↑91% | 0.07 | 0.18 | 0.22↑214% |
| RAG triples Llama accuracy. Reranking adds another boost. | |||||||||
| Faithfulness | — | 0.82 | 0.88↑7% | — | 0.82 | 0.85↑4% | — | 0.52 | 0.55↑6% |
| GPT models stay grounded. Llama often ignores context. | |||||||||
| Context Precision | — | 0.78 | 0.92↑18% | — | 0.78 | 0.92↑18% | — | 0.78 | 0.93↑19% |
| Cohere rerank: 0.78 → 0.92. Right chunks, right order. | |||||||||
| Context Recall | — | 0.88 | 0.92↑5% | — | 0.93 | 0.93 | — | 0.94↑4% | 0.90 |
| 94% recall — retrieval finds the right information. | |||||||||
| Answer Relevancy | 0.84↑8% | 0.78↓7% | 0.79↓6% | 0.83 | 0.86 | 0.87↑5% | 0.79↑36% | 0.58↓27% | 0.62↓22% |
| Bare LLM scores higher by confidently hallucinating relevant-sounding answers. RAG correctly refuses when unsure — for aviation safety, faithfulness > relevancy. | |||||||||
| Semantic Similarity | 0.69 | 0.77↑12% | 0.77↑12% | 0.74 | 0.80↑8% | 0.79 | 0.66 | 0.67 | 0.69↑5% |
| RAG answers are consistently closer to ground truth. | |||||||||
↑ Green = best score (improvement). ↓ Red = worst score (regression). Retrieval metrics are model-independent.
Built With
AI & Retrieval:OpenAIEmbeddings & LLM·PineconeVector DB·Cohere RerankReranking·WebLLMIn-browser LLM·OllamaLocal inference