TL;DR#
Large Language Models (LLMs) excel at answering questions about charts, but frequently produce inaccurate or ‘hallucinated’ answers. This lack of reliability hinders trust and practical use in professional settings. Existing methods for attributing answers to chart elements struggle due to complexities in visual-text alignment and bounding box prediction.
ChartCitor, a novel multi-agent LLM framework, directly addresses these limitations. It uses a series of specialized agents to extract data from charts, reformulate questions, retrieve relevant evidence using pre-filtering and re-ranking, and map that evidence back to specific chart elements. This approach significantly improves accuracy and provides users with readily understandable and trustworthy explanations for LLM-generated ChartQA responses.
Key Takeaways#
Why does it matter?#
This paper is important because it tackles the crucial problem of hallucination in LLMs for chart question answering (ChartQA). By introducing ChartCitor, it offers a novel multi-agent approach that enhances accuracy and trustworthiness, addressing a significant limitation of current LLMs. This work is relevant to researchers in NLP, computer vision, and explainable AI, opening up new avenues for improving the reliability and explainability of large language models for visual data analysis.
Visual Insights#
Full paper#
