PragmaticQA: Cooperative Question Answering with LLMs

This project explores pragmatic and cooperative question answering using large language models, focusing on moving beyond literal responses toward answers that better reflect user intent and conversational context.

The system is built on the PragmatiCQA benchmark, which requires models to answer questions while incorporating both explicit information and implicit context derived from dialogue history. The goal is to simulate more human-like responses that anticipate user needs and provide richer, more informative answers.

Two approaches are implemented and compared. The first is a traditional question answering baseline based on DistilBERT, which extracts answers from literal spans, pragmatic spans, or retrieved context. The second is a multi-step reasoning pipeline built using the DSPy framework, designed to generate cooperative answers through structured reasoning.

The DSPy-based pipeline includes several components. A conversation analyzer module summarizes user interests and goals from the dialogue. A pragmatic reasoner infers implicit intent and generates enriched queries. A cooperative answer generator synthesizes responses that combine retrieved knowledge with inferred context. These components are orchestrated in a multi-step retrieval and generation pipeline.

The system is evaluated using the Semantic F1 metric, which measures answer quality based on semantic similarity rather than exact match. Results show that while traditional QA performs well on direct factual queries, the multi-step LLM approach produces more context-aware and informative answers in conversational settings.

The project demonstrates how LLMs can approximate Theory of Mind-like reasoning by anticipating follow-up questions, clarifying intent, and adapting responses to user context. This makes it highly relevant for applications such as conversational agents, virtual assistants, and advanced search systems.