Interactive Query Generation and Reformulation

Kaustubh Dhole
GenQREnsemble
  • GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation Kaustubh Dhole, Eugene Agichtein [ECIR 2024] Paper 📎
    • Query Reformulation(QR) is a set of techniques used to transform a user’s original search query to a text that better aligns with the user’s intent and improves their search experience. Recently, zero-shot QR has been a promising approach due to its ability to exploit knowledge inherent in large language models. By taking inspiration from the success of ensemble prompting strategies which have benefited many tasks, we investigate if they can help improve query reformulation. In this context, we propose an ensemble-based prompting technique, GenQREnsemble which leverages paraphrases of a zero-shot instruction to generate multiple sets of keywords ultimately improving retrieval performance. We further introduce its post-retrieval variant, GenEnsemblePRF to incorporate pseudo-relevant feedback. On evaluations over four IR benchmarks, we find that GenQREnsemble generates better reformulations with relative nDCG@10 improvements of up to 18% and MAP improvements up to 24% over the previous zero-shot state-of-art. On the MSMarco Passage Ranking task, GenEnsemblePRF shows relative gains of 5% MRR using pseudo-relevance feedback, and 9% nDCG@10 using relevant feedback documents.
    • Also, check extended analysis on Ensemble-based Document Fusion, GenQRFusion/GenQRFusionRF, filtering, and Interpretable Queries in Generative Query Reformulation Using Ensemble Prompting, Document Fusion, and Relevance Feedback, Kaustubh Dhole, Ramraj Chandradevan and Eugene Agichtein
  • QueryExplorer: An Interactive Query Generation Assistant for Search and Exploration Kaustubh Dhole, Shivam Bajaj, Ramraj Chandradevan, Eugene Agichtein [NAACL 2024 -- System Demonstration Track]   Video 🎥  Google Colab Code 💻  Paper 📎

    • Formulating effective search queries remains challenging, particularly when users lack expertise in a specific domain or are not proficient in the language of the content. Providing example documents of interest might be easier for a user. However, such query-by-example scenarios are prone to concept drift, and the retrieval effectiveness is highly sensitive to the query generation method, without a clear way to incorporate user feedback. To enable exploration and to support human-in-the-loop experiments we propose QueryExplorer -- an interactive query generation, reformulation, and retrieval interface with support for HuggingFace 🤗 generation models and 🐕 PyTerrier's retrieval pipelines and datasets, and extensive logging of human feedback. To allow users to create and modify effective queries, our demo supports complementary approaches of using LLMs interactively, assisting the user with edits and feedback at multiple stages of the query formulation process. With support for recording fine-grained interactions and user annotations, QueryExplorer can serve as a valuable experimental and research platform for annotation, qualitative evaluation, and conducting Human-in-the-Loop (HITL) experiments for complex search tasks where users struggle to formulate queries.

  • An Interactive Query Generation Assistant using LLM-based Prompt Modification and User Feedback Kaustubh Dhole, Ramraj Chandradevan, Eugene Agichtein [IARPA, USA - Intelligence Advanced Research Projects Activity - BETTER 2023]. Paper 📎Video 🎥
    • While search is the predominant method of accessing information, formulating effective queries remains challenging, especially for situations where the users are unfamiliar with a domain, searching for documents in other languages, or looking for complex information such as events, which are not easily expressible as queries. Providing example documents or passages of interest might be easier for a user, however, such query-by-example scenarios are prone to concept drift and are highly sensitive to the query generation method. This demo illustrates complementary approaches of using LLMs interactively, assisting and enabling the user to provide edits and feedback at all stages of the query formulation process. The proposed Query Generation Assistant is a novel search interface that supports automatic and interactive query generation over a mono-lingual or multi-lingual document collection. Specifically, the proposed assistive interface enables the users to refine the queries generated by different LLMs, provide feedback on the retrieved documents or passages, and incorporate the users' feedback as prompts to generate more effective queries. The proposed interface is a valuable experimental tool for exploring fine-tuning and prompting of LLMs for query generation to evaluate the effectiveness of retrieval and ranking models qualitatively, and for conducting Human-in-the-Loop (HITL) experiments for complex search tasks where users struggle to formulate queries without such assistance.