Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models

1. 1
   Understand Single-Turn Evaluation Limits

   Recognize that traditional LLM evaluations primarily assess only the 'assistant turn,' often overlooking broader conversational context and an LLM's ability to anticipate future dialogue.
2. 2
   Grasp User-Turn Generation Concept

   Learn the new evaluation paradigm: instead of just assessing the LLM's response, prompt the LLM to generate what the *user* might say next, given the ongoing conversation history.
3. 3
   Design a Conversational Scenario

   Create a short, specific dialogue scenario. This should include an initial user query and a hypothetical LLM response, setting the context for the user's next turn.
4. 4
   Prompt for User Follow-up

   Instruct your LLM to predict and generate a plausible user follow-up question or statement, based on the provided conversation and its own previous output. Focus on realistic user intent.
5. 5
   Assess Interaction Awareness

   Evaluate the quality, relevance, and contextual appropriateness of the LLM-generated user turn. A highly relevant and natural user turn indicates better 'interaction awareness' and understanding of dialogue flow.

You are an AI assistant tasked with evaluating another AI's conversational understanding. Here is a conversation snippet:

User: "I'm looking for a low-calorie, high-protein recipe for dinner tonight."
Assistant: "How about grilled chicken with a quinoa salad and steamed broccoli? It's lean, packed with protein, and nutritious."

Based on the assistant's last response, what is a highly likely follow-up question the user would ask next? Generate only the user's question, without any additional commentary.

• Paperarxiv.org