SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

Leverage Large Language Models for Word Sense Disambiguation (WSD) by implementing plausibility scoring within narrative contexts. This approach addresses the gap between LLM benchmark performance and their practical application in complex Natural Language Understanding (NLU) tasks.

intermediate30 min5 steps

The play

Identify Ambiguous Context
Locate sentences or narrative segments where a specific word has multiple potential meanings that require deeper contextual analysis to resolve.
List Candidate Senses
For the identified ambiguous word, compile a list of distinct possible meanings (senses). You can use lexical resources (e.g., WordNet) or an LLM to generate these candidates.
Craft Plausibility Prompt
Design an LLM prompt that asks the model to evaluate the plausibility of each candidate sense fitting the given narrative context. Ensure the prompt is clear and requests a quantifiable score or ranking.
Query LLM for Scores
Send the crafted prompt, along with the narrative context and each candidate sense, to your chosen Large Language Model. Collect the plausibility scores or rankings returned by the LLM.
Disambiguate Based on Score
Analyze the scores from the LLM. Select the candidate sense that received the highest plausibility score as the most appropriate meaning for the ambiguous word in that specific narrative context.

Starter code

import os

# This is a placeholder function. In a real scenario, you'd integrate with an actual LLM API (e.g., OpenAI, Anthropic).
def query_llm(prompt_text):
    print(f"-- LLM Query --\n{prompt_text}\n-- End Query --")
    # Simulate an LLM response for demonstration
    if "financial institution" in prompt_text:
        return "Plausibility Score: 1"
    elif "land alongside a river or lake" in prompt_text:
        return "Plausibility Score: 5"
    else:
        return "Plausibility Score: 3"

narrative_context = "The bank was steep and overgrown, making it difficult to climb. We decided to fish from the other bank."
ambiguous_word = "bank"
candidate_senses = [
    "a financial institution",
    "the land alongside a river or lake",
    "a row or series of objects"
]

results = {}
print(f"Analyzing word '{ambiguous_word}' in context: '{narrative_context}'\n")

for sense in candidate_senses:
    prompt = f"""
    Given the narrative context: "{narrative_context}"
    Consider the ambiguous word: "{ambiguous_word}"

    How plausible is it that "{ambiguous_word}" means "{sense}" in this context?
    Rate its plausibility on a scale of 1 (very implausible) to 5 (very plausible).
    Provide only the numerical score.
    """
    llm_response = query_llm(prompt)
    try:
        # Extract numerical score from the simulated LLM response
        score = int(llm_response.strip().split(":")[-1].strip())
        results[sense] = score
    except (ValueError, IndexError):
        results[sense] = 0 # Handle cases where LLM doesn't return a number or format is unexpected

print("\nPlausibility Scores:")
for sense, score in results.items():
    print(f"  '{sense}': {score}")

if results:
    most_plausible_sense = max(results, key=results.get)
    print(f"\nMost plausible sense identified: '{most_plausible_sense}'")
else:
    print("No plausible senses could be determined.")

Source

Paperarxiv.org