Dynamic Dual-Granularity Skill Bank for Agentic RL

Implement a Dynamic Dual-Granularity Skill Bank (D2Skill) to enhance agentic Reinforcement Learning. This framework stores and evolves skills at both fine-grained and coarse-grained levels, providing reusable experience. It addresses limitations of current skill-based methods by enabling more robust and adaptable agent behaviors.

intermediate1 hour6 steps

The play

Understand Skill Bank Limitations
Identify the shortcomings of existing skill-based RL agents, specifically their reliance on trajectory-level guidance and static skill memories. Recognize the need for dynamic skill evolution.
Design Fine-Grained Skill Representation
Define how atomic actions, primitive behaviors, or low-level policies will be represented and stored as 'fine-grained' skills within your agent's architecture. Consider their input/output and conditions for use.
Design Coarse-Grained Skill Representation
Outline the structure for 'coarse-grained' skills, which represent higher-level behaviors, sub-goals, or meta-policies composed of fine-grained skills. These abstract skills guide broader agent objectives.
Implement Dynamic Skill Extraction & Storage
Develop mechanisms to automatically extract new skills from successful agent experiences. Implement a system to store these skills (both fine and coarse) in a bank, ensuring they are retrievable for future use.
Establish Skill Evolution & Maintenance
Design a strategy for dynamically updating, refining, or discarding skills in the bank based on their utility, frequency of use, or performance. This ensures the skill bank remains relevant and efficient over time.
Integrate Skill Retrieval & Execution
Incorporate the skill bank into your agent's decision-making process. Develop a method for the agent to query the skill bank, select appropriate skills based on the current state and goal, and execute them effectively.

Starter code

class AgentSkillBank:
    def __init__(self):
        self.fine_grained_skills = {}  # Stores low-level actions/policies
        self.coarse_grained_skills = {} # Stores high-level behaviors/meta-policies

    def add_fine_skill(self, name: str, skill_policy):
        """Adds a fine-grained skill (e.g., 'move_forward', 'grab_object')."""
        self.fine_grained_skills[name] = skill_policy
        print(f"Added fine-grained skill: {name}")

    def add_coarse_skill(self, name: str, skill_meta_policy):
        """Adds a coarse-grained skill (e.g., 'explore_area', 'build_shelter')."""
        self.coarse_grained_skills[name] = skill_meta_policy
        print(f"Added coarse-grained skill: {name}")

    def get_skill(self, granularity: str, name: str):
        """Retrieves a skill by granularity and name."""
        if granularity == 'fine':
            return self.fine_grained_skills.get(name)
        elif granularity == 'coarse':
            return self.coarse_grained_skills.get(name)
        return None

Source

Paperarxiv.org