Introduction: The Challenge of Multi-Agent Coordination

In reinforcement learning, training a single agent to solve a maze or collect targets is a well-understood problem. However, when you introduce multiple agents into the same environment, the complexity scales exponentially. This field, known as Multi-Agent Reinforcement Learning (MARL), introduces a fundamental problem: the environment becomes non-stationary because the actions of other learning agents constantly alter the state transitions and reward outcomes.

The repository swarm-foraging-qlearn offers a practical Python simulation exploring how decentralized agents can use Q-learning to solve a collaborative “swarm foraging” task in a dynamic grid. This guide breaks down the core concepts, reward architectures, and hyperparameter tuning tips required to get such cooperative AI swarms working.

Reward Design: Aligning Individual Actions with Collective Goals

The most critical factor in MARL is the reward function. If the reward function is purely individual, agents will compete with each other, leading to collisions and inefficient coverage. If the reward is purely collective, it can lead to the “lazy agent” problem where one agent does all the work while others wander aimlessly.

In the swarm-foraging-qlearn simulation, a hybrid reward function is used to align goals:

  • Resource Collection (+10): Rewarded to the specific agent that successfully harvests a resource cell.
  • Swarm Success Sharing (+2): Distributed to all active agents when any resource is collected, encouraging cooperative coverage.
  • Collision Penalty (-3): Deducted if two agents attempt to enter the same grid cell, teaching them to spread out.
  • Time Step Penalty (-0.1): A tiny constant negative reward for every movement step, driving the agents to find resources as quickly as possible.

Tuning Hyperparameters for AI Swarms

Tuning learning hyperparameters in multi-agent grids requires careful balancing. Let’s look at the key values to configure in config.py:

1. Learning Rate (Alpha)

A high learning rate (alpha > 0.5) will cause agents to overwrite their Q-tables too quickly in response to other agents’ behaviors, leading to unstable oscillations. Keep alpha between 0.05 and 0.15 to ensure slow, steady convergence.

2. Discount Factor (Gamma)

The discount factor determines how much future rewards are valued over immediate ones. For swarm foraging, a higher gamma (0.90 to 0.98) is preferred so agents prioritize long-term paths and returning to the home base over immediate local exploration.

3. Epsilon-Greedy Exploration (Epsilon Decaying)

Start with epsilon = 1.0 (pure exploration) and decay it slowly (epsilon_decay = 0.995) to 0.05. This ensures that early in training, the swarm explores all quadrants of the dynamic grid before settling into coordinated exploitation routes.

Code Implementation Overview

Below is a simplified architecture showing how agents update their local Q-tables during the simulation loop:

# Multi-agent Bellman equation update step
for agent in swarm:
    state = agent.get_local_state(grid)
    action = agent.select_action(state, epsilon)
    next_state, reward, done = grid.step(agent, action)
    
    # Q-value lookup and update
    old_value = Q_table[agent.id][state][action]
    next_max = np.max(Q_table[agent.id][next_state])
    
    # Update rule
    new_value = (1 - alpha) * old_value + alpha * (reward + gamma * next_max)
    Q_table[agent.id][state][action] = new_value

This decentralized update loop allows each agent to construct its own mapping of state-action valuations while interacting with others.

Key Takeaways

  • Decentralized Learning: Each agent maintains a separate Q-table, removing the need for a single centralized controller and improving system fault tolerance.
  • Stationarity Tuning: Keeping a lower learning rate (alpha) stabilizes the non-stationarity introduced by multiple learning entities.
  • Dynamic Adaptability: Agents dynamically update Q-values when resources spawn or disappear, showing natural adaptability.

FAQ

What is Q‑Learning swarm foraging?

It is the application of reinforcement learning to train a collection of decentralized agents (a swarm) to search for, gather, and secure resources in an environment. The agents learn coordination behaviors solely through environmental interactions.

How does multi‑agent reinforcement learning work in dynamic grids?

Each agent perceives a localized state representation (e.g., coordinates and neighboring grid cells) and performs actions. They update their individual Q-tables based on shared rewards and collision penalties, continuously adapting to the movements of other agents and dynamic resource spawns.

Where can I find the swarm foraging Q‑Learning code on GitHub?

The simulation source code and parameters reside in the repository jaimasih05-commits/swarm-foraging-qlearn. The repository provides pre-built grid environments, training logs, and plotting utilities.