Skip to content

Reward function and observation space

The robot needs to know whether its movements contribute to the ultimate goal of locomotion towards a target. Sensor inputs must be distributed fairly to guarantee an objective comparison between different architectures.

  • The distance from the robot to the target and/or the light intensity are treated as global inputs.
  • Positions and joints, which are normalized to floating-point values between 0 and 1, are considered local inputs.
  • The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal within a finite number of timesteps \(T\).
  • To motivate efficient movement, the amount of timesteps taken to reach the goal will be used as penalty.
  • An extra penalty based on movement relative to the current step and the previous is used to penalize a movement away from the target.

From reward to PPO

The resulting reward is passed to our PPO library. Our critic network (value function) predicts how good our eventual reward will be for the current state, this value is combined with the reward from the reward function to get advantages. These advantages are then used to calculate the losses to update both our critic and actor pipeline.

Rationale

Using a light source (or a gradient) is biologically plausible for many simple organisms. By normalizing all signals between 0 and 1, PPO training is highly stabilized. The timesteps must be finite to reset the environment in a timely manner if the policy gets stuck in a local minimum.

Limitations and alternatives

Providing global information to all individual decentralized segments can be considered biologically cheating or practically infeasible once the robot would be physically built. Some sensory input cannot be put in each joint, for example.

The alternative is to provide the global input to the outermost segments of the arms, or a specific set of segments assigned with this functionality. The network would then have to learn to propagate this signal throughout the body via message passing. While biologically more accurate, this drastically complicates the learning process. We have written this down as potential future research.