Investigating Shortcuts to Automata in Gridworld

We investigated shortcut solutions (sublinear depth) for simulating automata with Transformers, finding that the networks can learn algorithms resembling theoretical shortcut solutions for Gridworld-9.

REMIX Research Residency, Redwood Research, January 2023