Past research chapters

I previously explored the idea of anticipating the future and adapting to it, and proposed a simple template for accelerating policy gradient algorithms by integrating foresight into the policy improvement step, via optimistic and adaptive policy updates. I defined optimism as predictive modeling of the future behavior of a policy, and adaptivity as taking immediate and anticipatory corrective actions to mitigate accumulating errors from overshooting predictions or delayed responses to change. Currently, I am investigating acceleration within the Policy Mirror Descent (PMD) general family of algorithms, which cover a wide range of novel and fundamental methods in reinforcement learning.

I also studied policy optimization as a joint-maximization problem and worked on a surrogate policy learning objective for the joint maximization of a policy and its value-function. Practical implementations of policy-based algorithms rely on value-functions, represented as neural networks, to compute the policy gradient, introducing challenges related to the accuracy of the policy gradient, particularly in low-capacity regimes, characteristic of agents with bounded rationality.

Previously, I studied credit assignment in value-based agents, focusing on questions related to how agents should model the environments they interact with, whether that be in anticipation using forethought, or retrospectively using hindsight models and backward looking mechanisms for adaptivity.

I then showed how these models can be extended to include selectivity via simple contextual attention-based mechanisms and learn those from experience.

Traditionally, in machine learning, we often assume that the stream of data in the future will resemble the data seen so far, yet these assumptions may not align with the complexity of real-world settings, where the dynamics of the environment evolve in partially predictable ways over time and space. Previously, I have explored some of these challenges, and proposed incorporating predictive knowledge (of the agent's future behavior) to mitigate them.