*Python Code SARSA Lambda with Fourier Basis* (OpenAI Gym Simulator)
Plot generated to show the variation of the basis term (from matlab code)
This is a comparison of the fourier basis with plain discretization (plots from the matlab code above).
Clearly, Fourier basis approximation converges much much faster than plain discretization.We expect tile coding with randomly offset tilings to perform better, but even then the fourier basis will still outperform it.
It turned out in this problem that order 3 basis performed best.
1)Tile coding with hashing is much more challenging to implement compared to the fourier basis.
2) The number of terms grow exponentially with the number of observations, and manual tweaking to ensure only higher order terms for important features is bit more challenging compared to tile coding.
3) Also, fourier series being a fluctuating series, has difficulty representing flat areas in value functions as the paper mentions.
Despite, all this, like the author of the paper points out, fourier basis is an excellent starting steps in moderate sized continous state spaces.
Cheers.
