I was able to track down my issues with this, and I'm beginning to understand it better. gl_FragCoord is indeed usable but I learned two important points, which I'll document here in case they are useful to others in the future:
- The coordinate system is opposite to the normal AM screen coordinates. Within a shader (0,0) is bottom left instead of top left.
- The coordinates are relative to the entire screen size. Normally this should not be an issue but in my case I am setting fe.layout.width and fe.layout.height to make the dimensions smaller because I'm testing different layout resolutions. This leaves some unused space on the screen edges, and those pixels still count.
For point 1, we can change the behavior by placing this line at the top of the shader code, switching to the AM style layout coordinates:
layout(origin_upper_left) in vec4 gl_FragCoord;
For point 2, I'm now passing a set of offset coordinates to the shader, and using them to shift the position and accommodate the extra screen edges. In my case this was like:
shader.set_param("fragmentOffset", -(ScreenWidth - flw) / 2, -(ScreenHeight - flh) / 2);
And then used in the shader like so:
vec2 position = gl_FragCoord.xy + fragmentOffset;
(then I use position instead of gl_FragCoord)