Hi Guys,
I found a shader on Shadertoy that allows for nice looking pixel-art aesthetics even with non-integer scaling, and I was wondering if I could get some help understanding how the shader works.
https://www.shadertoy.com/view/MlB3D3
I feel like the shader is trying to mix bilinear and nearest neighbor sampling, such that nearest neighbor is always used except on pixels that span multiple texels; however, i am not quite sure as to whether this intuition is correct or how the shader manages to achieve it. any help understanding would be very appreciated.
I found a shader on Shadertoy that allows for nice looking pixel-art aesthetics even with non-integer scaling, and I was wondering if I could get some help understanding how the shader works.
https://www.shadertoy.com/view/MlB3D3
I feel like the shader is trying to mix bilinear and nearest neighbor sampling, such that nearest neighbor is always used except on pixels that span multiple texels; however, i am not quite sure as to whether this intuition is correct or how the shader manages to achieve it. any help understanding would be very appreciated.
Edited by Bronxolon
on
Casey had a chat episode where he explained how this works. The math is a bit different, but concept is exactly same.
Watch this: https://www.youtube.com/watch?v=Yu8k7a1hQuU
Shader you linked was mentioned in Q&A.
Watch this: https://www.youtube.com/watch?v=Yu8k7a1hQuU
Shader you linked was mentioned in Q&A.
i'm starting to understand it a bit more, but i read somewhere that an algorithm like this is the same as taking a pixel art image, scaling it up with nearest interpolation by like 400% then scaling it down with bilinear interpolation to the desired scale. apparently, doing this in theory is the same as scaling up with nearest interpolation by an infinite amount; however, avoids having a bigger texture in memory. would you be able to comment as to how the above approach is theoretically parallel with this shader approach? I suppose the idea is that if a pixel is closer to the center it'll be surrounded by texels that are identical to the one it is in, since we scaled up, which effectively makes it so only the edge pixels get a bilinear filter, and somehow by increasing the scale to infinity the granularity increases such that we only bilinearly interpolate pixels than span more than 1 texel. i don't get how increasing the scale makes it more likely to only filter pixels than span texels though.
Edited by Bronxolon
on
Could we have a link to where you read that ? I doesn't sound right to me.
The point of the shader is to do interpolation when a texel doesn't fully covers a frame buffer pixel and not filter otherwise. This depends on the scale and position of the texture/object on the screen and can't be achieved by pre-processing the texture.
A way that might work (I think) with just a texture and bilinear filtering, would be to have a huge texture rendered in a small area so that there would always be multiple texel in a frame buffer pixel. But that would waste a lot of memory/bandwidth and the result would probably not be that good (if it worked at all, I don't actually know).
The point of the shader is to do interpolation when a texel doesn't fully covers a frame buffer pixel and not filter otherwise. This depends on the scale and position of the texture/object on the screen and can't be achieved by pre-processing the texture.
A way that might work (I think) with just a texture and bilinear filtering, would be to have a huge texture rendered in a small area so that there would always be multiple texel in a frame buffer pixel. But that would waste a lot of memory/bandwidth and the result would probably not be that good (if it worked at all, I don't actually know).
yeah so I thought about it for a bit and I'd love to know your opinion on my ideas. So, as far as I can tell there are 3 ways to solve this problem
1. the above pixel shader
2. what you said about pre upsampling a texture, so for example take a small 16x16 texture and pre upsample the asset with nearest neighbour, then render it on a smaller space with linear.
3. MSAA (or maybe SSAA)
as far as I can tell 2 effectively mimics what something like MSAA would do (not overly familiar with MSAA or how it works, so I could be wrong) but wastes a ton of texture memory. As far as I know MSAA works by sampling at multiple points in a pixel and blending the results together, so scaling up the texture effectively mimics that by having more identical texels map to a single pixel. Therefore, it seems like option 1, the shader, is actually the most effective since instead of sampling at discrete points we have floating point resolution for the blending, thus giving close to perfect subpixel accuracy on our art.
EDIT: I guess with MSAA you'd actually want linear blending off (not sure on this and would appreciate input).
Finally, one last question I have on this which is sort of a digression, is do these methods work just as well on arbitrary pixel art rotation as they do on scaling.
"This works for arbitrary scaling, but what about arbitrary rotation? Well, although it is no longer an exact replication of the super-sampling effect when applied to rotation, it is a close approximation. The same shader code can be used for rotations, and it is hard to notice any deficiency." is something I read on https://medium.com/@michelotti.matthew/rendering-pixel-art-c07a85d2dc43
this confuses me because I feel like the properties/effects of the shader should be the same for both rotation and scaling
1. the above pixel shader
2. what you said about pre upsampling a texture, so for example take a small 16x16 texture and pre upsample the asset with nearest neighbour, then render it on a smaller space with linear.
3. MSAA (or maybe SSAA)
as far as I can tell 2 effectively mimics what something like MSAA would do (not overly familiar with MSAA or how it works, so I could be wrong) but wastes a ton of texture memory. As far as I know MSAA works by sampling at multiple points in a pixel and blending the results together, so scaling up the texture effectively mimics that by having more identical texels map to a single pixel. Therefore, it seems like option 1, the shader, is actually the most effective since instead of sampling at discrete points we have floating point resolution for the blending, thus giving close to perfect subpixel accuracy on our art.
EDIT: I guess with MSAA you'd actually want linear blending off (not sure on this and would appreciate input).
Finally, one last question I have on this which is sort of a digression, is do these methods work just as well on arbitrary pixel art rotation as they do on scaling.
"This works for arbitrary scaling, but what about arbitrary rotation? Well, although it is no longer an exact replication of the super-sampling effect when applied to rotation, it is a close approximation. The same shader code can be used for rotations, and it is hard to notice any deficiency." is something I read on https://medium.com/@michelotti.matthew/rendering-pixel-art-c07a85d2dc43
this confuses me because I feel like the properties/effects of the shader should be the same for both rotation and scaling
Edited by Bronxolon
on
If I remember correctly (I'm not an expert or anything), MSAA only works at polygon edges, so it wouldn't help with pixel art textures. SSAA would not help by itself I think. If I'm not mistaken it would end up do the inverse of having a huge texture rendered in a small area (a lot of pixels for a single texel).
The shader should work with rotations (I think). But doing rotation in pixel art generally doesn't look good (personal opinion).
The shader should work with rotations (I think). But doing rotation in pixel art generally doesn't look good (personal opinion).
It is not the same as upscaling the texture and then doing bilinear sampling at a smaller resolution, because bilinear sampling on the GPU only ever samples 4 neighboring texels. If the texture is being sampled at a smaller size than 1:1 pixel:texel, then you would need to sample coverage for more than 4 texels, so bilinear sampling with only 4 texels gives incorrect results when downscaling. This is why mipmaps are used to approximate coverage of more samples when sampling a texture at smaller sizes than 1:1. But it's only an approximation, and generating mipmaps will be a bit useless in this case, for reasons that should be obvious if you think about what happens when you apply a box filter to pixel art that has been upscaled. Hint: a box filter is basically the inverse of nearest-neighbor upscaling.
All that said, upscaling and then doing bilinear sampling with mipmaps would actually give something vaguely close to the correct result, but it will not quite be right (specifically, it will be up to 1 pixel blurrier than desired) for non-power-of-2 scale levels. And obviously it's needlessly expensive.
All that said, upscaling and then doing bilinear sampling with mipmaps would actually give something vaguely close to the correct result, but it will not quite be right (specifically, it will be up to 1 pixel blurrier than desired) for non-power-of-2 scale levels. And obviously it's needlessly expensive.
Hi, I didn't realize I had a new response on here. I mostly followed what you were saying except the end, where you said everything would be "up to 1 pixel blurrier than desired." Why would it be blurrier specifically on non-power of 2 scale levels (I'm assuming this means if we are rendering to a target that is in between 2 mip levels), and why by 1 pixel?
Also, I get what you said here:
"generating mipmaps will be a bit useless in this case, for reasons that should be obvious if you think about what happens when you apply a box filter to pixel art that has been upscaled. Hint: a box filter is basically the inverse of nearest-neighbor upscaling."
but if that is the case why would it still generate a pretty close result. based on what you said it actually seems to me like using mipmaps would kinda defeat the purpose of the whole thing.
Also, I get what you said here:
"generating mipmaps will be a bit useless in this case, for reasons that should be obvious if you think about what happens when you apply a box filter to pixel art that has been upscaled. Hint: a box filter is basically the inverse of nearest-neighbor upscaling."
but if that is the case why would it still generate a pretty close result. based on what you said it actually seems to me like using mipmaps would kinda defeat the purpose of the whole thing.
Edited by Bronxolon
on