Hello folks, I've been experimenting with some camera lens distortion in GLSL, and proceed to implement a simple “Fish Eye” lens. It was done as a post process with a textured fullscreen quad.

The code of shaders are here:

Vertex Shader:

#version 330 core

in vec3 a_position;
in vec2 a_uv;

out vec3 vertex_pos;
out vec2 texcoord;

void main() {
    vertex_pos = a_position;
    texcoord = a_uv;
    gl_Position = vec4(a_position, 1.0);
}

Fragment Shader:

#version 330 core

uniform sampler2D tex0;

out vec4 final_color;

in vec3 vertex_pos;
in vec2 texcoord;

const float PI = 3.14159256;

void main() {    
    float d = dot(vertex_pos, vertex_pos);
    vec2 s = vertex_pos.xy * d;
    vec2 tc = s.xy * 0.5 + 0.5;
    vec4 distorted = texture2D(tex0, tc);
    final_color = distorted;
}

and it works, except that I found out there was too much of a distortion right in the middle. otherwise I could live with that. Could it be done correctly purely using fragment shader, or should I tesselate the quad more finely and distort in the vertex shader instead?

the resulting shaders are shown here

I was aiming to make it sort of like the output of a dual paraboloid shader, in which the vertices are distorted around the middle, made it look “half-spherized” (dunno the term). What should I do?

You don't like the shape of your distortion: just replace uv=xy*r*r with something less harsh (in particular, with nonzero partial derivatives at xy=0 and anywhere else).

Two suggestions:

• The Wikipedia page on fisheye distortion lists several principled distortion types, actually adopted in real lenses, that have a realistic appearance and interesting properties. You'll need further sources for a good treatment of how they work.
• You could start from a reference picture that looks good for your purposes and reverse engineer what deformation would produce it. Your screenshot, showing a narrow FOV up close rather than a wide FOV embracing a whole scene, doesn't look particularly representative of typical fisheye uses.