Javascript - How do I get 3D coordinates of clicked point and in fragment shader in WebGL?

1valdis
June 4, 2024
107 views
0 votes
2 Answers

I draw a unit sphere of objects (stars, sprites etc), which all have 3D coordinates on the "surface" of that sphere. The camera is at the center of it and I use perspective projection to show the resulting sky chart. Here’s a view from the center of the sphere (how my app will be shown to a user):

And from the side (for you to better understand what I mean by sphere of objects):

I have read numerous articles on unprojections and ray casting, but they eventually

use some libraries like Three.js (I don’t use any libraries except for gl-matrix, just bare WebGL2)
resort to WebGL Picking using drawing IDs of objects (there is a lot of empty space on the chart, so I can’t do that)
say that it’s impossible to do such conversion because one needs depth as well (I have a unit sphere and want coordinates on the surface of it, so in my case depth, or vector length, would always be 1)
are not discussing WebGL2 and JavaScript but OpenGL and C-like languages which happen to have convenient methods for this problem.

Since I have no models and no camera movement other than rotations, I have super simple matrix code:

const projectionMatrix = mat4.create()

mat4.perspective(projectionMatrix,
  degreesToRad(fov),
  viewport.x / viewport.y,
  0,
  100)

const panningMatrix = mat4.create()
// in place of this comment, rotations are applied to it as the user pans the map
mat4.invert(panningMatrix, panningMatrix)

const groundViewMatrix = mat4.create()
mat4.multiply(groundViewMatrix, panningMatrix, groundViewMatrix)

// some-shader.vert; u_modelViewMatrix is groundViewMatrix in this case
gl_Position = u_projectionMatrix * u_modelViewMatrix * a_position;

So, how do I get the 3D coordinates on this unit sphere:

on JavaScript side for a point I click on (so that I can transform them to RA/Dec and display to the user)?
in a fragment shader for the currently drawn pixel (I want to draw quad to shade the sky in a realistic way, for this I want to calculate pixel’s angular distance from the Sun, pixel’s height above the horizon etc, all in the shader code)?

Tags: 3d gl-matrix javascript projection webgl

Answers

Chosen as BEST ANSWER

Capitalizing on the answer by @LJᛃ, here are the implementations of the transforms explained in their answer, that cover both JavaScript and GLSL shader parts of the question.

Important: your projection matrix needs to have near-clipping distance NOT equal to 0 for this to work. Otherwise it is impossible to get the inverse of the projection matrix. A value of 0.1 did the trick in my case.

Part 1 of the question, that is the JavaScript side for a clicked point, using gl-matrix methods, would look like this:

// below:
// - `projectionMatrix` is a result of `mat4.perspective`
// with non-zero `near` argument
// - `viewMatrix` is a matrix of camera transforms
// - `viewport.x` and `viewport.y` are, respectively,
// width and height of your WebGL viewport

// 1. Get the inverse of projection matrix
const invertedProjection = mat4.clone(projectionMatrix)
mat4.invert(invertedProjection, projectionMatrix)
// 2. Get mouse coordinates relative to WebGL viewport
// important: pay attention to the difference between your
// WebGL viewport width, canvas.width and canvas.clientWidth,
// same for height, to calculate the correct pixel in the WebGL
// viewport.
// The code below assumes that WebGL viewport dimensions are
// the same as `canvas.width` and `canvas.height`.
const event = thisIsYourClickEventInsideTheListener()
const { top, left } = event.currentTarget.getBoundingClientRect()
const [mouseX, mouseY] = [event.clientX, event.clientY]
const [x, y] = [Math.floor(mouseX - left), Math.floor(mouseY - top)]
const canvas = getYourCanvasHtmlElementHere()
const pixelX = x * canvas.width / canvas.clientWidth
const pixelY = canvas.height - y * canvas.height / canvas.clientHeight - 1
// 3. Transform pixel coordinates back to NDC
const ndcX = (pixelX / viewport.x) * 2 - 1
const ndcY = (pixelY / viewport.y) * 2 - 1

// 4. Unproject NDC coordinates to camera space coordinates
const ndc1 = vec3.fromValues(ndcX, ndcY, -1)
const ndc2 = vec3.fromValues(ndcX, ndcY, 1)
const near = vec3.create()
const far = vec3.create()
vec3.transformMat4(near, ndc1, invertedProjection)
vec3.transformMat4(far, ndc2, invertedProjection)
// 5. Invert view matrix
const inverseView = mat4.create()
mat4.invert(inverseView, viewMatrix)
// 6. Go from camera space to world space
const worldNear = vec3.create()
const worldFar = vec3.create()
vec3.transformMat4(worldNear, near, inverseView)
vec3.transformMat4(worldFar, far, inverseView)
// 7. Go from line segment to ray
const result = vec3.clone(worldFar)
vec3.subtract(result, result, worldNear)
vec3.normalize(result, result)
// result is now X, Y, Z coordinates on the unit sphere.

Part 2 of the question, the fragment shader side, using gl_FragCoord, would look like this:

#version 300 es
// the WebGL viewport width and height
uniform vec2 u_viewport;
// the inverse of projection matrix as calculated above in JavaScript
uniform mat4 u_invProjectionMatrix;
// the inverse of view matrix as calculated above in JavaScript
uniform mat4 u_invModelViewMatrix;

out vec4 fragColor;

void main() {
  // 1. Transform pixel coordinates back to NDC
  vec2 ndcPixelCoord = vec2(
    gl_FragCoord.x / u_viewport.x * 2.0 - 1.0,
    gl_FragCoord.y / u_viewport.y * 2.0 - 1.0
  );
  // 2. Unproject NDC coordinates to camera space coordinates
  vec4 ndc1 = vec4(ndcPixelCoord.xy, -1, 1);
  vec4 ndc2 = vec4(ndcPixelCoord.xy, 1, 1);
  vec4 near = u_invProjectionMatrix * ndc1;
  vec4 far = u_invProjectionMatrix * ndc2;
  // 3. Go from camera space to world space
  vec4 worldNear = u_invModelViewMatrix * near;
  vec4 worldFar = u_invModelViewMatrix * far;
  // 4. Go from line segment to ray and apply perspective divide
  vec3 perspectiveDividedWorldNear = worldNear.xyz / worldNear.w;
  vec3 perspectiveDividedWorldFar =  worldFar.xyz / worldFar.w;
  vec3 result = normalize(perspectiveDividedWorldFar - perspectiveDividedWorldNear);
  // at this point, result is coordinates on the unit sphere.
  // in this example, we shift the cordinates from [-1, 1] range to [0, 1] range
  // and output as color
  fragColor = vec4(result.xyz / 2.0 + 0.5, 1.0);
}

(Edit)

- LJ
- May 30, 2024 at 3:45 pm
- 0 votes
0
To approach this we want to reverse the transform chain:

So the steps are:

Transform pixel coordinates back to NDC (normalized device coordinates)

Normalized device coordinate space is a 3d unit cube, so to unproject the screen pixel coordinates into that cube we can simply do this:
```
ndcX = (screenX / screenWidth) * 2 - 1
ndcY = (screenY / screenHeight) * 2 - 1
ndcZ = ???
```
in a matrix this would be expressed as:

Note: WebGL Y pixel coordinates are inversed compared to the browsers, meaning if you’re using the ClickEvents clientY/offsetY property you want to reverse it.

Unproject NDC coordinates to camera space coordinates

With the previous step we calculated the NDC coordinates on a 2D plane with unknown depth within the NDC space. To take it back to 3D we need to realize that a 2D point is the collapsed representation (a projection) of a ray in 3D space, for which we need at least two points on the ray, a line segment, to represent it. For the commonly employed projection matrices (like the ones generated via glMatrix’es perspective and ortho) we can simply choose the extents of the volume, giving us two coordinate sets:
```
ndc1 = [ndcX, ndcY, -1]
ndc2 = [ndcX, ndcY,  1]
```
Transforming those using the inverse of the projection matrix will give us the points on the near and far clipping plane in camera space.
```
near = ndc1 * inverseProjectionMatrix
far = ndc2 * inverseProjectionMatrix
```
Note: You want to use gl-matrix’es vec3.transformMat4 here

Go from camera space to world space

To undo the camera transforms we simply transform our two points using the inverse view matrix.
```
worldNear = near * inverseViewMatrix
worldFar = far * inverseViewMatrix
```
Note: Again, you want to use gl-matrix’es vec3.transformMat4 here

Go from line segment to ray

To get the point on the unit sphere we just convert our line segment to a ray representation, which is commonly expressed as a ray origin and a normalized ray direction, the latter being the point on the unit sphere you’re looking for. To do so we just subtract near from far and normalize the resulting vector.
```
rayOrigin = near
rayDirection = normalize(far-near)
```
Note that, in case you specify your camera position in world space, or you don’t translate the scene in any way (camera position being 0,0,0) you only need to do all of this for far, as rayOrigin will simply be cameraPosition.
Login or Signup to reply.