RayCaster returns an inconsistent point when the camera moves during the pick

[ShaMan123]

Describe the bug 📝

If the camera is moving while the castRay promise is resolving the RaycastResult.point will drift from the actual hit position.
Investigation has proved that the methods used by castRay use the mutable camera freely in an async block:

OBC.Raycasters → SimpleRaycaster.castRay → FastModelPicker.getFullPick

engine_components/packages/core/src/core/FastModelPicker/src/fast-model-picker.ts

Lines 286 to 307 in 5c2dd9d

	async getFullPick(position?: THREE.Vector2): Promise<{
	modelId: string;
	localId: number;
	/**
	* Internal item index (FlatBuffer `sample.item()`). Exposed so
	* SnapResolver and other internal consumers can hit fragments'
	* itemId-keyed fast paths (`boxes.sampleOf`) without paying a
	* second worker round-trip to translate back from localId.
	*/
	itemId: number;
	point: THREE.Vector3;
	normal: THREE.Vector3 | null;
	distance: number;
	} | null> {
	const item = await this.getItemAt(position);
	if (!item) return null;
	const point = await this.getPointAt(position);
	if (!point) return null;
	const normal = await this.getNormalAt(position);
	const distance = point.distanceTo(this.world.camera.three.position);
	return { ...item, point, normal, distance };
	}

Summary

getFullPick(position) resolves a pick over three sequential asynchronous passes, and each pass reads the live this.world.camera.three:

getItemAt   → renderPickPass (id)                       // item, from camera state A
            → await getLocalIdsFromItemIds(...)          // ← async boundary: a frame can run here
getPointAt  → renderWithTileMaterial (depth)             // point, from camera state B (> A)
            → unprojectToWorld(pos, depth, world.camera.three)   // unproject with state B
getNormalAt → renderWithTileMaterial (normal)
distance    = point.distanceTo(world.camera.three.position)      // state C

The GPU readbacks are synchronous, but getItemAt awaits an item-id resolution between the id pass and the depth pass. If the camera moves during that await (rapid wheel zoom, camera-controls damping/inertia — anything that advances the camera on the app's render loop), the passes render from different camera states:

• item is picked at camera state A;
• point is unprojected from the depth pass at a later state B;
• distance uses a later state still.

So the returned point is not consistent with the returned item, and it does not lie on the ray the caller cast — it's reprojected through the moved camera.

Impact

Any consumer that positions something at castRay().point while the camera is moving gets a wrong world position — the point lands off the cursor / off the picked surface. We hit this anchoring an orbit pivot at the pointer-down pick during a fast zoom: the anchor jumps off the clicked geometry. With a static camera everything is correct.

Root cause

getFullPick re-reads this.world.camera.three in every pass instead of latching one camera for the duration of the pick.

Expected

A single castRay / getFullPick call returns item, point, normal, distance all consistent with one camera state (the state at the start of the call), regardless of camera motion during the async passes.

Suggested Fix

This is a real caveat of design - state must be immutable during an async operation.
Does it occur in other parts of the code? It is a real design issue since frags are threaded.
I am not sure what the correct fix is.
Could getPointAt resolve in the same sync context as getItemAt? That would be the best fix. Are the method separate due to reusability of code or due to real reasons. I have not investigated any further.
The quick fix is to freeze the camera and pass it around so that it is immutable.

Reproduction ▶️

No response

Steps to reproduce 🔢

1. Load any model into a SimpleWorld with an OrthoPerspectiveCamera (default Raycasters / FastModelPicker).
2. Call raycaster.castRay({ position }) at a fixed position while continuously moving the camera (e.g. a wheel-zoom loop, or controls.dolly(...) each frame).
3. Compare result.point against the ray raycaster.three built from position and the pre-pick camera: result.point is off the ray, and the error grows with camera speed. Static camera → correct.

Deterministic variant: inside getFullPick, translate the camera by a known delta between the getItemAt and getPointAt awaits — result.point shifts by the corresponding reprojection, proving the passes use different camera states.

      const result = (await this.rayCaster.castRay({ position: ndc })) as
        | RaycastResult
        | undefined;
      // zoom or drag rapidly
      if(!result) return
      const { point } = result;
      const { controls } = this.world.camera;
      controls.setOrbitPoint(point.x, point.y, point.z); // drifts with the camera

System Info 💻

irrelevant

Used Package Manager 📦

npm

Error Trace/Logs 📃

No response

Validations ✅

• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a repository issue and not a framework-specific issue. For example, if it's a THREE.js related bug, it should likely be reported to mrdoob/threejs instead.
• Check that this is a concrete bug. For Q&A join our Community.
• The provided reproduction is a minimal reproducible example of the bug.

[ShaMan123]

The fix is involved.
The camera is not the only mutable object being passed around making #776 a workaround only for camera mutability.
This is related to #773.

[ShaMan123]

Reading the source code - I love the GPU picker, such a great idea.
I see it is capped by 8 bit length which in practice is more than enough.
The code could run a while loop (0...254, 255...) for code resilience and eliminate this by design but that sounds tricky (render order) => not worth it.

Investigating getLocalIdsFromItemIds which is the only async operation blocking the entire pass from being sync:
Could the localId be available on the mesh userData so that it can be encoded into the pixel?
Or could the itemId => localId map be available sync? I don't see caching.

engine_components/packages/core/src/core/FastModelPicker/src/fast-model-picker.ts

Lines 20 to 21 in 5c2dd9d

	* lookup; the first time we resolve a given `itemId` to a `localId`
	* we ask the worker, and afterwards that lookup is cached.

[ShaMan123]

Making the pass sync will be a major win - raycasting should be sync as it is used in pointer events, async making it awkward (need to handle complex down-move-up state)

[ShaMan123]

I will suggest 2 fixes:

1. The quick an simple fix is to reorder function calls to obtain all data from rendering up front and only then resolve item => localId
2. Turn the entire pass to sync - I see that to make ray casting fully sync _getItemSnapData needs to be reworked as well