Picking
Make meshes, sprites and UI respond to the mouse and touch with pointer events such as PointerClick and PointerDrag, observed on the entity.
On this page
Picking answers "which entity is under this pointer?" for every mouse and finger, and turns the answer into events on that entity: hovered, pressed, clicked, dragged, dropped. You attach observers to the entities you care about, the same way for a 3D mesh, a sprite or a UI node.
Click a cube#
use neo::prelude::*;
fn main() {
App::new()
// mesh picking isn't one of the default plugins
.add_plugins((DefaultPlugins, MeshPickingPlugin))
.add_systems(Startup, setup)
.run();
}
fn setup(
mut commands: Commands,
mut meshes: ResMut<Assets<Mesh>>,
mut materials: ResMut<Assets<StandardMaterial>>,
) {
commands.spawn((Camera3d::default(), Transform::from_xyz(0.0, 3.0, 6.0).looking_at(Vec3::ZERO, Vec3::Y)));
commands.spawn((PointLight::default(), Transform::from_xyz(3.0, 6.0, 3.0)));
commands
.spawn((
Mesh3d(meshes.add(Cuboid::new(1.0, 1.0, 1.0))),
MeshMaterial3d(materials.add(Color::srgb_u8(124, 144, 255))),
))
.observe(|click: On<PointerClick>| info!("clicked {} with {:?}", click.entity, click.button))
.observe(|drag: On<PointerDrag>, mut transforms: Query<&mut Transform>| {
if let Ok(mut t) = transforms.get_mut(drag.entity) {
t.rotate_y(drag.delta.x * 0.02);
}
});
}The pieces#
| Piece | What it does |
|---|---|
| Pointers | The mouse (PointerId::Mouse) and each finger (PointerId::Touch(id)). Custom pointers are possible too. |
| Backends | Hit tests that say what each pointer is over. UI and sprite picking are added by the UI and sprite plugins. Mesh picking needs MeshPickingPlugin, which you add. |
| Hover and events | The engine sorts the hits by camera order and depth, works out what's hovered, and triggers pointer events. |
| Your observers | .observe(...) on an entity, or a global add_observer, reacts to the events. |
NEO's builds include all three backends (mesh, sprite and UI picking) on every platform, including the web.
Pointer events#
Every event has entity (the target) and pointer (its id, render target and position in the window). Most also carry hit, the HitData with the camera, depth, and the world position and normal where the backend knows them.
| Group | Events |
|---|---|
| Hover | PointerOver, PointerOut (the pointer moved onto or off this entity), PointerEnter, PointerLeave (like the web's mouseenter: sent to the entity and the ancestors it newly entered), PointerMove. |
| Press | PointerPress, PointerRelease, PointerClick (with button, duration and count for double clicks), PointerCancel. |
| Drag | PointerDragStart, PointerDrag (with delta and distance in screen pixels), PointerDragEnd. |
| Drop | PointerDragEnter, PointerDragOver, PointerDragLeave, PointerDragDrop on the entity under the dragged one. |
| Scroll | PointerScroll with the wheel's x, y and unit. |
Bubbling#
Pointer events bubble: an event on a child is then sent to its parent, then the parent's parent, up to the window. That's what lets one observer on a UI panel handle clicks on all its children. Stop it in an observer with event.propagate(false):
commands.spawn(Node::default()).observe(|mut click: On<PointerClick>| {
click.propagate(false); // the parents don't get this click
});Pointer events are also messages, so a system can read them with MessageReader<PointerClick> instead of observing. See Observers for how observers and propagation work.
Choosing what can be picked#
The Pickable component controls an entity:
| Field | Default | What it does |
|---|---|---|
should_block_lower | true | Stops the entities behind this one from being hovered. Set false for a see-through overlay. |
is_hoverable | true | Whether this entity can be hovered and get events at all. |
Pickable::IGNORE sets both to false: the pointer goes straight through. NEO's touch controls use it so the on-screen stick never blocks the game.
By default mesh picking tests every mesh, which is handy while you build. For a large scene, make it opt-in: insert MeshPickingSettings { require_markers: true, ..default() }, then add Pickable to the meshes and MeshPickingCamera to the cameras that should pick. Sprites test the opaque pixels of the sprite by default; SpritePickingSettings can switch to the bounding box.
Hover and press state#
To style something by its state instead of handling events, use these components:
| Component | What it holds |
|---|---|
Hovered | Insert it on an entity: Hovered(true) while a pointer is over it or its children. It only changes on enter and leave, so Changed<Hovered> is cheap. |
PickingInteraction | None, Hovered or Pressed, across all pointers. The engine inserts it on entities that are hovered. |
UI buttons also have Interaction; see Interaction.
Ray casts#
For gameplay hit tests that aren't tied to a pointer, such as a laser or line of sight, use the MeshRayCast system parameter:
fn laser(mut ray_cast: MeshRayCast, mut gizmos: Gizmos) {
let ray = Ray3d::new(Vec3::new(0.0, 1.0, 5.0), Dir3::NEG_Z);
if let Some((entity, hit)) = ray_cast.cast_ray(ray, &MeshRayCastSettings::default()).first() {
info!("hit {entity} at {}", hit.point);
gizmos.sphere(hit.point, 0.05, Color::WHITE);
}
}cast_ray returns every hit, nearest first. MeshRayCastSettings filters by visibility and by entity. NEO's character games use the character kit's collision, or the physics world and its ray casts, for gameplay; ray casts against render meshes suit tools and simple games.
Debugging#
DebugPickingPlugin shows what each pointer hits and logs events, but it's part of the developer tools, which aren't in NEO's default builds; see Developer tools.
Examples: cargo run --example simple_picking, mesh_picking, sprite_picking, dragdrop_picking, draggable_slider and custom_hit_data.