Unity Custom Editors: dragging handles in 3D

Here’s two quick tips for people writing custom 3D editor controls (something you should be doing a lot of in Unity, because it’s very good at it, and it saves a lot of dev time and design time!)

Find the 3D point on a plane where the user’s mouse clicked, in Editor mode

Bizarrely, Unity doesn’t give you access to their core libraries for interpreting mouse position when you’re in Editor mode. They only exist for “play” mode. Hmm. Oh well…

So, one of the common questions is:

Find the 3D position on an object corresponding to where the mouse is currently positioned on screen

Most of the source code I’ve seen on the Unity3D forums, and on the web in general, for calculating this is wrong.

There’s a couple of problems you have to solve, not least: Unity’s mouse position is reported upside-down in the Editor. But do NOT use “Screen.height” to reverse it – Screen.height is NOT the height of your visual screen!. Instead, you want “Camera.pixelHeight” (read the Camera docs carefully, and you’ll eventually find the bit that explains that your visual screen is defined as Camera.pixelWidth x Camera.pixelHeight).

[csharp]
/** calculate the point in 3D where the mouse clicked,
using the plane of the "floor" of your building (assuming your 3D arrow
is on the floor */
Ray rayCameraThroughMouse = editorCamera.ScreenPointToRay( new Vector2(e.mousePosition.x, (editorCamera.pixelHeight – e.mousePosition.y)) );

Plane planeOfFloor = new Plane( myObject.transform.up, myObject.transform.position );
float distanceAlongRay;
planeOfFloor.Raycast( rayCameraThroughMouse, out distanceAlongRay );
Vector3 worldPointForMousePositon = rayCameraThroughMouse.GetPoint( distanceAlongRay );
[/csharp]

Trick 2: If you want an Arrow that’s draggable in 3D, but only along it’s direction

NB: if you don’t know how to make draggable, interactive, 3D editors in Unity, this isn’t the solution; this is only a small part of what you need to do. Most of the rest is undocumented. If I get time to write-up the missing docs myself, I’ll replace this paragraph with a link

If you do this by hand (as I used to do), your code is something like this:

[csharp]
/**
* This code works great – and it works outside the Editor – but it’s long-winded
*/
/** 3d position of arrow start */
Vector3 arrowBase = myObject.transform.position + myObject.transform.localRotation * myComponent.localOffset;
/** 3d position of arrow tip */
Vector3 worldCapPosition = arrowBase + myObject.transform.localRotation * myComponent.localOutDirection;
/** 3d direction of arrow */
Vector3 worldOutDirection = myObject.transform.localRotation * myComponent.localOutDirection;

/** calculate the point in 3D where the mouse clicked,
using the plane of the "floor" of your building (assuming your 3D arrow
is on the floor */
Ray rayCameraThroughMouse = editorCamera.ScreenPointToRay( new Vector2(e.mousePosition.x, (editorCamera.pixelHeight – e.mousePosition.y)) );
Plane planeOfFloor = new Plane( myObject.transform.up, myObject.transform.position );
float distanceAlongRay;
planeOfFloor.Raycast( rayCameraThroughMouse, out distanceAlongRay );
Vector3 worldPointForMousePositon = rayCameraThroughMouse.GetPoint( distanceAlongRay );

/** Find the distance along the 3D line that the mouse "latest" position equates to,
compared to its initial position */
Vector3 localOffsetInWorld = worldPointForMousePositon – worldCapPosition;
float distance = Vector3.Dot( localOffsetInWorld, worldOutDirection );

/** add this to current handle’s position to move it on-screen */
Vector3 currentHandleDragOffset = distance * worldOutDirection;
[/csharp]

But it turns out you can throw away more than half of that code, and get something much simpler, easy to read (and less likely to have bugs in it!). Unity has a built-in method for doing this that seems to work well. The official Unity docs are no help here:

Helper function for doing arrows.

(@Unity team: if you’re not going to write something useful, why bother writing it at all?)

…but reading between the lines, this method was invented to solve the exact problem outlined above. Look how much shorter the code is:

[csharp]
/**
* This code is much cleaner/shorter … but you can’t use this in your game,
* Unity won’t allow you to (it’s Editor-only). Seems a bit short-sighted?
*/
/** 3d position of arrow start */
Vector3 arrowBase = myObject.transform.position + myObject.transform.localRotation * myComponent.localOffset;
/** 3d position of arrow tip */
Vector3 worldCapPosition = arrowBase + myObject.transform.localRotation * myComponent.localOutDirection;
/** 3d direction of arrow */
Vector3 worldOutDirection = myObject.transform.localRotation * myComponent.localOutDirection;

/** Let Unity’s one-liner do all the hard work for you */
float distance = HandleUtility.CalcLineTranslation( currentDragStartPoint, e.mousePosition, arrowBase, worldOutDirection );

/** add this to current handle’s position to move it on-screen */
Vector3 currentHandleDragOffset = distance * worldOutDirection;
[/csharp]