Cursor doesn't interact with Unity UI using input system and controller

hedgefield

Hey there,

I've recently switched an AC project over to the new Unity input system, and it work perfectly, except that I can't use the controller to interact with Unity UI menus.

Here are the project specs:

• AC 1.78.4, Unity 2022.3
• AC Input system package used
• Interaction method is point and click, input method is mouse and keyboard + keyboard or controller (the setting dynamically switches depending on which input method is active)
• Menu's are set to NOT be directly navigable (so there's always a cursor)
• Cursor is set to use a Unity UI prefab. Same problem occurs when set to Software mode. Cursor doesn't move at all when set to Hardware mode and using a controller

I have an inventory UI that is set to display during gameplay, and a pause menu that, well, pauses the game. Neither responds to the controller's 'clicks', nor does it trigger a hover state (however, when hovering over an inventory item it does highlight it and show the hotspot UI). Walking around and using hotspots is no problem with the controller. UI responds fine to mouse clicks.

EDIT: Important addendum; in a build the mouse input also has the same problem!

I made a video showing the problem. You can see when it switches to controller when the icons change. https://www.dropbox.com/scl/fi/phc8xorj0sgqyn8ji8vjy/cursor_trouble.mp4?rlkey=xs62f46h9mgl7ux2680ysrus6&dl=0

Let me know if more info is needed to diagnose where the issue is.

ChrisIceBox

If you're using a gamepad but not direct-navigation of Menus, you're relying on a simulated cursor. In my experience, Unity's Input System doesn't play so nicely with this and menu selection.

To get around this, the Input System integration has a "Cursor Position" processor that you can attach to a Vector2 input. Try defining one, and then listing that in the Input System UI Input Module's "Point" variable, available on the Event System.

hedgefield

Hey Chris, thanks for the suggestion, makes sense that it would be something to do with simulated cursor vs hardware cursor. I added the AC cursor position processor to the Point action, but it didn't quite fix the problem.

On the one hand, the mouse cursor now works on UI again in the build, but the gamepad cursor is still non-functional over UI. It appears to be separate from the mouse cursor, too; if for example I put the mouse cursor over the menu button in the top right, and then switch to the gamepad, the cursor jumps to the center of the screen. If I then move it somewhere, then change to the mouse again, the cursor jumps back to the last place that I left it in (over that menu button). It's like they're tracked individually.

I'm reading up on the input system's VirtualMouseInput class and trying the gamepad cursor example that Unity made, will report back if I find something in there that works. Is that what the AC cursor processor hooks into aswell?

ChrisIceBox

No - the processor reads AC's own GetMousePosition function, which will rely on the CursorHorizontal/Vertial inputs used to move the simulated cursor.

It would be good to have another set of eyes on this - certainly if you make progress on your end, please do let me know.

hedgefield

Quick report from the frontline: VirtualMouseInput is interesting, but it seems to rely on the cursor being in the same Canvas as the UI you want to manipulate. CodeMonkey has a good video on how it works. But I wasn't able to figure out how to apply it to the AC cursor, even though in terms of setup it seems to be pretty similar.

Something else that the Input System offers is WarpCursorPosition. That function allows you to move the actual system cursor. I wrote a little script to keep it synced to where the AC cursor is at. This is when using a Unity UI prefab as cursor, and I attached this script to the same object that the canvas component is on (root of the cursor prefab):

using UnityEngine;
using UnityEngine.InputSystem;

public class GamepadCursor : MonoBehaviour
{
    private PlayerInput playerInput;
    [SerializeField] private RectTransform cursorTransform;

    private void Start() {
        playerInput = GameObject.Find("ControlsReader").GetComponent<PlayerInput>();
    }

    private void Update() {
        //Debug.Log(playerInput.currentControlScheme);
        if (playerInput.currentControlScheme == "Gamepad") {
            Mouse.current.WarpCursorPosition(cursorTransform.position);
        }
    }
}

This works pretty well, but unfortunately warping the cursor also counts as mouse-based input, so the input system keeps flicking back and forth between mouse and gamepad input 😅 so I haven't gotten it to actually 'click' a button or inventory item yet...

ChrisIceBox

CodeMonkey has a good video on how it works.

Thanks, I'll take a look. It should be that the provided processor is enough, but I need to refresh myself on the exact way to set up. Either way, it's not something Input System makes the easiest to handle.

unfortunately warping the cursor also counts as mouse-based input

The other issue here is that it'd only be compatible with platforms that support the mouse. Though AC doesn't officially support consoles, the best solution would be one that bypasses the mouse entirely.

Stand by - will report back.

ChrisIceBox

OK. So the Cursor Position processor works - but only on frames that actually recieve input. For example, if you assign mouse input to your "Point" input action, with the processor attached to the action, you can move the simulated cursor over to a UI element nudge the mouse, and it'll select.

Not ideal. So far, I haven't found a way to get the processor to be read for each frame, rather than only on frames where input is actually detected.

A separate approach is to force the Input System's internal Mouse to AC's cursor position, which can be done in an Update loop, i.e.:

using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.InputSystem.LowLevel;
using AC;

public class MouseTest : MonoBehaviour
{
    void Update ()
    {
        InputState.Change (Mouse.current.position, KickStarter.playerInput.GetMousePosition ());
    }
}

This issue here is that - while UI elements become selected, they don't seem to be interactable when clicked. Do you get a similar behaviour on your end?

hedgefield

Can confirm, with the processor on the Point action, and your script in the scene, at least the cursor doesn't jump to another position when you nudge the mouse anymore, and after the nudge the UI is indeed hovered. But also as soon as I try to use the gamepad submit button it doesn't respond, like you say, because it goes back to the gamepad input state. But I think this may be partially due to a different 'bug' in Unity that I've encountered before.

See, the mouse cursor triggers a UI's hovered state, but gamepad navigation triggers its selected state. I don't understand why Unity didn't merge those, because since hovered only exists for mouse input, you can have one element hovered and another selected, or like in this case, nothing selected.

I wrote a script to fix that, which you have to apply to each button/slider/checkbox etc.

using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.UI;

[RequireComponent(typeof(Selectable))]
public class UIHoverToSelectFix : MonoBehaviour, IPointerEnterHandler, IPointerExitHandler, IDeselectHandler
{
    public void OnPointerEnter(PointerEventData eventData)
    {
        if (!EventSystem.current.alreadySelecting) EventSystem.current.SetSelectedGameObject(this.gameObject);
    }

    public void OnPointerExit(PointerEventData eventData)
    {
        // Deselecting on pointer exit is nicer for mouse users, but with this commented out, nothing is ever deselected, which is very beneficial for gamepad users.
        //EventSystem.current.SetSelectedGameObject(null);
    }

    public void OnDeselect(BaseEventData eventData)
    {
        this.GetComponent<Selectable>().OnPointerExit(null);
    }
}

I tried it here, and this is the result:

• moved gamepad over UI button
• nudged mouse to trigger hover
• script converts it into a select
• press A button (submit) and the UI interaction fires

It's a big hack, but maybe it gives more insight into what's going on here. Still, can't believe the unity input system doesn't have a nicer (more obvious) way of solving this.

ChrisIceBox

Indeed, it's a system with a lot of quirks.

This alternative to MouseTest should do away with the need for a separate component attached to each Button:

using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.InputSystem.LowLevel;
using UnityEngine.EventSystems;
using AC;

public class MouseTest : MonoBehaviour
{
    void Update()
    {
        InputState.Change (Mouse.current.position, KickStarter.playerInput.GetMousePosition ());
    }

    void OnEnable () { EventManager.OnMouseOverMenu += OnMouseOverMenu; }
    void OnDisable () { EventManager.OnMouseOverMenu -= OnMouseOverMenu; }

    void OnMouseOverMenu (Menu menu, MenuElement element, int slot)
    {
        if (element != null)
            EventSystem.current.SetSelectedGameObject(element.GetObjectToSelect (slot));
    }
}

hedgefield

Thanks, that's a great simplification

There's probably some way to connect these wires, but I'm not good enough at coding to get 100% there, so I've chosen to resolve it a different way: when I detect that the input method switches to gamepad, I turn on the direct-navigate UI settings via the Kickstarter. And for my during-gameplay UI I linked its elements to input actions so they can also be operated without needing to click on them. Works pretty well. Thanks for thinking along!