Sometimes in game development (and in development at large) you spend a lot of time on a task that's less important because you think it should be simple. We ran into one of those issues lately involving an input method for our nearly-finished remastering of "(I Fell in Love With) The Majesty of Colors."
This one gets a bit technical.
The Problem
We both use Microsoft Surface Pro tablet/laptops. They're very nice machines that support a bunch of pointing devices: touchpad, mouse, touchscreen, and a pressure-sensitive pen. We noticed during testing of "The Majesty of Colors" that the Surface pen wasn't working to control the monster's tentacle. This was weird, because Unity usually handles this sort of input stuff automatically. We made a task in our issue-tracking system to fix the issue, figuring it would be simple.
The Red Herring
Unity is a nice piece of software, but its support community isn't great. Because it's free and easy to get started with, and because it lets you make games, a lot of the people asking for help aren't very experienced with software development. You get folks asking very basic questions, folks responding with solutions that are inelegant or risky, and often no sense of whether a solution is still valid several years and major Unity releases later.
After a bunch of searching, we found a promising response from Zaeran on the Unity Answers site:
It's a problem with Microsoft's Pen APIs not being included in standalone builds.
If you build to a Universal Windows 10 app, the pen will work fine. The other option is to have a background app running to tell unity when the pen touches the screen
A Universal Windows 10 app is one that uses Microsoft's UWP framework, which lets you build one app for devices ranging from desktop PCs to Windows Phones to the Hololens. It's a cool thing, but because it supports some tiny devices, it doesn't include the full set of .NET libraries.
Notably, when we tried to implement it, we discovered that the UWP framework set doesn't include some serialization libraries that are being used by a library we are using. If we want to release a UWP app, we need to not only replace our own serialization but also give up our existing error tracking library. This didn't seem like a good tradeoff, so we turned to the community for help.
A Clue From a Stranger
We've had good success with the Game Development Stack Exchange, so Gregory made a question summarizing the issue. Unfortunately, these things don't always get an immediate, clear answer, especially with a relatively niche issue like this one. Eventually, though, we did get a partial answer.
An anonymous user noted that they observed the Surface Pen's position being correctly tracked in Unity. It turns out this is working for us as well! We didn't notice because we're still not seeing the pen's taps being tracked. If you pretend the pen is a mouse, you could say we can tell when you move the mouse but not when you click. Clicking is important to us, so we don't yet have a solution. But this narrows the scope of the problem!
An Ongoing Search
A recurring theme that came up in responses to this issue was the idea that pen APIs weren't being included or made available to Unity. Various workarounds were proposed, but the main issue for us is that it's totally unclear which APIs specifically were missing.
Is the pen handled by the Microsoft Surface SDK? No, it turns out that's for the old tabletop device that's now called PixelSense. Does this article on Stylus Input apply, despite being filed under "Legacy User Interaction Features?" On the other hand, the pen is semi-secretly a Wacom device; should we be using their APIs? And why is any of this necessary when the pen appears to work with most Windows applications seamlessly?
A Bit of Perspective
In the end, we took a step back. We had put in hours of work to try and fix an issue that was showing up on a single device, albeit one we're fond of. And that device is one which has other native pointing methods. If a player with a Surface discovered their pen didn't work, they'd probably shrug and use touch instead.
In fact, part of our research revealed that this is a common enough issue in many major games. It's certainly not all of them; many games seem to work fine with the pen. But it's likely that there's something about the exclusive way that some games use Windows fails to hook into that aspect of the device.
For now we're tabling the issue. If we have some revelation or hint that lets us fix the issue quickly, we'll jump on it. Otherwise, we're okay with leaving this issue as something to fix post-release, if at all. We'll be sad that we can't use our pens in "The Majesty of Colors" on our Surfaces, but touch works so well that we won't be too disappointed.
If you have any ideas, comment on this post! And if you have a magical answer for us, toss it up on Stack Exchange and we'll be happy to give you some rep.
View on YouTube
