My first (and last?) computer science paper

Some years back I was searching for a better method to play lightgun games like Duck Hunt under emulation by using a Wii remote as a light gun.

The standard Wii remote way of pointing at a TV with a sensor bar underneath is not precise enough. The Wii remote has an infrared camera that tracks the LEDs in the sensor bar, and uses that to estimate where the remote is pointing. But since all the LEDs in the sensor bars lie in a single line, there is no way to determine from the camera image exactly how the Wii remote points. (It’s easy to visualize how rotating the Wii remote about the axis defined by the sensor bar won’t change the camera image in it.) The Wii just guesses, but that’s not good enough for lightgun games.

I already had a way that involved putting four infrared LEDs around my TV, but I was wondering if one could do this with a standard sensor bar by leveraging the additional information from the accelerometer in the Wii remote. A web search came up with a paper by d’Alfonso et al. on determining camera pose using two landmarks and accelerometer. The paper claimed a theorem that, except in certain singular cases, one could determine the pose up to a two-option ambiguity—i.e., there will be two camera orientation/position setups that would result in the same landmark images and accelerometer data.

After thinking about the paper, I realized that the claimed theorem was incorrect: there were singular cases the authors didn’t know about, but more interestingly I discovered cases where one could determine camera pose from two landmark images and accelerometer data without any ambiguity. Moreover, one of these cases was precisely the one relevant to me: the two landmarks are on the same horizontal line (the Wii sensor bar supplies precisely that) and the camera is above or below the plane of that line (easy to arrange). Sadly, experiments showed that Wii remote accelerometer precision was insufficient to determine camera pose in this way.

Anyway, I don’t like the idea of there being a published “theorem” that is incorrect. At some point I worked out all the details with trigonometry, and found a precise characterization of the exact conditions under which in theory (i.e., given a sufficiently precise accelerometer, unlike the one in the Wii remote) one could determine camera pose from two landmark images and gravity data with no ambiguity, with an ambiguity of two, or with an ambiguity of infinity (in some singular cases).

I’ve just got published a paper with all the details.

Sadly, as noted, the Wii remote application didn’t pan out. However, I did write some proof of concept code for my Pixel 7 Pro, and found that it worked pretty decently with its accelerometer. Here’s a screenshot where the accelerometer data plus the screen positions of the (centers of) the bottom two Aruco markers, together with data about the camera and the dimensions of my display board, is used to calculate the screen positions of the upper two Aruco markers (the calculated positions are marked in red).

I kind of suspect that the accelerometer in the right-hand Nintendo Switch joycon might be sufficiently precise for use as a lightgun with two infrared LEDs for positioning, but I haven’t had time to implement this. (Part of the problem is that the infrared LEDs that I have at home are the wrong wavelength for the joycon.)

While this may be my last computer science paper, if some computer science grad student or the like is interested in working with me, we could do some empirical work on seeing how applicable the algorithm is.