My first (and last?) computer science paper

Some years back I was searching for a better method to play lightgun games like Duck Hunt under emulation by using a Wii remote as a light gun.

The standard Wii remote way of pointing at a TV with a sensor bar underneath is not precise enough. The Wii remote has an infrared camera that tracks the LEDs in the sensor bar, and uses that to estimate where the remote is pointing. But since all the LEDs in the sensor bars lie in a single line, there is no way to determine from the camera image exactly how the Wii remote points. (It’s easy to visualize how rotating the Wii remote about the axis defined by the sensor bar won’t change the camera image in it.) The Wii just guesses, but that’s not good enough for lightgun games.

I already had a way that involved putting four infrared LEDs around my TV, but I was wondering if one could do this with a standard sensor bar by leveraging the additional information from the accelerometer in the Wii remote. A web search came up with a paper by d’Alfonso et al. on determining camera pose using two landmarks and accelerometer. The paper claimed a theorem that, except in certain singular cases, one could determine the pose up to a two-option ambiguity—i.e., there will be two camera orientation/position setups that would result in the same landmark images and accelerometer data.

After thinking about the paper, I realized that the claimed theorem was incorrect: there were singular cases the authors didn’t know about, but more interestingly I discovered cases where one could determine camera pose from two landmark images and accelerometer data without any ambiguity. Moreover, one of these cases was precisely the one relevant to me: the two landmarks are on the same horizontal line (the Wii sensor bar supplies precisely that) and the camera is above or below the plane of that line (easy to arrange). Sadly, experiments showed that Wii remote accelerometer precision was insufficient to determine camera pose in this way.

Anyway, I don’t like the idea of there being a published “theorem” that is incorrect. At some point I worked out all the details with trigonometry, and found a precise characterization of the exact conditions under which in theory (i.e., given a sufficiently precise accelerometer, unlike the one in the Wii remote) one could determine camera pose from two landmark images and gravity data with no ambiguity, with an ambiguity of two, or with an ambiguity of infinity (in some singular cases).

I just got a paper with all the details published.

Sadly, as noted, the Wii remote application didn’t pan out. However, I did write some proof of concept code for my Pixel 7 Pro, and found that it worked pretty decently with its accelerometer. Here’s a screenshot where the accelerometer data plus the screen positions of the (centers of) the bottom two Aruco markers, together with data about the camera and the dimensions of my display board, is used to calculate the screen positions of the upper two Aruco markers (the calculated positions are marked in red).

I kind of suspect that the accelerometer in the right-hand Nintendo Switch joycon might be sufficiently precise for use as a lightgun with two infrared LEDs for positioning, but I haven’t had time to implement this. (Part of the problem is that the infrared LEDs that I have at home are the wrong wavelength for the joycon.)

While this may be my last computer science paper, if some computer science grad student or the like is interested in working with me, we could do some empirical work on seeing how applicable the algorithm is.

A new kind of project

I did something new and fun this fall: I wrote a computer science paper. It's an analysis of the conditions under which a device equipped with a camera and an accelerometer can identify its position relative to two observed landmarks with known positions. Except for a measure zero set of singular cases with infinitely many solutions, there are always at most two solutions for device positions (this was previously known), and I found necessary and sufficient conditions for there to be a single solution. In particular, if the two landmarks are at the same altitude, there is always a single solution, unless the device is at the same altitude as the landmarks.

I implemented the algorithm on a phone (code here). In the screenshot, the markers 1 and 2 are landmarks, identified and outlined in green with OpenCV library code, and then the phone uses their positions and the accelerometer data to predict where the control markers 3 and 4 are on the screen, outlining them in red.

For someone like me who does some philosophy of science, it was an interesting experience to actually do a real experiment and collect data from it.

I am planning at some point to try to implement the algorithm using infrared LEDs under a TV and the accelerometer and infrared camera inside a right Nintendo Switch joycon. To that end, over the last couple of days I've reverse-engineered two of the joycon infrared camera blob identification modes.

An introduction to simple motion detection with Python and OpenCV

One of my kids really liked a cool thing in a children's museum where they had a camera pointed down a hallway, with a screen, and if you waved your hands in certain areas you got to play music. So I decided to make something like this myself. I found this tutorial and with its help produced some simple code. I then wrote up an Instructable explaining how the code works.

Color-remapping

I’ve often wondered what would happen if one wore color-remapping glasses. Would the brain adapt, and everything would soon start looking normal, much as it does when you wear vision-inverting prisms. Well, it turns out that there is an interesting two-subject (the investigators) study using LCD glasses linked to a camera and did a color-space rotation. They found interesting results, but found that over six days there was no adaptation: stop signs still looked blue, the sky still looked green, and broccoli was red. This is not conclusive, since six days might just not be long enough. I would love to see the results of a longer study.

The philosophical relevance, of course, is to inverted-spectrum thought experiments.

Cambridge events and objects

Suppose Joe Shmoe died on February 17, 1982, sadly leaving no relatives or friends behind. Every year, on February 17, the anniversary of Shmoe's death occurs. No one marks it in any way. But it occurs, every year, invariably. It is what one might call a Cambridge event, whose occurrence does not mark any real change in the world.

Similarly, there seem to be Cambridge objects. Just as the anniversary is defined by a certain temporal distance, we can define an object by a certain spatial distance. For instance, let me introduce an object: my visual focus. My visual focus is a moving object a certain distance in front of my eyes--sometimes moving very fast (in principle, a visual focus could move faster than light!). My visual focus is a persisting object, unless I close my eyes (I am not sure whether it persists when I blink or just blinks out of existence). Curiously, my visual focus, while typically having a spatiotemporal location, could also exist outside of spacetime. Imagine that I am focused a meter ahead of my nose, and space has an edge. I walk towards that edge, unblinking and never refocusing, rapt in thought about ontology. Before my nose touches the edge of space, my visual focus will have moved beyond it! We can say that the visual focus is "a meter ahead of my face", but that isn't an actual place. So we cannot identify the visual focus with a whole made up of spacetime locations.

My brief remarks have taught you, I think, a little bit about how to talk about visual foci. You now know roughly when my, or your, visual focus exists. You know something about its persistence conditions. You know a little bit about what predicates apply to it. And there is a vast range of stuff that's as yet underdetermined, and could be determined in more than one way. For instance, how wide is the visual focus? Does it shift very quickly with saccades?

But of course it's also clear that there has to be a sense in which there really are no visual foci. Objects that can leave our spacetime so easily, that can move faster than light, and that are entirely outside us but are entirely grounded in our state just aren't really there. They are Cambridge objects instead of real ones, akin to Cambridge events, Cambridge properties and Cambridge changes.

This post is inspired by John Giannini's dissertation.

Retinal images

Here's a nice way to get an idea just how much processing intervenes between retinal images and conscious perception. Unless you're reading this post through a screen reader or some other assistive technology, while you're reading the post, your eyes are moving in very fast saccades. So your retinal images are a mess. They are like a video produced by a camera that is waving around apparently randomly at high speed. But screen and the text on it looks to you as if it was basically still. You do not see the mess of retinal images with different orientations—they are all stitched together in complex post-processing into an image of something unmoving. Now, walk around while looking at some three solid and unmoving dimensional object in your environment. In addition to the shifts in retinal image produced by the eye motion, there are shifts due to head motion, due to the up-and-down rhythm of your walk, as well as the exposures of hidden surfaces. Yet the solid and unmoving object looks unmoving.

Of course, there are times when the post-processing fails—there are illusions of motion. For instance, because there is a background behind the laptop screen, and parallax ensures apparent relative movement of the laptop and the background, when I move my head, it does look like the laptop is moving relative to the background.

In any case, what we see does not look like what you get from a head mounted camera. And our retinal images are even jumpier than that, as the head mounted camera does not follow the eye-muscle saccades. We have some really quite amazing image stabilization.