A new kind of project

I did something new and fun this fall: I wrote a computer science paper. It's an analysis of the conditions under which a device equipped with a camera and an accelerometer can identify its position relative to two observed landmarks with known positions. Except for a measure zero set of singular cases with infinitely many solutions, there are always at most two solutions for device positions (this was previously known), and I found necessary and sufficient conditions for there to be a single solution. In particular, if the two landmarks are at the same altitude, there is always a single solution, unless the device is at the same altitude as the landmarks.

I implemented the algorithm on a phone (code here). In the screenshot, the markers 1 and 2 are landmarks, identified and outlined in green with OpenCV library code, and then the phone uses their positions and the accelerometer data to predict where the control markers 3 and 4 are on the screen, outlining them in red.

For someone like me who does some philosophy of science, it was an interesting experience to actually do a real experiment and collect data from it.

I am planning at some point to try to implement the algorithm using infrared LEDs under a TV and the accelerometer and infrared camera inside a right Nintendo Switch joycon. To that end, over the last couple of days I've reverse-engineered two of the joycon infrared camera blob identification modes.

An introduction to simple motion detection with Python and OpenCV

One of my kids really liked a cool thing in a children's museum where they had a camera pointed down a hallway, with a screen, and if you waved your hands in certain areas you got to play music. So I decided to make something like this myself. I found this tutorial and with its help produced some simple code. I then wrote up an Instructable explaining how the code works.

Color-remapping

I’ve often wondered what would happen if one wore color-remapping glasses. Would the brain adapt, and everything would soon start looking normal, much as it does when you wear vision-inverting prisms. Well, it turns out that there is an interesting two-subject (the investigators) study using LCD glasses linked to a camera and did a color-space rotation. They found interesting results, but found that over six days there was no adaptation: stop signs still looked blue, the sky still looked green, and broccoli was red. This is not conclusive, since six days might just not be long enough. I would love to see the results of a longer study.

The philosophical relevance, of course, is to inverted-spectrum thought experiments.

Cambridge events and objects

Suppose Joe Shmoe died on February 17, 1982, sadly leaving no relatives or friends behind. Every year, on February 17, the anniversary of Shmoe's death occurs. No one marks it in any way. But it occurs, every year, invariably. It is what one might call a Cambridge event, whose occurrence does not mark any real change in the world.

Similarly, there seem to be Cambridge objects. Just as the anniversary is defined by a certain temporal distance, we can define an object by a certain spatial distance. For instance, let me introduce an object: my visual focus. My visual focus is a moving object a certain distance in front of my eyes--sometimes moving very fast (in principle, a visual focus could move faster than light!). My visual focus is a persisting object, unless I close my eyes (I am not sure whether it persists when I blink or just blinks out of existence). Curiously, my visual focus, while typically having a spatiotemporal location, could also exist outside of spacetime. Imagine that I am focused a meter ahead of my nose, and space has an edge. I walk towards that edge, unblinking and never refocusing, rapt in thought about ontology. Before my nose touches the edge of space, my visual focus will have moved beyond it! We can say that the visual focus is "a meter ahead of my face", but that isn't an actual place. So we cannot identify the visual focus with a whole made up of spacetime locations.

My brief remarks have taught you, I think, a little bit about how to talk about visual foci. You now know roughly when my, or your, visual focus exists. You know something about its persistence conditions. You know a little bit about what predicates apply to it. And there is a vast range of stuff that's as yet underdetermined, and could be determined in more than one way. For instance, how wide is the visual focus? Does it shift very quickly with saccades?

But of course it's also clear that there has to be a sense in which there really are no visual foci. Objects that can leave our spacetime so easily, that can move faster than light, and that are entirely outside us but are entirely grounded in our state just aren't really there. They are Cambridge objects instead of real ones, akin to Cambridge events, Cambridge properties and Cambridge changes.

This post is inspired by John Giannini's dissertation.

Retinal images

Here's a nice way to get an idea just how much processing intervenes between retinal images and conscious perception. Unless you're reading this post through a screen reader or some other assistive technology, while you're reading the post, your eyes are moving in very fast saccades. So your retinal images are a mess. They are like a video produced by a camera that is waving around apparently randomly at high speed. But screen and the text on it looks to you as if it was basically still. You do not see the mess of retinal images with different orientations—they are all stitched together in complex post-processing into an image of something unmoving. Now, walk around while looking at some three solid and unmoving dimensional object in your environment. In addition to the shifts in retinal image produced by the eye motion, there are shifts due to head motion, due to the up-and-down rhythm of your walk, as well as the exposures of hidden surfaces. Yet the solid and unmoving object looks unmoving.

Of course, there are times when the post-processing fails—there are illusions of motion. For instance, because there is a background behind the laptop screen, and parallax ensures apparent relative movement of the laptop and the background, when I move my head, it does look like the laptop is moving relative to the background.

In any case, what we see does not look like what you get from a head mounted camera. And our retinal images are even jumpier than that, as the head mounted camera does not follow the eye-muscle saccades. We have some really quite amazing image stabilization.