Our backyard had been free of black cats for as long as we've lived in this house, well over 400 days, except that over the last two nights, a black cat has visited our yard, meowing at the doors and windows. It's reasonable to think that it will visit again tonight. Yet 99.5% of evenings have been free of black cats. So how can it be inductively reasonable to think a black cat will visit tonight?
Presumably, it is because the data from the last two days is more relevant than the data from the earlier days, even though there are two orders of magnitude more black-cat-free days. But why is that data more relevant?
Granted, yesterday and the day before are more temporally similar to today than the other days. But why should temporal similarity override other kinds of similarity? No doubt there are many features (say, temperature, lunar phase, etc.) in respect of which today is more like some other day in the past 400 than like yesterday or the day before—after all, the earlier 398 days have a wide diversity of properties. But temporal similarity seems particularly important.
Maybe it is because we expect clumping, both in time and in space. Two black-cat evenings suggest the beginning of a clump.
I am curious: Is our expectation of clumping a priori justified or only a posteriori? Clumping seems to be a kind of
continuity. Is an expectation of continuity a priori justified or only a posteriori?
Any prospector will tell you "Gold is where you find it!"
ReplyDeleteI would think the assumption that stray-cats-at-the-back-door events tend to clump is aposteriori justified. Other kinds of events tend not to clump (neighbor-mowing-his-grass events, say) and we learn that aposteriori too.
ReplyDeleteThat may be. I have a hard time thinking exactly *what* data that I have justifies it, though. Maybe general data that animal behavior tends to repeat?
ReplyDeleteOh, and my prediction turned out correct. :-)
ReplyDeleteDr. Pruss,
ReplyDeleteLet's think about this. Halloween is upon us and very oddly a black cat has crossed your path three times. At the very least you should throw a pinch of salt over your left shoulder.
Cats move along continuous paths in space. Proximity today is more closely related to proximity tomorrow. I'm surprised you find this problematic.
ReplyDeleteContinuous paths by themselves don't lead to a prediction of recurrence. continuous paths, I'd expect an ever-wider cut-off (cats move fast) Gaussian distribution of the cat's position centered on the last observed position (a Brownian motion with a speed limit). Such a distribution would predict that the cat would not be in the same place at the same time the next night.
ReplyDeleteAnd I wonder if it has much to do with cats. Suppose we just have an event of some sort, but have no data on events of this sort except that we first got 398 events of subtype A and then 2 of subtype B. Should we have a high confidence, say 398/400=0.995, that the next event will be of subtype A? I think not. My intuition is that we should probably assign something around 0.5 to the next event being an A.
ReplyDeleteHere is an interesting paper on how groups respond to perceived clumping.
http://www.vanderbilt.edu/csdi/research/CSDI_WP_05-2013.pdf
The black cat who suddenly frequents your backyard is clearly a familiar spirit who just appeared in the run-up to Halloween. Hopefully, this isn't a prelude to something more sinister, along the lines of Rosemary's Baby or The Omen.
ReplyDelete
ReplyDeleteHey Dr. Pruss!
You say:
"And I wonder if it has much to do with cats. Suppose we just have an event of some sort, but have no data on events of this sort except that we first got 398 events of subtype A and then 2 of subtype B. Should we have a high confidence, say 398/400=0.995, that the next event will be of subtype A? I think not. My intuition is that we should probably assign something around 0.5 to the next event being an A."
So there should be low confidence that a subtype B event will occur since 2/400=0.005 yet since a subtype B event has occurred there is now a 0.5 level of confidence that it will occur again. So that while clumping may be rare, once it begins it's duration should not be surprising?
I'll stick with the claim that, for things moving along continuous paths, proximity today is more closely related to proximity tomorrow. So that's still my response to your original point. Of course, you're right to point out that that's not sufficient for the more general question about the cat. But cat's surely don't move by Brownian motion. They aren't random (well, that's my bet). What the distribution is, that's an interesting question. But it's a question for animal behaviorists. (And I think there's been quite a bit of work on this - I recall such studies being applied to human movements too.)
ReplyDeleteEric:
ReplyDeleteThat line of thought would suggest that it's an _a posteriori_ matter. But I am not confident of this.
I am no microbiologist but I can tell you that if someone feeds a subtype B event at location PH then there is almost a 100% chance of a subtype B event occurring at location PH the following day.
ReplyDelete