Using general purpose LLMs to help with set theory questions

Are general purpose LLMs useful to figuring things out in set theory? Here is a story about two experiences I recently had. Don’t worry about the mathematical details.

Last week I wanted to know whether one can prove a certain strengthened version of Cantor’s Theorem without using the Axiom of Choice. I asked Gemini. The results were striking. It looked like a proof, but at crucial stages degenerated into weirdness. It started the proof as a reductio, and then correctly proved a bunch of things, and then claimed that this leads to a contradiction. It then said a bunch of stuff that didn’t yield a contradiction, and then said the proof was complete. Then it said a bunch more stuff that sounded like it was kind of seeing that there was no contradiction.

The “proof” also had a step that needed more explanation and it offered to give an explanation. When I accepted its offer it said something that sounded right, but it implicitly used the Axiom of Choice, which I expressly told it in the initial problem it wasn’t supposed to. When I called it on this, it admitted it, but defended itself by saying it was using a widely-accepted weaker version of Choice (true, but irrelevant).

ChatGPT screwed up in a different way. Both LLMs produced something that at the local level looked like a proof, but wasn’t. I ended up asking MathOverflow and getting a correct answer.

Today, I was thinking about Martin’s Axiom which is something that I am very unfamiliar with. Along the way, I wanted to know if:

1. There is an upper bound on the cardinality of a compact Hausdorff topological space that satisfies the countable chain condition (ccc).

Don’t worry about what the terms mean. Gemini told me this was a “classic” question and the answer was positive. It said that the answer depended on a “deep” result of Shapirovskii from 1974 that implied that:

2. Every compact Hausdorff topological space satisfying the ccc is separable.

A warning bell that I failed to heed sufficiently was that Gemini’s exposition of Shapirovskii included the phrase “the cc(X) = cc(X) implies d(X) = cc(X)”, which is not only ungrammatical (“the”?!) but has a trivial antecedent.

I had trouble finding an etext of the Shapirovskii paper (which from the title is on a relevant topic), so I asked ChatGPT whether (2) is true. Its short answer was: “Not provable in ZFC.” It then said that the existence of a counterexample is independent of the ZFC axioms. Well, I Googled a bit more, and found that the falsity of (2) follows from the ZFC axioms given the highest-ranked answer here as combined with the (very basic) Tychonoff theorem (I am not just relying on authority here: I can see that the example in the answer works). Thus, the “Not provable” claim was just false. I suspect that ChatGPT got its wrong answer by reading too much into a low-ranked answer on the same page (the low ranked answer gave a counterexample that is independent of the ZFC axioms, but did not claim that all counterexamples are so independent).

A tiny bit of thought about the counterexample to (2) made it clear to me that the answer to (1) was negative.

I then asked Gemini in a new session directly about (2). It gave essentially the same incorrect answer as ChatGPT, but with a bit more detail. Amusingly, this contradicts what Gemini said to my initial question.

Finally, just as I was writing this up, I asked ChatGPT directly about (1). It correctly stated that the answer to (1) is negative. However, parts of its argument were incorrect—it gave an inequality (which I haven’t checked the correctness of) but then its argument relied on the opposite inequality.

So, here’s the upshot. On my first set theoretic question, the incorrect answers of both LLMs did not help me in the least. On my second question, Gemini was wrong, but it did point me to a connection between (1) and (2) (which I should have seen myself), and further investigation led me to negative answers to both (1) and (2). Both Gemini and ChatGPT got (2) wrong. ChatGPT got the answer to (1) right (which it had a 50% chance of, I suppose) but got the argument wrong.

Nonetheless, on my second question Gemini did actually help me, by pointing me to a connection that along with MathOverflow pointed me to the right answer. If you know what you’re doing, you can get something useful out of these tools. But it’s dangerous: you need to be able to extract kernels from truth from a mix of truth and falsity. You can’t trust anything set theoretic the LLM gives, not even if it gives a source.

Unifying Separation and Choice

Let's round out Axiom of Choice Week. :-)

It’s occurred to me that there is a somewhat pleasant way to integrate the Axioms of Separation and Choice into one axiom schema.

Let’s say that a formula F(x,y) is a partial equivalence (is that the right term?) provided that it’s symmetric and transitive. Now consider this schema (understood to be universally closed over all free variables in F other than x and y):

• If F(x,y) is a partial equivalence, then for any set a there is a subset b such that for every x ∈ b we have F(x,x), and for any x ∈ a such that F(x,x), there is a unique y ∈ b such that F(x,y).

We might call this the Axiom (Schema) of Representatives.

To get the Axiom of Separation, given a formula G(x), let F(x,y) be the formula G(x) ∧ y = x. To get the Axiom of Choice, if c is a set of nonempty disjoint sets, let F(x,y) be ∃d ∈ c(x∈d∧y∈d) and let a = ⋃c (so we need the Union Axiom).

So what?

Nothing earthshaking.

But, first, while there is an advantage to keeping axioms separate for purposes of proving independence results, the more unified our axiomatic system is, the less ad hoc it looks. Unifying Separation and Choice can make us less suspicious about Choice, for instance.

Second, the Axiom Schema of Representatives has nice analogues in some other contexts than set theory. It seems to directly generalize to classes, for instance. Moreover, it extends very nicely to plural quantification to integrate Plural Comprehension with a version of Choice:

• If F(x,y) is a partial equivalence, then there are bs such that (i) for every x among the bs we have F(x,x), and (ii) for any x such that F(x,x), there is a unique y among the bs such that F(x,y).

I don’t know if there is a natural way to extend this to mereology.

One might complain that partial equivalence is less natural than equivalence. I don’t think so. First, it is defined by two instead of three conditions, which makes it seem more natural. Second, examples of partial equivalence relations tend to be more natural than examples of full equivalence relations if our domain is all of reality. For instance, “same color”, “same shape”, “same size”, “same species”, etc., are all partial equivalence relations, since only things with color are the same color as themselves, only things with shape are the same shape as themselves, etc. To form full equivalences, one needs to stipulate awkward relations like “same color or both colorless”.

An odd thought about ZFC

The axioms of ZFC set theory can be divided into (a) the positive axioms, that say that a set with certain properties exist, and (b) two negative axioms that deny the existence of certain sets (Extensionality: given any set, there is no other set with the same members; Regularity: no irregular sets).

The positive axioms divide further into two classes: (i) those that are obvious special cases of naive set theory’s Axiom of Comprehension, and (ii) the Axiom of Choice.

Here is an alternate intellectual history thought experiment. Suppose we never discovered the contradiction in naive set theory or anything like it, maybe because we had a psychological block against thinking about non-self-membered sets, applying Cantor’s Theorem to the universal set, etc. The Axiom of Choice would continue to have an intuitive plausibility, and the “mathematical need” for it, say in the case of the Hahn-Banach Theorem, would likely still arise. And so we would be pulled to adopt it.

This makes me think this. The other positive axioms of ZFC (i.e., the positive axioms of ZF) have an ad hoc feel to them. They are special cases of Comprehension, carefully chosen to both give enough applications of Comprehension and to avoid contradiction (we hope). I feel that much of the plausibility of the other positive axioms of ZFC comes from their being special cases of the highly intuitive—but incoherent—Axiom of Comprehension. And that’s a little suspicious.

Normally one thinks of the Axiom of Choice as the most suspicious of ZFC’s axioms. But here we have a source of suspicion for axioms of ZFC that does not affect Choice.

Well, maybe. Maybe an enemy of Choice could say that both Choice and Comprehension are the fruit of the poisonous tree of principles of plenitude.