There are some sets we need just because of the fundamental axioms of set theory, whatever these are (ZF? ZFC?). Probably, we could satisfy the fundamental axioms of set theory with a collection of sets that in some sense is countable. But then we need to add some sets because the world is arranged thus and so. For instance, we may need to add a real number representing the exact distance between my thumbs in Planck units. (If the world is describable as a vector in a separable Hilbert space, all we need to add can be encoded as a single real number.) This is a very Aristotelian paper: the sets are an abstraction from the concrete reality of the world.
On this Aristotelian picture, what sets exist might well have been different had I wiggled my thumb. Perhaps, then, some of the non-fundamental axioms of set theory are contingent.