Finitists say that it is impossible to actually have an infinite number of things. Now, either it is logically contradictory that there is an infinite number of things or not. If it is not logically contradictory, then the finitist's absurdity-based arguments are seriously weakened. If it is logically contradictory, on the other hand, then standard mathematics is contradictory, since standard mathematics can prove that there are infinitely many things (e.g., primes). But from the contradictory everything follows ("explosion", logicians call it). So in standard mathematics everything is true, and standard mathematics breaks down.
I suppose the finitist's best bet is to say that an infinite number of things is not logically contradictory, but metaphysically impossible. That requires a careful toning down of some of the arguments for finitism, though.