The phrase "finite chance" (see Google) seems to be consistently used to mean a a nonzero chance, or maybe a chance that is neither zero nor one. The phrase is very commonly used in the longer phrase "small but finite chance" (oddly, Google has more hits for the longer phrase).
Yet zero is as finite a number as you can get! So what is going on? Maybe people are implicitly thinking in terms of something like von Neumann's log odds (log p/(1−p)), where probability zero is represented by −∞ and probability one by +∞? In that case, "finite probability" would indeed mean what Bayesians call "non-extreme probability", i.e., a probability strictly between zero and one.
By the way, it seems to me that when we connect probability with evidence it is natural to think in the von Neumann way (the force of new evidence will be additive then), while if we connect it with statistical expectations it is natural to think in the classical way.