There is a presumption against the state imposing or enforcing restrictions on people's behavior. That's why, for instance, the state does not enforce private promises where money doesn't change hands. Now, marriage has two primary normative effects:
- Make sexual union permissible;
- Impose a rich tapestry of duties that the spouses owe to one another.
In those jurisdictions where fornication is not a crime, I think it is helpful to start debate about things like same-sex marriage or polygamy with a presumption against state involvement in any marriages whatsoever, and then ask in what cases, if any, that default negative judgment can be overcome.
(For the record, I do think the presumption can be overcome in opposite-sex cases, because of the connection with procreation. But I am not arguing for this here.)