If casual sex is permissible, so is premarital sex. Now, on a view on which premarital sex is permissible, marriage is a complex normative institution that removes the right to have sex with others and confers on the spouses various duties—such as of loving, cherishing, honoring and caring for—to the other, as well as makes for at least a ceteris paribus commitment that the couple will strive to have a sexual relationship. (If premarital sex is impermissible, then marriage has one more normative component: it confers a permission for sex.)
But there need not be anything morally wrong with x's promising y that she will not have sex with anyone other than y and z. Promises of loving, cherishing, honoring and caring for another are a great thing when taken really seriously, and are simply a higher and deeper form of the commitment that we all have anyway to our friends, and there seems nothing wrong with making such promises to multiple people, as long as there are implicit or explicit rules on how apparent conflicts of love and care are to be resolved (a problem that is already anyway present in the case of a monogamous marriage, since it can come up with respect to duties to spouse and to children, since these can be in tension). The only component possibly problematic in the normative complex is the ceteris paribus commitment to a sexual relationship with multiple people. But it is hard to see what is wrong with that if casual sex is permissible. If it would be permissible for Jane to have sex with Sid and Roman on alternate days, why would it not be permissible for her to make a ceteris paribus promise to do so? This is particularly unproblematic if one thinks of marriage as permissibly dissoluble, as most people who think casual sex is permissible do.
So it seems that if casual sex is permissible, then the normative complex of commitments that constitutes marriage can be permissibly modified to a plural form. One may ask whether the modified version would still count as a marriage. If not, then polygamy is misnamed: it's not a plural marriage (poly-gamy) but a plural marriage-like relationship. But either way, we get the conclusion: If casual sex is permissible, so is polygamy.
An interesting question is whether we can prove the stronger claim that if premarital sex is permissible, so is polygamy. Probably not, since someone could think that premarital sex is permissible only in the context of a relationship with an exclusive commitment to one person. But if one thinks that something weaker than an exclusive commitment is sufficient for permissibility, maybe love, or maybe mutual respect (Martha Nussbaum), then one may still get the conclusion that polygamy is permissible.
Of course, the right conclusion to draw is that casual sex is impermissible.