Plenty of the mathematics in science is ugly. But the mathematics in fundamental physics tends to be beautiful. It could be that in the correct fundamental physics it won't be beautiful. I wonder if we have reason to think it will be. The fact that in our current fundamental physics theories there is mathematical beauty doesn't seem to say much, because our current fundamental physics is probably false.
Still, I hope that the book of the world is written by God in such a way that the mathematical elegance of approximately true theories points to the mathematical elegance of the true theory in physics. I wonder, though, if an atheist could have any reason to have such a hope.