I keep on going back and forth on the question whether the beauty of mathematics is something surprising and metaphysically significant. I find myself going between two views.
Deflation: Mathematical beauty is just a matter of selection. There are many beautiful theorems. But there are many, many more ugly theorems. It's just that the ugly theorems don't get published, unless they are of practical importance or are appropriately connected with beautiful mathematics. Imagine that we got a book of all the theorems of arithmetic. There would be many beautiful things in the book. But intuitively a large part of the book (if that makes sense to say: it's an infinite book after all!) will just be boring theorems like "18883 x 77891 = 1470815753" or "The equation x2+9873773873+8383883=0 has no solutions."
Theology: Mathematics seems to be have more in the way of surprising beauty than we would expect from the selection hypothesis. It happens not infrequently that as a working mathematician one writes down some obvious necessary conditions for something to happen, and then one proves—often in a highly nontrivial fashion—that these necessary conditions are also sufficient. Or maybe there is just a little bit to add, and then they become sufficient. Of course, often no such thing happens—we're just stuck with necessary conditions. But the number of times that the necessary conditions are also sufficient is surprisingly large, large enough to call out for an explanation.
And that need for explanation pulls me in one of two theological directions. First, there is Augustine's idea that mathematical objects are in the mind of God, and so we would expect to find beauty in them, since God is supremely beautiful. Second, one might have the thought that we are divinely designed, among many other things, for the kind of reasoning found in mathematics. Of course, one might also offer a naturalistic evolutionary explanation. But I am not sure that will be satisfactory: finding utterly exceptionless necessary and sufficient conditions is just not something that happens much in the practical life that our evolutionary development is driven by.