This post is inspired by Heath White's comment here.
There are lots and lots of different kinds of mathematical structures. Here's an operation on the real numbers: a#b = a3b+7. You can study this operation heavily, but chances are that you won't get anything very interesting (but maybe you will!).
But on the other hand, take something like addition or multiplication (or both). These have many beautiful properties, and lend themselves to many kinds of abstraction: groups, fields, rings, monoids, etc. When this happens, it is evidence that the structure one was studying is somehow natural. While in some way any coherent set of coherent axioms might be fruitfully studied, there both seem to be particularly natural axioms for a structure--like, commutativity--and particularly natural clusters of axioms--like those defining a group or a ring--that seem worthy of study. Anyway, around a particularly natural structure there springs up a wealth of mathematics.
Some of the most creative mathematics seems to be the identification and introduction of natural structures. For instance, one of the things I learned in my recent work in formal epistemology is that classical probability is a very natural structure. On the other hand, hyperreal-valued probabilities of the sort that some philosophers like seem to be quite an unnatural structure--one doesn't get the same wealth of neat results. The more one plays with hyperreal probabilities, the more they look like Frankenstein's monster. (On the other hand, the R(I) monoid I discuss in a recent post is rather more natural, though it may not seem that way initially.)
What is this naturalness of structure? David Lewis took natural properties to be more basic, and unnatural ones to be constructions from the more basic ones. That is not the case for natural mathematical properties. If we consider mathematics set-theoretically, all the properties--both the natural and the unnatural ones--we are studying are constructions out of set-theoretic properties. A natural cluster of axioms might be no simpler than an unnatural cluster of axioms. Moreover, the naturalness seems independent of the foundational grounding. Suppose one day we have a better foundation for mathematics than set theory. (Not unlikely!) Group theory and probability theory will still be studying something natural.
What, then, makes a mathematical structure natural? Is it purely extrinsic, with the natural properties and clusters of axioms being those that are mathematically fruitful? Or maybe there is no distinction: Maybe if the amount of effort that has gone into analyzing addition were put into analyzing the # operation I gave at the beginning of this post, we would find just as beautiful mathematics? Maybe such deflationary stories are the whole story about mathematical naturalness. But maybe there something deeper about the natural properties and clusters of axioms. Aquinas thinks all creation in some way reflects God. Perhaps the more natural properties--whether empirical or mathematical--are those that somehow more deeply reflect God's mind?
This post comes after spending over a week on some mathematical issues only to find today that I committed a subtle (perhaps only to me!) error at the beginning of the investigation, and almost all of the work has come to naught. This reminds me of the famous joke about dean talking to the physicist: "You always want money for more equipment! Why can't you be like the mathematicians? All they need are paper, pencils and garbage cans. Or better yet, why can't you be like the philosophers? They don't even need the garbage cans."