The obvious model of a Newtonian space is as the set of all triples (x,y,z) of real-numbered coordinates. But the model does not have isotropy that Newtonian space does. It has privileged directions, such as the x-axis, the y-axis and the z-axis. It has privileged coordinates such as (0,0,0). Of course, physical models generally do have properties that aren't found in what is modeled. If I build a model of the solar system out of fruit, the fact that some of the fruit is sweeter need not model any property of the solar system. If I make a model of an ethanol molecule out of sticks and balls, the balls that represent hydrogen atoms differ in their exact mass, and exhibit scratches, in a way that the hydrogen atoms do not.
Nonetheless, even though this is common to all modeling, there really is something a little unsatisfying when the mathematical model does this. Typically when we mathematically model something, we have to abstract or forget on both sides. On the side of what is modeled, the side of the world, we ignore aspects of the physical structure because otherwise things get too complicated. On the side of the model, we ignore aspects of the mathematical structure because they don't, as far as we know, correspond to anything in the physics. Wouldn't it be nice if we could abstract only on one side, that of the world? But some things that would be nice are not an option.
The above remarks do, I think, make Pythagoreanism less plausible. There seems to be structure in the mathematics that models the world that isn't found in the world. This makes it implausible that the world just is composed of the mathematics.