I have phrased it exactly that way for years and get immensely frustrated when people don't understand. I agree with every point. Actually, I do have this to add:
Physics is, as you said, fundamentally how we explain the phenomena of the universe, how it all fits together and works to create that which we experience. Mathematics is the language of physics--not just of the sweeping and grandiose gravitational movement of the stars and of our more familiar interactions, but of the workings of the tiniest subatomic particles and forces.
Language that is the language of literature is the language of the soul, if you will. It is what describes those experiences, life, emotion... everything so hard to define with science (even when we can quantify it with chemical reactions in the brain), and even use it to reach out beyond that other language. It is also a language of concept and which can be explored for its own sake, but it's all ultimately about that communication of experience. The interesting thing about mathematics is that like language, it can be explored for its own sake. It can be pure concept and abstraction, of concept and abstraction, but so many things considered 'pure math' have become applicable, and it often is--even if only by creating something astonishing.
How do I feel about this? One of the most wonderful days of my life, when I nearly cried, was seeing a proper real analysis derived proof of the Fundamental Theorem of Calculus. The way logic fit together, as the logic of mathematics must, was nearly heartbreaking.
Mathematics helps describe light, and therefore color, color which then gets combined for any number of arts. Mathematics aids in construction, of life, of architecture, of sets for theater performances. It can create glorious vector maps, fractals--algorithms are used to make those visualizations of what is being played in some music player programs. Mathematics is at the heart of music itself, in the vibrations of the air, in the ratios which determine what note is played, in the very rhythms and at least Western notation. Music taught me fractions. The two are inextricably intertwined.
In the sixth and fifth centuries BCE, Pythagoras proposed the Music of the Spheres. Today, string theory posits that at the heart of everything, tiny strings somehow vibrate to determine what tiny particle exists, or what force. From that possible tiny son to our everyday improvisational chaos to the endless dance of the stars, mathematics describes it all.
Mathematics is the language of the universe.
I have never been able to conceive of anything more beautiful than that.