You are correct. There is the question of "invariant to what?". For example, Special Relativity is invariant to Poincaré transformations, whereas General Relativity is invariant to all coordinate transformations, a much bigger group. But if the group of transformations for which the laws of physics are invariant is limited, and that the laws of physics are not invariant to transformations outside this group, then it becomes impossible to state what these varying laws of physics are because one doesn't have a reference against which one can state which laws of physics apply to which location in the space to which the transformations apply. But if the laws of physics are invariant, then they can be stated unequivocally due to that invariance.
For example, if the laws of physics are expressed entirely in terms of partial derivatives, then they will vary according to the coordinate system. But the coordinate system is not actually known, so one can't know which laws of physics apply. But if the laws of physics are expressed entirely in terms of covariant derivatives, then they will be invariant to coordinate transformations, and therefore one no longer needs to know the coordinate system because the same laws of physics apply to all of them.
I often see the question "What is time?". The answer I give is "Time is what a clock measures.". But what is a clock? A clock is defined by the instructions that are used to construct it. While this may not be particularly helpful in understanding the nature of time, time is nevertheless well defined by the instructions.
