"rulers do not physically change their lengths as they move from one frame of reference to another":
probably you mean, the lengths of rulers that change in velocity does not physically change.
If with "physical" you mean the same as I do, then that can't be correct, as moving clocks that are co-moving with such a ruler tick slower; the two cannot be logically disconnected! For the inverse argument, see the Kennedy-Thorndike experiment: assuming (physical) length contraction, the null result implied a change in the emission frequency. Since that time we got positive confirmation of the change in emission frequencies, from which we nowadays logically conclude that length contraction also has to occur (with a few plausible assumptions).
- https://en.wikipedia.org/wiki/Kennedy%E2%80%93Thorndike_experiment#Importance_for_relativity
It is also not an original premise that there is no "fundamental" frame of reference. As a matter of fact, SR was already worked out, on the assumption of such a frame, by Lorentz and Poincaré a few months before Einstein; and he merely declared that such an assumption is "superfluous" for the derivation.
- http://www.fourmilab.ch/etexts/einstein/specrel/www/
Apart of the error indicated above, you are in good company with your reasoning, see Langevin's interpretation here, from p. 47:
https://en.wikisource.org/wiki/Translation:The_Evolution_of_Space_and_Time
Note one very important point however, which also is stressed in the preceding part of that same paper: all frames in uniform rectilinear motion are equivalent for doing observations, the same laws of physics hold in all of them (that's the PoR). As long as one keeps the phrasing non-philosophical (Einstein was good in that!), everyone can agree.
Concerning your last point, I have difficulty grasping what the issue is; but possibly the discussion by Langevin is useful for you.
PS. did you actually try to calculate an example? In my experience that is an almost necessary step for understanding.
You will then find that if you transpose your reference frame to one that is co-moving with the outbound going astronaut:
1. his wristwatch now appears to tick at the normal rate; you may think that this will prevent his watch to loose as much time as from the point of view when you use the Earth's rest frame as reference (or even that the earth clocks will seem to end up loosing more time).
2. however after turn-around, on the inbound trip his watch ticks much slower - even more so than if you would use the earth's rest frame, because now the speed that you assign to the astronaut on the inbound trip is much higher.
3. As a result (that's a design feature of the Lorentz transformations, in order to fulfill the PoR), the total amount that the astronaut's watch will be behind at the event of arriving home, is the same no matter what inertial reference frame you use.
