It is the outcomes of measurements of time that are numbers, being the readings on idealised clocks. That is a subtle but important difference.
Also, just because something can be quantified does not imply that it is an illusion - that’s a non-sequitur.
The reason we know that time isn’t just an “illusion” is first and foremost the existence of gravity, specifically the tidal aspects of gravity. If you had only three spatial dimensions, but no time, it can be formally shown that tidal gravity as we observe and experience it could not exist. But since it evidently does, we know that time (as the concept is used in physics) is quite real, at least in the classical domain. On quantum scales on the other hand, the issue is more subtle and rather less straightforward - a case could potentially be made for time to not be fundamental on small enough scales. However, that would make it emergent, and still not an “illusion”; again, an important difference.