Satellites have/can/will measure the movement of the plates. Which is a couple of centimers per year if I am correct.
Besides that, it is also the theory which fits best with the data. It explains why we find rocks normally found in the tropics or deserts in areas surrounding the poles but even on antarctica itself. You could also think about fossils which match on very different continents and magnetite in magmatic rocks which point to very different poles than we have today. Surely you could find a theory for everyone of them, but as far as I am aware of continental drift is the only one that links them all.