Another thing to consider is that these hypothetical aliens wouldn't have "paid their dues" on earth, meaning that they would have evolved in a completely different environment than us.
So something like a common cold or even a bacteria living in a symbiotic relationship with humans could wipe them out completely, assuming the parasite takes to this new host and said host has evolved an immune system that was meant to deal with an entirely different set of parasites and so is defenseless in this case.
This is the "War of the Worlds" defense
