That could be the reason we don't get visitors, maybe the abundance of some element here on earth is high enough to deter them. I'm not sure we can make assumptions about how they could or could not protect themselves from toxins elemental or biological.
Here on Earth we have some pretty toxic biologicals, poison ivy is hell on earth to us if we get into it. I think allergens could be a big problem, what we see as just hay fever could be a deadly plague to them. In fact we have some resistance to biologicals they might not have any resistance at all to things that don't affect us in the slightest.
Contact between ecosystems could be a bigger barrier that we think.
I have a particular fetish for stanford torus type habitats, another planet enough like Earth to support us seems unlikely. In the range of planets we have discovered so far none are really identical to the Earth. It raises the question of just how much like Earth would a planet have to be or maybe more importantly how different could it be and still support us?