The fear/terror of invasive species is mostly based on past experience with non-native species. Things like Zebra Mussels have had an extreme negative impact on ecosystems and industry along the great lakes. Other non-native animals and plants have extreme impacts and some have little to none. Lionfish have all the qualities of a negative impact and this can be seen by their impacts on native fishes in the areas the lionfish has taken over. The lionfish has a negative impact on the diversity of native fishes. This is mostly due to the extreme numbers of these fish and the fact they eat the small native fishes that other native predators eat there by affecting the numbers of native predators like groupers and other native predators who rely on the same food sources. This in itself is enough to show a negative impact.
Other invasive have been around so long we have little to no data about what the ecosystem was like before they were introduced. The common Carp is one example of this, they destroy the nests of native species, stir up sediments, and outcompete natives that occupied the same ecological niches. Fishes affected by this range from sturgeon to buffalo to trout with many other species who occupy similar niches. Carp are so ubiquitous they have become tolerated and even loved by fishermen and others, but do not doubt they have reshaped the ecosystems they reside in drastically.
The effect any non-native species has on the ecosystems they are introduced into is difficult if not impossible to predict so all are treated as ticking time bombs. Who could have predicted the effect of north american crayfish in European waters? All we can really do is do our best to prevent the introduction of any non-native species into ecosystems worldwide.
Sadly, introducing non-native species outside their former range is pretty much what humans do, when humans migrate to a new place they have historically taken their domestic animals with them. Count things like rats in the group as well but things like goats, pigs, dogs, cats, horses, and other domestic animals have gone feral and some cause considerable damage to the environment. Feral pigs are an excellent example. We have taken everything from fish to pigs and more with us as we spread to environments not native to us.
We humans are an invasive species extraordinaire, the worst of the worst, but we just keep on doing our thing and then stumbling across the effects and wondering why.