It's the ion that is part of the strongest acid/base. E.g. Na2CO3 is basic because NaOH is a strong base but H2CO3 is only a weak acid. NaCl is neutral because NaOH is a strong base and HCl is a strong acid. Being a strong base means having a large base dissociation constant (Kb) and being a strong acid means having a large acid dissociation constant (Ka) . In other words, if Kb is greater than Ka then the hydrated cation affects the pH more than the hydrated anion. But to answer your question correctly, both hydrated ions always affect the pH, but by how much depends on their dissociation constants.
If you want more info on this or how to do the calculations, you can check out http://www.chembuddy.com/?left=pH-calculation&right=introduction-acid-base-equilibrium.
Hope this helps