While the earths gravitational effect on all objects is constant, all objects are also creating their own (admittedly tiny) gravitational fields because they also have mass. Therefore, the mass of the object does have an effect. It may be considered insignificant, but *technically* it is still there.
An object with sufficient mass (say, a piece of a neutron star) would fall at a much greater speed than a bowling ball, even in a vacuum.
So doesn't that mean that mass does matter, it's just that nothing has enough mass for it to be significant?