Do you have a source for this? The data does not seem to agree with you, as is shown here:
It has been decreasing steadily since 1979 per year. The fact that it reforms in the winter doesn't matter, we are looking at the possibility of no summer ice in the next few years.
As for the polar bears, it doesn't really matter if they can swim long distances, the warmer waters make it harder for them to live and find food. If all the ice melts, they are as good as extinct.
The North Pole is a problem too. It is not necessarily the melting of the Arctic Ocean that we are worried about, but the glaciers and ice in Greenland. If that melts, we would see dramatic increases in sea levels. Here is a basic overview of how much they change per year:
The whole South Pole doesn't have to melt for it to be a major problems. In fact, the average temperature of the South Pole doesn't have to rise by that much. Even if only the edges melt, that alone would cause quite a bit of coastal flooding problems. One of the major concerns is the Ross Ice Shelf, if that melts, you are looking at up to an increase of sea levels by 5 meters.