Second semester calculus based physics. E&M beat me up pretty bad. I'm better at it now, but at the time it was not pretty.
Interesting perspective and not at all uncommon. There were times when I was sitting in college (undergrad) and I to loathed my time there. Fortunately, time brings perspective, and those classes are rather useful, as in most cases in the outside world you will deal with non-science people.
Those classes provide insight to other fields and how those perceive the world around them, which is valuable when communicating with them.
The world is larger then just science, and one of the important things about science is being able to communicate it to the rest of the world.
Of course, if you have no interest in communicating with non-scientists, by all means, stay the course.
