I'm a third year mechanical engineering student in Australia. It is my experience that students that are poor at maths, tend to really struggle. There are quite a few who enter my university via the back door, coming from technical college (called TAFE) rather than high school, who actually tend to have a much better attitude towards study, but really struggle with the maths (resulting in a high proportion to drop out).
Yes, the course is much more applied (less mathsy) as you reach the third and fourth years, but you still have to reach these years.
I'd say that you won't be able to hack the course if you don't have a good grasp on maths, even if your understanding of physics is that good. However, I'm guessing that if you are as good at physics as you seem to think, you probably underestimate your understand of calc.
Maths is one of those subjects that if you don't understand the basic stuff, your not going to be able to understand the more complex stuff. For example if you skip a chapter in your text book, it probably will start to look like heiroglyphics.
My advise would be to take a step back and maybe work through a more basic calculus text... real introductory stuff. Then build on this until you understand at least the high school level. You can do it, but clearly you are going to have to do some work, and lets face it, you've got that extra time while every one else is studying physics!
You can ask me if you've got any questions about Mech. Eng,
Josh