I have a question before I give you any suggestion.
1) Are these two sites in your vector present in MCS of the plasmid?
If yes, then don't gel purify your vector, it will avoid loss of plasmid during the process.
2) If you are just deleting few bases from the PCR product, I would recommend skipping the gel purification step.
3) Ligate it for more than 12 hrs/over night at 4 degrees
4) Have a negative control where there is no insert added. In that way, you would get some idea about the background ligation.
Reply if you have any more questions