Should I pay off my 4.5% mortgage

I am 24 and I bought this house when I was 21 and had very little credit. 30 year fixed with 4.5 interest. The loan was 50k and I currently have it paid down to 22k. Is it smart to keep paying extra towards the principal since my interest rate is on the higher side? I am on the understanding that a refinance would not be worth it with such a small principal amount. I should add we might move in a year or two if our family grows.