Should I aggressively pay off mortgage or continue to invest?

I am 33, married with an infant

My total compensation now is ~$350k (base+bonus)

In 2 years or so my total compensation will be ~$430k when my RSUs start vesting

I have about ~$527k loan left on a house worth ~$720k with an interest rate of 6.05%

Total investments in brokerage accounts ~$110k

Have about ~$150k in 401k and another ~$20k in separate pension accounts

Lastly, I have ~$30k in my checking account

My net worth roughly equates to $500k

I have no other significant expenses moving forward except for a $5k monthly mortgage payment (covers mortgage+property tax+insurance)

My wife pays the utilities, groceries etc.

I recently bought this house so may have some one time expenses upcoming but overall nothing recurring that is significant

Now the question is, with my interest rate being so high, I’m keen on getting my loan cleared ASAP. On the other hand, aggressive payments towards mortgage limits my ability to continue to invest

I am seeing a lot of similar posts in this sub and others but the responses seem to indicate that it varies for case to case as to which is better.

I plan to continue working for the next 20 years or so and expect salary bumps and the way. I think if I continue on my trajectory I will be fine financially but if I can accelerate my growth, I’d like to retire early.

My retirement goal is to make ~$400-500k per year in interests/dividends. I don’t plan to spend it all and can continue to re-invest the money that I don’t use. Of course, this is a pretty lofty goal and I would need close to $10 million if I assume 5% returns (interest/dividends) to make the $400-500k/year

Given my goal and where I am today, what would be the best course of action? Should I pay off loans as fast as possible so as to limit the money I am losing to interest or should I make the monthly payments and just continue to invest the rest?

Dragging on the loan just seems like so much money going to waste on interest over the long run though but appreciate any advice from this sub