"On some it goes to instantly but on others can take 3-4 minutes to load up"
That's not unusual. Many websites/pages do load at varying speeds, depending on the complexity of their content.
"My internet speed appears fine, between 1 & 5mps"
Use this click here to test your download and upload speeds (ideally at different times of the day) and report back.