The joy of USB is that it is hot swappable- Therefore if it is not in use -leave it unplugged. except that is for the printer. These usually set themselves up with a 'printer support' USB which is separate. As discussed the more 'bare' USB devices plugged in the more the 'pull' on the PSU, thus slowing or completely stalling the machine. A self powered hub is a remedy.