Unless the default has been changed, ‘Safely Remove’ a USB device is only necessary for devices that are using ‘delayed write’, like internal hard disks.
With devices like memory keys go to Start > Run type diskmgmt.msc and press Enter to see if the drive shows up under ‘Disk drives‘. If so, right-click it > Properties > Policies Tab and check if you have a ‘Quick Removable’ check box. A USB external drive should have a check in this box. Reboot if you change anything.
Hard disk drives use ‘Optimize for performance‘ and also ‘Enable write caching on the disk’. I suggest you always use the same port and don’t be too hasty in unplugging a device that has just been written to.