Depends on how big you want them.
For 'new' (smallish) potatoes leave them until after the flowers have died off. For main crop (largish) potatoes leave them until around the time the haulms start to go over or around 6 to 8 weeks after the flowers die back.
If the potatoes are in a tub (in bagged compost?) you have no need to worry about blight and the odd slug is no more than you will get in commercially grown produce. Make certain to keep them moist (the tub has holes?) as excess water will drain off. Also remember the potato needs the top to grow so don't cut it back.
I have a couple of tubs myself and now the tops are getting bigger I am awaiting the flowers as I really enjoy freshly gathered new potatoes with my salad.