The worst thing you can say to these people is "No" or "Is there any reason I should?" because then you get a very detailed explanation of who they are and why you should recognise them.
These ripostes usually occur following some unflattering happening to the person concerned. The area where I'm located (Goodwood/Chichester) has more than its fair share of people of that ilk and my receptionist once told some arrogant so-called personality in response to the question "You are a rude and self satisfied person too full of your own self importance." The person concerned was one of the local "big-wigs" who had come in to the studio to complain that our company motorbike (which we use for our own courier service to London) was making too much noise at night. The fact is that the bike is never used at night and is not there anyway.
My brother responded once saying "Yes, Madam and I'm your gynaecologist" - which he was and is.