I think it may be necessary to delete the contact list(s) not required. Firstly though, make sure they are backed up as I have not tried this and cannot guarantee success.
Go to the Tools menu and select E-mail Accounts.
Click "View or change existing directories or address books" and click Next.
Select "Outlook Address Book" and click Change.
At this stage I hope you would see the contacts list(s) shown. Select the Contact list(s) that you want to remove and then click Remove Address Book.
Exit and restart Outlook.