Of course you make sense.
If you clear out ALL the downloaded files you might not miss them, because if you have installed these programs they exist elsewhere in your computer.
No need to go mad and clear everything, but if you downloaded from the net some application or program, you can do it again if you need to.
Just don't delete your own reference data which might have found its way in there.
If you want to delete other lumps scattered about drive C: ask in this thread naming anything that puzzles you. Someone will give you advice on keeping or deleting it.