Read these threads click here I think you will find the PC has a PCI Express slot so the answer is yes it will but you have to consider the power implications. More powerful graphics cards all need their own addition power supply connector and it has to be stable.
If you go for a really powerful card it might not run as it might need too much power and you might find it can't run the full hog as it might be slowed down by the processor.
You have your cards mixed up by the way Radeon are ATI cards and the 8800GT is an Nvidia GeForce card.
Read the threads perhaps someone else has the same card and has found it runs ok.
If you did decide to replace the motherboard and processor you would also have to buy a new copy of windows as the Dell would fail validation so personally I would forget that option and I think if you really want a newer faster PC than you already have you'd be as well buying a new one.
I wouldn't add more RAM simply because you wouldn't be able to use it in your next PC so it would have to go with the Dell and its a bit of a waste.
My PC is a 3500 with 1 gig of RAM and a poor 7600GS graphics card and it ran the Crysis demo fine which you can download. I had to drop the settings admitadly but it still looked good.