I don't know of any "cheap" external backup devices. You only have two options: a Zip drive or a Tape Streamer, neither of which are cheap. Your best bet would be Acronis True Image which, if you have a CD Rewriter will backup everything to CD for you.
Of these an external hard drive is likely to be compatible for the longest period of time.
That is, so long as you can connect a cable to it and of course a Firewire or USB Port.
CD's and DVDs frequently need special software to read them, and deteriorate at a stunning speed - 2 to 5 years.
I am not an expert on the stability of tapes and they are rarely discussed her but have a high reputation. However, in the old days, the magnetic medium on the surface of the tape was prone to migrate to the adjacent tape on the reel.
I'm sure that there is something wrong with external Hard Drives but that's the way I've gone for long term back-up, using Zip Disks for the short term.