You have the blame pointed in the right direction. Usually the easiest to check is the memory, if you have two sticks take one out at a time and reboot to see if one of them is flaky. New DDR2 memory is very cheap now.
Power supplies can only be checked by substitution unless you have a friendly local repairer who might have a test rig. Again not too expensive to buy and always worth an upgrade.
If it runs OK once booted up it's not likely to be the hard drive or the motherboard or the graphics card come to that.