Try Marg7's idea of plugging into different part of your ring main.
Other than that, all the symptoms you've described are those of a faling PSU. USB ports will only fail if a specific part (5v rail) of the PSU fails.
Your graphics card will use a different part of the PSU to power it.
To elimate other possibilities:
If it won't invalidate your warranty on anything, then it's worth pulling your PC apart and cleaning any dust out. Household dust isn't generally very conductive, but on rare occasions it can lead to shortouts.
If you have a different graphc card you can try that wouldn't be a bad idea.