Why did you uninstall the drivers? As they would have been fine. And it certainly will not be a power issue as I think they are pretty much the same as far power needs are concerned.
I would stick the 640 back in, install the drivers, then swap the GPU's making sure it is properly seated and connect any power cable.
I can see no reason why the 660 should not work on the Gigabyte mobo.