Have a look at threads in the Networking forum; many others with the same quest as yours.
You will need an ADSL modem and router. As the BT Voyager modem only offers USB connectivity and not Ethernet, you won't easily find a generic router to connect to it. Besides, it only does 4Mbps as you say. So might as well get a combined ADSL modem and router as they are much easier to setup and cheaper then getting 2 units.
You have a large choice of standard 54Mbps (802.11g) ADSL-routers, which is all you'll need as you're after a wired network.
I'd helped a friend who selected a Netgear 834 dg834g (v3) combined ADSL modem and wired/wireless router, who is also on BT 8Mb (although he gets 6 almost next door to the local exchange, he tells me).
From memory, you can turn off the wireless function and as there are the standard 4 ethernet ports on the back of the router, you can connect up to 4 PCs/devices.
There are many other
His main PC is also XP as is one son's laptop but with another son's being Vista.
There is a known issue with Vista being able to communicate with XP machines on the same network. Those XP machines needs a special package to be downloaded and installed;
Hope that helps. It will be worth your while to go over to the networking forum.