How much RAM do you have? XP typically needs about 256MB to run,and each new user needs at least another 64MB. If you are switching users, as opposed to logging off, then User 1's processes are still using RAM, while User 2 requires additional.
This effect will be compounded if you need to use the pagefile (virtual memory) to swap memory from RAM to disk, and back again.
See click here and click here and click here
If the above does not apply, you could also look in Event Viewer, in case you have a rogue driver / software that is hanging or failing to initialise as you try to log in.