Dial up or broadband? Which version of Realplayer do you have installed, and what browser are you using to listen to the audio? Can you download .ram or .ra audio and listen to it normally when it's been saved to your hard-drive? click here for example (first reasonably sized .ra file I could find quickly)
the reason download speeds decrease is due to the way TCP/IP error correction works.
Whenever your PC detects a dropped packet it sends a request for a repost, simultaneously the download rate is reduced by half. After a period the download rate is gradually increased, but drops again as soon as another dropped packet is detected.
New transport protocols are being developed to alleviate this problem which is a legacy of the old days 14.4kbs dialup.