be very afraid.
If you allow remote linking to avatars on your site you could indeed find that your bandwidth consumptions soars - particularly if/when your forum starts to get busy. Avatars may only be 5 or 6k files, but imagine a busy thread with dozens, or even hundreds of people checking in and out - all of them downloading avatars. It doesn't take long for the bandwidth consumption to rack up.
There's another problem with remote linking too. Suppose you have a couple of hundred members - all with their avatars on your webspace. Then imagine that you decide to do a spot of site reorganisation. You'll only have to rename a file or so, and you'll get a flood of email about avatars not working.
My advice - give it a wide berth Barry, but if you do decide to go ahead, set a 6k limit on avatar file size.