Distance from the exchange affects the potential connection speed. The further away from the exchange the higher your loop loss or attenuation figure is likely to be. If you check the graph click here you can see that with a downstream attenuation of 40db or less you should exect to a connection speed of 8 Meg on ADSL Max but as this increases the speed decreases.
Children being on holiday and peak time slow downs doesn't affect your connection speed but can affect your download speed, this tends to be down to contention or capacity issues. There's a brief explanation of contention at click here Capacity is down to your ISP providing enough bandwidth for everyone. Any decent ISP should have enough bandwidth to cope with increased demand at peak times. I have experienced slowdowns at peak times but my ISP responded by upgrading the exchange.
Another factor that can have an effect on speed is excess noise on your phone line either caused by internal or external sources. Removng the ring wire from your master sockets removeable faceplate can help reduce internal noise.
"My Sky broadband was lousy until we realised it was being restricted by BT!"
Are you on Sky's Connect service?