To do it, you need to bring up the Google Assistant, then tap on the Explore compass, then your account icon in the top right, then Settings. From here go to the Assistant tab, then Phone and then Speech output. Then you can change the setting.
As you can see, this option is not exactly front and centre, and until now it was always touch and go as to whether the Assistant would blurt out your answers or keep quiet. Google probably wants you to keep it on to encourage you to have a conversation with your phone more often and for longer, therefore feeding the big G with gallons of delicious user data to improve its services.