It will be a struggle to get quiet never mind super silent with a small case. The heat build up with components in close proximity to each other will require a decent airflow and that often means noise.
You don't say which replacement fans you've tried but there's a limited number of options at click here I have some 120mm 9dB fans and I can still hear them when I'm sat next to the PC.
Resonance is often caused by the hard drive and although you can get rid of that by suspending the drives with elastic I don't think you have that option with a tiny case.
You can cut down some noise by fitting noise absorbing foam but it's expensive and doesn't do much to cut down on fan noise click here