I've worn specs since I was about 8 (many years ago). And I am _extremely_ myopic (short-sighted). My glasses tended to resemble some of those comedy glasses that you see people wearing on TV.
When high-index glass first came out, I changed to those for cosmetic reasons - they looked less like bottle bottoms. But, they had a peculiar effect on verticals near the edge of my vision. These verticals had one heck of a curve to them. That wore off.
Then, as my eyes got older, my close vision deteriorated (the equivalent of needing reading glasses for those with good eyes). As I did close, mid and long-range work, I went for varifocals - to heck with the cost! It also took a while to get used to them. I used to want to fall down stairs because I couldn't focus on them properly. That wore off as I learned to 'use' my eyes and glasses to suit.
Prismatic distortion tends to be that one eye is not looking in the exact line as the other, so a prismatic correction is applied to the lens to cure this. When I first had this done, I used to suffer from the same symptoms as you appear to be getting.
So, I would say, stick with it for a couple of more weeks - and try not to use your old glasses as that 'confuses' the eyes. The eyes should learn to adapt - but it isn't an instantaneous thing. On average, I tend to take about three weeks to get used to a change of specs.