This has happened to me before - a messed up z height will affect all layers because the squish isn't there at any layer. You'll always have that difference in height between the surface being built and the nozzle no matter what layer you're on.
It appears that you need to get the nozzle closer to the print bed. You shouldn't have gaps between the lines like the rough side. Also, check the slicer settings to make sure it's doing several top layers/walls.
No worries mate, always happy to help when I can!