Using Chromium de-googled there's no harm done. Chrome's engine is much faster than Firefox's I'm afraid. It's much easier seen on older hardware, where you can actually feel the difference in speed big time. It happened to me recently on a laptop with only 600 passmark CPU points.
I see. I rather use only open source apps on that phone then (I have one normal google one, and one totally de-googled and open source).