Secure file transfers frequently trade off some performance for their crypto. You can't have it both ways. (Well, you can but you'd need hardware crypto offload or end to end MACSEC, where both are more exotic use cases)
rsync is basically a copy command with a lot of knobs and
stream optimization. It also happens to be able to invoke SSH to pipeline encrypted data over the network at the cost of using ssh for encrypting the stream.
Your other two options are faster because of write-behind caching in to protocol and transfer in the clear-- you don't bog down the stream with crypto overhead, but you're also exposing your payload
File managers are probably the slowest of your options because they're a feature of the DE, and there are more layers of calls between your client and the data stream. Plus, it's probably leveraging one of NFS, Samba or SSHFS anyway.
I believe "rsync -e ssh" is going to be your best over all case for secure, fast, and xattrs. SCP might be a close second. SSHFS is a userland application, and might suffer some penalties for it
