@Kutzi everytime, when you switch from 20.04 to 24.04, after booting, the system creates a backup of your user-config-data.
That's also the reason, why the system feels a bit more slow, directly after the first boot of 24.04.
I am not sure if also backups are created, when switching to 20.04 again.
Have a look at /userdata/backups and clean it up.
Maybe do it from recovery: Boot to recovery, connect through adb, mount -a, and then check user-data backups folder.
Everytime you switch, there will be changes in your /home/phablet/.config and other dot-folders, because there is always something that needs to be migrated e.g. regarding system-apps.
So I won't recommend switching back and forth.