FileVault is never technically turned off on Macs with a T2. Everything is always encrypted. It's just that when FV is on, the user has the decryption key that the T2 will require to allow the volume to be mounted either on login or when mounting via TDM. When FV is off, the T2 uses its own (hidden) key and doesn't ask for it. This is why turning it on/off doesn't take any time.