::
Think of the secure element as a much better SD card. It’s a SE, after all, one letter better 🙂
The SD card just has a static secret on it. If you forget it in the Trezor, or if someone finds it, they can just pop it into a computer and copy the secret.
The SE has its own 16 tries limit (in addition to the one checked by Trezor), and won’t give out the secret unless you already know the right PIN.
As for the „closed source“ thing, your SD card is also very much closed source.
There is literally no downside — even in the absolutely worst case, if it turned out that the SE is totally evil and malicious, the Trezor is still exactly as secure as if the SE was just a fancy SD card.