Upstream says the problem is in FreeIPA. Sounds like it needs to be upstream in FreeIPA or in a patch.
This is a consequence of two things: 1) after the python 2/3 code merge what used to be unicode objects in the TXT record strings field became bytes objects, and 2) the FreeIPA code is directly invoking the TXT RR constructor instread of calling dns.rdata.from_text(), which is how dnspython would like you to do this kind of thing. If FreeIPA had called dns.rdata.from_text(), the right thing would have happened.
So at this point I think the best thing is to change the FreeIPA code as it will then work with any version of dnspython. I don't want to start making RR constructors apply lots of conversions as that's a job already handled in from_text().