How does it work?

Not sure of the exact magic behind it, it seems that these files are supposed to be generated, but they need to exist as well.

Solution taken from https://mail.python.org/pipermail/python-dev/2010-December/106425.html

These files are generated using Python – basically, you need to have Python installed to fully compile Python.
For this reason, CPython release tarballs (and even the VCS tree) includes such generated files. But, make still re-makes them according to the usual rules, i.e. if they're old enough.

According to the Fedora packaging guidelines, it would be preferred to BuildRequire python2 here, and re-build these files. But I don't think it really matters; these are hardly the only files that are regenerated like this.
The compliant thing to do would involve bootstrapping, but if we want to go that way, let's do it for current python3 only.

Also for some reason this happens only at F23 and F24, not at F25 or F26

The compliant thing to do would involve bootstrapping, but if we want to go that way, let's do it for current python3 only.

python3 in Fedora is already bootstrapped.