I generally agree that there's no reason to distrust what people say they're doing here, so we do have data to go on, but assuming because someone is a bodybuilder, for example, that their nutrition and other lifestyle factors are optimized for PE gains doesn't seem like a given to me.
In your initial post, you said:
This actually puts us in more agreement than you might think. In the remodeling hypothesis, mechanical stress changes the makeup of the tissue over time. It's a living process with living tissue. If too much stress is applied, injury can happen. However, another consequence of too much stress is a microscopic thickening. This thickening occurs because the living organ is trying to make itself stronger to resist the stress. This works against gains because the tissue is actively trying to resist getting longer (and ultimately breaking) under these conditions. This is exactly the protective process you describe.
My current sense is that there is a sweet spot that generates remodeling over time while also not causing so much stress that the penis becomes too fibrous to produce gains. This is another reason breaks are important. As this article explains, the remodeling process is actually separate from acute inflammation and the generation of scar tissue. Without sufficient time off, all you're doing is generating microscopic collagen fibers without encouraging the remodeling process (which happens after the stress is removed).
Here is another excellent article about the mechanics of connective tissues.
For those interested in a deeper dive, go to your favorite search engine and type in "The Long Game: Learning with Longerstretch". It's a post on another site, but it's chock full of links and great information.