In any codebase, regardless of its size or number of contributors, branching is at odds with refactoring.
A refactoring, using its strict definition (Fowler, 2019: “Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.” ), can create changes across the entire codebase, including moving and renaming files, replacing ubiquitous constant names, and even modifying indentation. While any kind of change runs the risk of creating merge conflicts, refactorings are even more likely to cause conflicts, as they often aren’t as localized.
This problem is not limited to environments with multiple collaborators. It can become an issue for personal projects as well. For example, Nanoc, my arguably most popular personal project, used to follow a branching strategy with feature branches and long-term release branches. This created plenty of merge conflicts, especially when I was preparing for Nanoc 4.0: the merge conflicts were so enormous that I had to abandon the effort.
To elaborate: The risk of merge conflicts (either immediate or semantic) will make people avoid refactoring, which decreases code quality further.
To elaborate: Bad code quality increases the risk of breakage, and makes people avoid risky changes (including refactorings), which makes code worse. The spiral of death has begun.
To elaborate: Brittle code will drive people towards apparent safety. Isolated branches provide the appearance of safety. This apparent safety is a total lie, as branches will need to be integrated at some point or another. Furthermore, the longer it has been since a change has been integrated, the different the merge will be from the code in the branch, which means that code has been integrated that hasn’t been properly tested.
Fowler, Martin. 2019. Refactoring: Improving the Design of Existing Code. Second edition. Addison-Wesley Signature Series. Boston: Addison-Wesley.