I have a single server which has several services on it; it is my nextcloud server, my collabora server (installed from the deb package), AND my DNS server (pihole with bind as an upstream). Because of this, my DNS configuration is a bit strange; if I add an entry to /etc/hosts for nextcloud.domain.com to 127.0.0.1 it changes my DNS resolver to not properly resolve that domain, however my understanding is that host entry is required for collabora to communicate with nextcloud. Is there a way to specify the collabora server by IP or to specify localhost or something to bypass this issue?
You should have both Nextcloud and Collabora having valid DNS names (which is prerequisite for TLS). All this names should exists in DNS and be resolvable by each component - client Nextcloud server and Collabora must resolve all the DNS names to the right ip (your real server ip in this case as 127.0.0.1 might be not the right one especially if you use some virtualization)
You can refer to Nextcloud forum where I posted some drawings and references: