I am experiencing a rather frustrating problem that has me at my wits end using CODE as a Document Editor with Nextcloud. FYI I tried entering the urls in a more readable fashion and was told I couldn’t post that many links because this is my first post.
I have a Nextcloud instance in a Proxmox LXC that I have maintained for years for myself and family members at cloud dot domain1. I have a Collabora instance on a separate LXC that has been working since I set it up at collabora dot mydomain. Both servers are running behind a Caddy reverse proxy that is handling SSL. Collabora is set up with SSL false, and termination true. I can create and edit documents from the browser, a tablet, a phone, etc… and I’m pleased with the system.
Recently I started working on setting up another Nextcloud server for a local volunteer organization that I work with, so we can digitize our documentation and stop printing mountains of paper. The Nextcloud instance is setup exactly as my original instance only at cloud dot domain2 and it is working fine for file sharing purposes. However, I have setup a 2nd Collabora instance at collabora2 dot mydomain and it simply does not work.
All 4 LXC containers are running on the same physical server and storage space. I have confirmed that the Caddy reverse proxy configuration is identical for both collabora instances other than the hostname and IP address.
In my investigations I have discovered several oddities. I do not know which of these items might be relevant so I am including them all here.
1: If I go to the Office settings in either Nextcloud server I can enter either collabora dot mydomain or collabora2 dot mydomain and I get a Green symbol and a message that the server is connected correctly. This apparently doesn’t mean what one would think it means.
2: cloud dot domain1 can connect to collabora dot mydomain even if it is NOT listed as a host in the coolwsd.xml file.
3: cloud dot domain2 cannot connect to collabora dot mydomain even if it IS listed as a host in the coolwsd.xml file.
4: Neither cloud dot domain1 or cloud dot domain2 can connect to collabora2 dot mydomain, whether they are listed as hosts or not.
5: I tried the “groups” setting instead of “first” on both collabora and collabora2, That made no difference.
6: If I go to https://collabora.mydomain.net/hosting/discovery? or https://collabora2.mydomain.net/hosting/discovery? I get identical output, other than the extra digit in the hostname.
7: I have checked the error logs and I’m seeing these errors in journalctl on both the working and non-working collabora instances when I attempt to open a document.
Dec 31 10:53:29 collabora coolwsd[783]: kit-00783-00696 2022-12-31 10:53:29.347688 -0500 [ kit_spare_003 ] ERR mknod(/opt/cool/child-roots/Q011lXWcQqd6JKsM//tmp/dev/random) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:300
Dec 31 10:53:29 collabora coolwsd[783]: kit-00783-00696 2022-12-31 10:53:29.347963 -0500 [ kit_spare_003 ] ERR mknod(/opt/cool/child-roots/Q011lXWcQqd6JKsM//tmp/dev/urandom) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:312
Dec 31 10:53:54 collabora coolwsd[709]: Forced Exit with code: 0
Dec 31 10:53:54 collabora coolwsd[709]: kit-00709-00696 2022-12-31 10:53:54.761138 -0500 [ kitbroker_002 ] FTL Forced Exit with code: 0| common/Util.cpp:1117
Dec 31 10:53:55 collabora coolwsd[792]: /usr/bin/coolmount: forced unmount of [/opt/cool/child-roots/YusWWkirppN2wquq/lo] failed: Permission denied.
Dec 31 10:53:55 collabora coolwsd[794]: /usr/bin/coolmount: forced unmount of [/opt/cool/child-roots/YusWWkirppN2wquq] failed: Permission denied.
Even with these errors the document opens just fine on cloud dot domain1 when it is pointed at collabora dot mydomain. The document fails to load on cloud dot domain1 when it is pointed at collabora2 dot mydomain, and fails to load on cloud dot domain2 when pointed at either collabora instance.
8: On collabora2 dot mydomain I also get these errors in journalctl when I attempt to open a document and it fails to load. I am leaving the errors that show up on both instances in case the sequence helps diagnose anything.
Dec 31 11:02:59 collabora2 coolwsd[635]: kit-00635-00510 2022-12-31 11:02:59.991314 -0500 [ kit_spare_003 ] ERR mknod(/opt/cool/child-roots/daVducP88W8Nmkhw//tmp/dev/random) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:300
Dec 31 11:02:59 collabora2 coolwsd[635]: kit-00635-00510 2022-12-31 11:02:59.991556 -0500 [ kit_spare_003 ] ERR mknod(/opt/cool/child-roots/daVducP88W8Nmkhw//tmp/dev/urandom) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:312
Dec 31 11:03:29 collabora2 coolwsd[503]: wsd-00503-00634 2022-12-31 11:03:29.624351 -0500 [ docbroker_002 ] ERR WOPI::CheckFileInfo failed for URI [https://cloud.domain2.net/index.php/apps/richdocuments/wopi/files/671_ocdap12jokzu?access_token=VFiLJxEwmxzISPUBG08GUH76Lr8nEtHc&access_token_ttl=1672538578000]: 0 . Headers: Body: []| wsd/Storage.cpp:687
Dec 31 11:03:29 collabora2 coolwsd[503]: wsd-00503-00634 2022-12-31 11:03:29.624555 -0500 [ docbroker_002 ] ERR loading document exception: WOPI::CheckFileInfo failed: | wsd/DocumentBroker.cpp:2339
Dec 31 11:03:29 collabora2 coolwsd[503]: wsd-00503-00634 2022-12-31 11:03:29.624615 -0500 [ docbroker_002 ] ERR Failed to add session to [https://cloud.domain2.net:443/index.php/apps/richdocuments/wopi/files/671_ocdap12jokzu] with URI [https://cloud.domain2.net/index.php/apps/richdocuments/wopi/files/671_ocdap12jokzu?access_token=VFiLJxEwmxzISPUBG08GUH76Lr8nEtHc&access_token_ttl=1672538578000]: WOPI::CheckFileInfo failed: | wsd/DocumentBroker.cpp:2301
Dec 31 11:03:29 collabora2 coolwsd[503]: wsd-00503-00634 2022-12-31 11:03:29.624671 -0500 [ docbroker_002 ] ERR Storage error while starting session on https://cloud.domain2.net:443/index.php/apps/richdocuments/wopi/files/671_ocdap12jokzu for socket #19. Terminating connection. Error: WOPI::CheckFileInfo failed: | wsd/COOLWSD.cpp:4734
Dec 31 11:03:29 collabora2 coolwsd[503]: wsd-00503-00634 2022-12-31 11:03:29.631761 -0500 [ docbroker_002 ] ERR #31: Read failed, have 0 buffered bytes (ECONNRESET: Connection reset by peer)| net/Socket.hpp:1133
Dec 31 11:03:29 collabora2 coolwsd[640]: /usr/bin/coolmount: forced unmount of [/opt/cool/child-roots/vpNptFYoRj8GewNe/lo] failed: Permission denied.
Dec 31 11:03:29 collabora2 coolwsd[642]: /usr/bin/coolmount: forced unmount of [/opt/cool/child-roots/vpNptFYoRj8GewNe] failed: Permission denied.
Dec 31 11:03:30 collabora2 coolwsd[643]: kit-00643-00510 2022-12-31 11:03:30.299400 -0500 [ kit_spare_004 ] ERR mknod(/opt/cool/child-roots/UUTkZdJLthVfvRD5//tmp/dev/random) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:300
Dec 31 11:03:30 collabora2 coolwsd[643]: kit-00643-00510 2022-12-31 11:03:30.299612 -0500 [ kit_spare_004 ] ERR mknod(/opt/cool/child-roots/UUTkZdJLthVfvRD5//tmp/dev/urandom) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:312
I have gone over the configuration files and file and directory permissions on all 4 containers and everything seems to match perfectly whether it is a working combination or not.
I also spun up a full Collabora VM in case the containerization was causing anything. I get the same errors there so that’s not it.
I have rebuilt the Collabora LXC multiple times following the official directions and several online tutorials in case there was a missing step, and no matter how I go through it nothing I create works with either cloud dot domain1 or cloud dot domain2. At this point I would be ready to say that it will never work except that I have indisputable evidence that it’s possible since collabora dot domain is still working with cloud dot domain1.
Any ideas what would be causing the storage errors and other errors referenced in the log?
What else can I check here to get this working? Is there anything else I can be checking on the nextcloud side to further investigate?