We’re trying to debug a performance and stability issue in our Collabora Online setup in Production, which is integrated with Nextcloud. During our investigation on our Staging/Acceptance environment, we found out we can reproducibly crash coolwsd
with a segmentation fault, when we run the automated typer with two simultaneous users in the same document.
Is anyone else seeing this too? Any ideas on how to fix this issue? It might be related specifically to using the shapes tools combined with the typer
tool, as having multiple users run the typer individually seems to be stable.
I’ll gladly provide more information or testcases if required.
Steps to reproduce
- Go to Nextcloud Files
- Open a (new or existing) document for testing
- Start the Debug Tools (
CTRL-Shift-ALT-D
) - Under
Functionality
: checkTyper
- Under
Automated User
, check bothEnable automated user
andInsert and delete shape
The Debug Tools will start to type content and insert/delete random shapes.
Now, use a second user account to open the same document and follow steps three to five. In our case, this triggers a segmentation fault and (usually) an automated restart of the coolwsd
service.
Setup
We run Collabora 24.04.4-1
on a dedicated AMD64 Debian 12.5 VM with the latest updates installed (Nextcloud itself runs on another separate VM).
(rdo-stag)root@collabora-1:~# dpkg -l|grep collab
ii collaboraoffice 24.04.4-1 amd64 Brand module for Collabora Office 24.04
ii collaboraoffice-dict-en 24.04.4-1 amd64 English dictionary for Collabora Office 24.04
ii collaboraoffice-dict-nl 24.04.4-1 amd64 Dutch dictionary for Collabora Office 24.04
ii collaboraoffice-ure 24.04.4-1 amd64 UNO Runtime Environment
ii collaboraofficebasis-calc 24.04.4-1 amd64 Calc module for Collabora Office 24.04
ii collaboraofficebasis-core 24.04.4-1 amd64 Core module for Collabora Office 24.04
ii collaboraofficebasis-draw 24.04.4-1 amd64 Draw module for Collabora Office 24.04
ii collaboraofficebasis-en-gb 24.04.4-1 amd64 Language module for Collabora Office 24.04, language en_GB
ii collaboraofficebasis-en-us 24.04.4-1 amd64 Language module for Collabora Office 24.04, language en_US
ii collaboraofficebasis-extension-pdf-import 24.04.4-1 amd64 PDF import extension for Collabora Office 24.04
ii collaboraofficebasis-graphicfilter 24.04.4-1 amd64 Graphic filter module for Collabora Office 24.04
ii collaboraofficebasis-images 24.04.4-1 amd64 Images module for Collabora Office 24.04
ii collaboraofficebasis-impress 24.04.4-1 amd64 Impress module for Collabora Office 24.04
ii collaboraofficebasis-math 24.04.4-1 amd64 Math module for Collabora Office 24.04
ii collaboraofficebasis-nl 24.04.4-1 amd64 Language module for Collabora Office 24.04, language nl
ii collaboraofficebasis-ooofonts 24.04.4-1 amd64 3rd party free fonts for Collabora Office 24.04
ii collaboraofficebasis-ooolinguistic 24.04.4-1 amd64 Linguistic module for Collabora Office 24.04
ii collaboraofficebasis-writer 24.04.4-1 amd64 Writer module for Collabora Office 24.04
Logs
Attached below is the coolwsd
log (produced with journalctl -u coolwsd.service
)
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: kit-1637697-1637697 2024-06-24 17:49:18.690144 +0200 [ kitbroker_005 ] SIG Fatal signal received: SIGSEGV code: 128 for address: 0x0
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: Recent activity:
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: Session count: 2
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: view: 4, session 02d, rw, user: 'Dennis van Zuijlekom', loaded, type: text, lang: en-US
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: view: 5, session 038, rw, user: 'Tjerk', loading, type: text, lang: en-US
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: Commands:
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : ToolbarMode?Mode:string=Default - 2024-06-24 15:48:39
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : ToolbarMode?Mode:string=notebookbar_online.ui - 2024-06-24 15:48:39
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : ToolbarMode?Mode:string=Default - 2024-06-24 15:48:39
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 038: unoCommand : SidebarShow - 2024-06-24 15:48:39
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 038: unoCommand : Navigator - 2024-06-24 15:48:39
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : ToolbarMode?Mode:string=notebookbar_online.ui - 2024-06-24 15:48:39
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: save background= {}
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : Save - 2024-06-24 15:48:53
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : BasicShapes.diamond - 2024-06-24 15:49:00
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : BasicShapes.diamond - 2024-06-24 15:49:05
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : BasicShapes.pentagon - 2024-06-24 15:49:10
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 038: unoCommand : ChangeTheme - 2024-06-24 15:49:11
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 038: unoCommand : SpellOnline - 2024-06-24 15:49:11
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 038: unoCommand : SidebarShow - 2024-06-24 15:49:11
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 02d: unoCommand : BasicShapes.pentagon - 2024-06-24 15:49:15
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: session: 038: unoCommand : BasicShapes.pentagon - 2024-06-24 15:49:16
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: Backtrace 1637697 - kit startup of 24.04.4.1 9091043:
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /usr/bin/coolforkit(_ZN7SigUtil13dumpBacktraceEv+0x73)[0x61d383]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /usr/bin/coolforkit[0x61d7a5]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f12cb23b050]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN3vcl6Window24GetParentWithLOKNotifierEv+0xd)[0x7f12c8a5bb2d]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN3vcl6Cursor9LOKNotifyEPNS_6WindowERKN3rtl8OUStringE+0x3e)[0x7f12c89b150e]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN3vcl6Cursor10ImplDoHideEb+0x86)[0x7f12c89b1d86]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN3vcl6Window9SetCursorEPNS_6CursorE+0x2a)[0x7f12c8a56f4a]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(+0x1b2aca1)[0x7f12c6d2aca1]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN14SdrObjEditView15ModelHasChangedEv+0x4a9)[0x7f12c81742e9]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/../program/libswlo.so(+0x6e0683)[0x7f12bdce0683]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN9Scheduler22CallbackTaskSchedulingEv+0x1593)[0x7f12c8d975f3]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN14SvpSalInstance12CheckTimeoutEb+0x12b)[0x7f12c8f5620b]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN14SvpSalInstance9ImplYieldEbb+0x7d)[0x7f12c8f5653d]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN14SvpSalInstance7DoYieldEbb+0x15d)[0x7f12c8f5694d]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(+0x3bb0601)[0x7f12c8db0601]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_ZN11Application7ExecuteEv+0xa5)[0x7f12c8db0db5]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(+0x28184ba)[0x7f12c7a184ba]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(_Z10ImplSVMainv+0x61)[0x7f12c8db9ae1]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(soffice_main+0x9c)[0x7f12c7a3bdac]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /opt/collaboraoffice/program/libmergedlo.so(+0x2851ff5)[0x7f12c7a51ff5]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /usr/bin/coolforkit(_Z10lokit_mainRKSsS0_S0_S0_bbbbm+0x26cb)[0x5d681b]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /usr/bin/coolforkit[0x5a26fa]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /usr/bin/coolforkit(_Z18forkLibreOfficeKitRKSsS0_S0_+0xc1d)[0x5a41bd]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /usr/bin/coolforkit(_Z11forkit_mainiPPc+0x2331)[0x5a7c91]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f12cb22624a]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f12cb226305]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1637697]: /usr/bin/coolforkit[0x56c412]
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1637765 2024-06-24 17:49:18.710312 +0200 [ docbroker_005 ] ERR #42: Read failed, have 0 buffered bytes (ECONNRESET: Connection reset by peer)| net/Socket.hpp:1146
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1637765 2024-06-24 17:49:18.710370 +0200 [ docbroker_005 ] WRN #42: DocBroker [https%3A%2F%2Fnextcloud.acc.irealisatie.nl%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F22255_octvs8hl1iug] got disconnected from its Kit (1637697) unexpectedly. Closing| wsd/COOLWSD.cpp:3576
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1637765 2024-06-24 17:49:18.710411 +0200 [ docbroker_005 ] ERR Cannot save because CanSave::NoKit though NeedToSave::Yes_Modified. May have data loss, but must stop| wsd/DocumentBroker.cpp:2607
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636162]: frk-1636162-1636162 2024-06-24 17:37:08.364068 +0200 [ forkit ] INF Forkit initialization complete: setting log-level to [warning] as configured.| kit/ForKit.cpp:789
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636162]: frk-1636162-1636162 2024-06-24 17:49:18.710783 +0200 [ forkit ] WRN Successfully sent 'segfaultcount' message segfaultcount 1
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636162]: | kit/ForKit.cpp:344
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1636161 2024-06-24 17:49:18.710877 +0200 [ prisoner_poll ] TST UnitWSD [exitTest] (+0ms): ERROR: FAILURE: exitTest: TestResult::Failed: kit segfault| common/Unit.cpp:530
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1636161 2024-06-24 17:49:18.710902 +0200 [ prisoner_poll ] TST UnitWSD [exitTest] (+0ms): Dumping state| common/Unit.cpp:536
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1636161 2024-06-24 17:49:18.710917 +0200 [ prisoner_poll ] TST UnitWSD [endTest] (+0ms): Ending test by stopping SocketPoll [UnitWSD]: kit segfault| common/Unit.cpp:595
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1636161 2024-06-24 17:49:18.710924 +0200 [ prisoner_poll ] TST UnitWSD [endTest] (+0ms): ==================== Finished [UnitWSD] ====================| common/Unit.cpp:604
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1636161 2024-06-24 17:49:18.710927 +0200 [ prisoner_poll ] TST UnitBase [onExitTest] (+0ms): UnitWSD was the last test. Finishing FAILED| common/Unit.cpp:714
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1636161 2024-06-24 17:49:18.710933 +0200 [ prisoner_poll ] TST UnitWSD [onExitTest] (+0ms): Setting TerminationFlag as there are no more tests| common/Unit.cpp:720
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1636161 2024-06-24 17:37:08.383033 +0200 [ prisoner_poll ] TRC ppoll start, timeoutMicroS: 17981617 size 3| net/Socket.cpp:414
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1636096 2024-06-24 17:49:18.711046 +0200 [ coolwsd ] SIG User signal received: SIGUSR1
Jun 24 17:49:18 collabora-1.acc.rdo.prolocation.net coolwsd[1636096]: wsd-1636096-1637765 2024-06-24 17:49:18.711014 +0200 [ docbroker_005 ] WRN DocBroker [https%3A%2F%2Fnextcloud.acc.irealisatie.nl%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F22255_octvs8hl1iug stopped although have unsaved modifications: Broker: https%3A%2F%2Fnextcloud.acc.irealisatie.nl%3A443%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F22255_octvs8hl1iug pid: 1637697 has live sessions