The en_{US,GB}.dic files were in /opt/collaboraoffice/share/wordbook/ as you suggested, but they were only 441 bytes (46 lines). I removed the dict_path from my config, moved the small dictionaries out of the way and restarted collabora, but it didn’t make a performance difference.
I suspect your explanation as to why it’s slow (1B+ string comparisons) is probably correct and I just need to figure out where the dictionaries are coming from and why. And I’ve made some progress on that front. Time for another deep dive…
According to strace, the dictionary that’s being read is 551806 bytes:
openat(AT_FDCWD, "/lo/program/../share/wordbook", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 14
fstat(14, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents64(14, 0x33d62f50 /* 3 entries */, 32768) = 80
lstat("/lo/program/../share/wordbook/en_US.dic", {st_mode=S_IFREG|0644, st_size=551806, ...}) = 0
And this file size is consistent throughout the entire strace. Here’s the lstat calls for all dictionary files.
grep lstat .30591 | grep dic
lstat("/lo/program/../share/wordbook/en_US.dic", {st_mode=S_IFREG|0644, st_size=551806, ...}) = 0
lstat("/lo/program/../share/wordbook/en_US.dic", {st_mode=S_IFREG|0644, st_size=551806, ...}) = 0
lstat("/lo/program/../share/wordbook/en_US.dic", {st_mode=S_IFREG|0644, st_size=551806, ...}) = 0
lstat("/tmp/user/user/wordbook/standard.dic", {st_mode=S_IFREG|0644, st_size=45, ...}) = 0
lstat("/tmp/user/user/wordbook/standard.dic", {st_mode=S_IFREG|0644, st_size=45, ...}) = 0
lstat("/tmp/user/user/wordbook/standard.dic", {st_mode=S_IFREG|0644, st_size=45, ...}) = 0
lstat("/lo/program/../share/wordbook/en_US.dic", {st_mode=S_IFREG|0644, st_size=551806, ...}) = 0
lstat("/lo/program/../share/wordbook/en_US.dic", {st_mode=S_IFREG|0644, st_size=551806, ...}) = 0
lstat("/lo/program/../share/wordbook/en_US.dic", {st_mode=S_IFREG|0644, st_size=551806, ...}) = 0
lstat("/tmp/user/user/wordbook/standard.dic", {st_mode=S_IFREG|0644, st_size=45, ...}) = 0
Side note: the standard.dic seen above appears to be coming from /opt/cool/cache/ and it is an empty dictionary, having only the header (which is why it’s only 45 bytes). The path seems to indicate that this is my user-specific custom dictionary which came from the Nextcloud server.
But that file size for en_US.dic does not match up at all with what’s in /opt/collaboraoffice/share/wordbook/ (I put the small files back after running my test).
ls -la /opt/collaboraoffice/share/wordbook/
total 20
drwxr-xr-x 2 root root 4096 Jul 8 10:15 .
drwxr-xr-x 29 root root 4096 Jul 2 17:02 ..
-rw-r--r-- 1 root root 441 Jun 14 17:32 en-GB.dic
-rw-r--r-- 1 root root 441 Jun 14 17:32 en-US.dic
-rw-r--r-- 1 root root 2804 Jun 14 17:25 technical.dic
The file size seen in strace almost matches up with the dictionary found in /opt/collaboraoffice/share/extensions/dict-en/
ls -l /opt/collaboraoffice/share/extensions/dict-en/en_US.dic
-rw-r--r-- 1 root root 551762 Jun 14 17:32 /opt/collaboraoffice/share/extensions/dict-en/en_US.dic
However, the strace data shows that /opt/collaboraoffice was overridden by a mount point.
mount("/opt/collaboraoffice", "/opt/cool/child-roots/30551-8e4abf70/l1s6HHGMp1Dz1vrg/lo", NULL, MS_MGC_VAL|MS_BIND|MS_REC, NULL) = 0
...
mount("/opt/collaboraoffice", "/opt/cool/child-roots/30551-8e4abf70/l1s6HHGMp1Dz1vrg/lo", 0xb6c3ba, MS_RDONLY|MS_NOSUID|MS_NODEV|MS_REMOUNT|MS_BIND, NULL) = 0
And the file size does match up exactly with a file on disk under /opt/cool/child-roots/
ls -l /opt/cool/child-roots/30551-8e4abf70/tmp/sharedpresets/shared-https_nextcloud.ucimc.org%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Fsettings-6a45ec7bdf7aa696d89c3cf3f4/wordbook/en_US.dic
-rw-r--r-- 1 cool cool 551806 Jul 8 10:35 /opt/cool/child-roots/30551-8e4abf70/tmp/sharedpresets/shared-https_nextcloud.ucimc.org%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Fsettings-6a45ec7bdf7aa696d89c3cf3f4/wordbook/en_US.dic
The latter dictionary appears to be the same as the former, only with a header added.
head /opt/collaboraoffice/share/extensions/dict-en/en_US.dic
49568
0/nm
0th/pt
1/n1
1st/p
1th/tc
2/nm
2nd/p
2th/tc
3/nm
vs
head /opt/cool/child-roots/30551-8e4abf70/tmp/sharedpresets/shared-https_nextcloud.ucimc.org%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Fsettings-6a45ec7bdf7aa696d89c3cf3f4/wordbook/en_US.dic
OOoUserDict1
lang: <none>
type: positive
---
49568
0/nm
0th/pt
1/n1
1st/p
1th/tc
I feel like this is substantial progress, even though it’s not yet solved. I really appreciate the succinct answers on things like why the .aff files weren’t being read, why the CPU was pegged, etc.
Because I found the mount points so confusing, I decided to disable the jail and get a trace of that. My document loaded in the expected <= 2 seconds and spell checking was working! I don’t understand why, and I don’t really want to run it without a chroot, but I’m very excited to have stumbled upon this fact. My straces didn’t capture as much data in this configuration. Apparently this changes the interprocess communications in a way that only gets me a few dozen lines of strace output (no calls to lstat nor open, just a bunch of polling). That’s not what I expected, but it’s also not terribly surprising.
To be clear, I was only disabling the jail in coolwsd.xml. The restrictions from systemd are still in place.
If you have any pointers to source code for me to take a look at, I’m familiar with C++ (or at least I was until they started adding all sorts of wild new stuff over the past couple decades 
). I’m also willing to put a custom build of coolforkit-ns on the server temporarily to do additional debugging if necessary.
Once I figure this out, the Ansible role will be immediately updated in whatever way is needed, that’s for sure! I plan on destroying and re-building the entire Collabora VM to test it too, because I’ve done a ton of things manually on this VM trying to troubleshoot this issue.