VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed
-
I almost do not use Waydroid, the crashes and freezes happen when waydroid is not started and have not been started since boot.
Well I've investigated and it seems to be an out of memory problem.
Because I realize that now if I start a few apps (like the Morph browser + whatsweb + Signal UT ) then I rapidly run out of Ram AND Swap. (I can see it in top)
Then my question is why did it not happen before? Is the memory usage is higher in 24.04-1.1 compared to 24.04-1.0 and if so why ?
And secondly why do we have so few swap (2 Go) if there are Out of Memory problems? Or maybe we had more swap in 24.04-1.0 and it was reduced ?
Is it possible to increase the swap?
Edit: By the way I've reported the problem here. https://gitlab.com/ubports/porting/reference-device-ports/android11/volla-phone-22/volla-mimameid/-/issues/24
-
Ps: I also just experimented a black screen when receiving a call. In Adb I could see the phone was not in out of memory at that time. The phone was vibrating and I could not turn on the screen to see the call.
Although previously in this boot I had an Out of Memory problem, it freezed temporarilly, but the system recovered, so I wonder if some important system processes may have been killed in the process.
Honestly I'm wondering whether I should downgrade, because this starts to be a real problem.
-
@pparent said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
I can see it in top
When using top, did you have enough time to notice if one particular application's Ram use was growing without limit ?
To elaborate, here is the grand total of my experience with OOM.
First generally with Linux (desktop and servers): it has always happened in the context of a 'rogue' application, never because I loaded too much the system. If I was loading too many apps on a given Linux system, I could always unload one of them and get back to a more normal behaviour; when a rogue application was running, I could see this application eating more and more Ram and if I could not kill it fast enough (before I lost control of the console to use kill -9) the system was all but dead, only a hard reset could get it back. Needless to say (it's a pretty well known fact of using Linux) the system OOM handler was never of any help.
Then with UT
First I have found a 'rogue' app that can reliably hang my phone. This was the first experience of it, and I could see the very same behaviour using top (it is hanging progressively, it takes a few minutes to eat all my 4 GB of swap)
Then with the dialer: this has happened very recently, while I was trying to experiment with some advanced particularities of it and suddenly the system became sluggish and the UI unusable, connecting with ssh I could see with top that lomiri-dialer-app was growing more and more and the swap was eaten. The ssh session got soon unresponsive (the phone UI itself was already gone so it was another hard reboot. I am sure that at this point the dialer was the only application active, so it's the dialer or more probably one of its libraries that is the culprit. As the dialer use quite a lot of libraries, it's not helping much, but it's possible that one of the apps you were using has triggered the same bad code path. Unfortunately I could not repro it with the dialer.
I don't know if you have noticed it, but in the system there is something of a special OOM handling, application can submit a specific OOM profile when starting. Unfortunately I have not seen it any more useful than the pretty useless kernel default handling, the rogue applications can do their evil deeds without being bothered by it.
A last detail: on my FP5 (I'd suprised if your Volla was different because it's using half of your Ram like for my FP5, it's a giveaway), the swap handling is not the 'classical' sort, it's the more advanced type of 'compressed Ram'. I don't think it's a novelty of 24.04.1.1.
-
About the dialer, I cannot exclude that the app was opened (or had been used) at the time of every crash.
It's true also that I now use VoLTE, that I did not use before 24.04.1 , so it could have a impact on the RAM(?)
It's also true that I cannot reproduce the bug by opening a lot of apps straight after a reboot.
-
@pparent said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
if some important system processes may have been killed in the process.
Well, the OOM killer is legendary for it's lack of discrimination. If the system recovered somehow, there should be some trace in the system log.
I know from experience that when the system has to be rebooted after an OOM, there is no trace at all in the sytem log (and that is by itself a very major problem as it makes diagnostic impossible) -
@gpatel-fr for me when my system black screens all i can do is type, i still have access to ssh though and can run commands from my laptop.
-
@killclique said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
when my system black screens
See my reply in your dedicated thread, I think that your problem may be specific to the Fairphone 5 and so be different from the problem as explained by @pparent (who said that the problem occurred without any use of Waydroid).
-
Here is a situation where I'm close to Out-of-Memory, (my phone freezed for 15s and then recovered). If I reboot the phone and open the same apps I will have a lot more swap availiable. Surely if I keep using it like that, it will crash at some point. I have not made a call today, but I've taken pictures. I don't understand why the swap use is so much higher than normal because top does not show any processes that use abnormal amount of Ram. Could the problem come from swap itself?
top - 13:31:14 up 1 day, 25 min, 2 users, load average: 21.65, 22.07, 20.99 Tasks: 640 total, 1 running, 587 sleeping, 2 stopped, 50 zombie %Cpu(s): 12.1 us, 4.9 sy, 0.1 ni, 82.1 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st MiB Mem : 81.1/3498.4 [ ] MiB Swap: 82.5/1924.1 [ ] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3415 phablet 20 0 6229920 388144 303092 S 0.7 10.8 32:31.70 lomiri 22255 phablet 20 0 196.4g 162708 60184 S 0.3 4.5 0:26.95 signal-desktop 17763 phablet 20 0 3545708 152084 145080 S 0.3 4.2 1:46.33 qmlscene 17888 phablet 20 0 14.0g 130968 25928 S 0.7 3.7 4:06.44 QtWebEngineProc 22374 phablet 20 0 2414496 112092 61980 S 2.0 3.1 0:11.02 lomiri-camera-a 22131 phablet 20 0 204.4g 105492 23376 S 7.9 2.9 0:42.14 signal-desktop 4854 phablet 20 0 2283820 63164 59408 S 0.3 1.8 1:39.14 maliit-server 2551 1047 20 0 12.7g 57016 28064 S 99.3 1.6 3:21.46 camerahalserver 22343 phablet 20 0 1977936 42924 41036 S 2.3 1.2 0:05.77 qmlscene 22221 phablet 20 0 32.5g 40492 38812 S 0.0 1.1 0:05.71 signal-desktop 5103 phablet 20 0 2132348 40060 40060 T 0.0 1.1 1:53.34 lomiri-system-s 5222 phablet 20 0 1919912 39960 39960 T 0.0 1.1 0:34.41 lomiri-dialer-a 21580 phablet 20 0 293540 30952 30472 S 0.0 0.9 0:17.62 Xwayland 2520 1013 20 0 10.9g 13532 12296 S 8.9 0.4 0:16.99 minimediaservic 712 root 19 -1 83872 11136 10304 S 0.0 0.3 0:52.47 systemd-journal 2148 root 20 0 1214404 9508 7156 S 1.3 0.3 16:42.81 lomiri-system-c 1 root 20 0 24248 9496 5516 S 0.0 0.3 0:57.16 systemd 22745 phablet 20 0 178464 9164 7852 S 0.0 0.3 0:00.08 mtp-server 22738 phablet 20 0 326744 9056 8216 S 0.0 0.3 0:00.14 adbd 22755 root 20 0 1155836 8052 5828 S 0.0 0.2 0:00.05 adbd-pam-sessio 2465 phablet 20 0 22468 7728 5136 S 0.0 0.2 0:16.46 systemd 4858 phablet 20 0 2666196 7260 4108 S 0.0 0.2 0:31.06 lomiri-push-ser 1654 root 20 0 2286604 6788 1184 S 0.0 0.2 0:58.29 snapd 22773 phablet 20 0 57208 5608 3388 R 1.6 0.2 0:01.15 top 4855 phablet 20 0 294540 5496 4320 S 0.0 0.2 0:03.85 mtp-server-usb- 5261 phablet 20 0 350504 5320 4488 S 0.0 0.1 0:06.35 zeitgeist-fts 4857 phablet 20 0 405916 5188 4340 S 0.0 0.1 0:16.78 lomiri-indicato 22763 phablet 20 0 53404 5080 3584 S 0.0 0.1 0:00.08 bash -
Actually there is a very strange phenomenon as I navigate in Signal-Destkop (without doing anything else that navigating in the menu) then the Ram used by signal desktop increases rapidly:
%Cpu(s): 9.9 us, 2.1 sy, 0.0 ni, 87.5 id, 0.1 wa, 0.0 hi, 0.4 si, 0.0 st MiB Mem : 3498.4 total, 368.6 free, 2913.7 used, 793.1 buff/cache MiB Swap: 1924.1 total, 752.5 free, 1171.7 used. 584.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22255 phablet 20 0 196.7g 649492 519052 S 46.4 18.1 1:19.36 signal-desktop 3415 phablet 20 0 6140780 418372 295348 S 10.5 11.7 32:59.29 lomiri 17763 phablet 20 0 3553392 198512 160808 S 0.0 5.5 1:50.02 qmlscene 17888 phablet 20 0 14.0g 159116 38372 S 0.0 4.4 4:15.90 QtWebEngineProc 22131 phablet 20 0 204.6g 134476 32876 S 1.6 3.8 1:13.70 signal-desktop 22974 phablet 20 0 2640604 108728 63600 T 0.0 3.0 0:05.26 lomiri-clock-ap 5222 phablet 20 0 1919932 91832 52740 T 0.0 2.6 0:35.71 lomiri-dialer-a 4854 phablet 20 0 2283820 71756 60872 S 0.0 2.0 1:40.69 maliit-server 22221 phablet 20 0 32.5g 60192 57196 S 21.7 1.7 0:23.63 signal-desktop 21580 phablet 20 0 277844 52512 50644 S 2.6 1.5 0:21.82 Xwayland 5103 phablet 20 0 2132348 40060 40060 T 0.0 1.1 1:53.34 lomiri-system-s 2551 1047 20 0 11.6g 14280 1848 S 0.0 0.4 7:02.53 camerahalserver 4858 phablet 20 0 2666196 8568 4996 S 0.3 0.2 0:31.43 lomiri-push-ser 2148 root 20 0 1214576 7852 5532 S 3.6 0.2 16:55.07 lomiri-system-c 1832 root 20 0 364824 6392 2300 S 0.0 0.2 1:43.98 upowerd 1654 root 20 0 2286604 6372 0 S 0.0 0.2 0:58.48 snapd 1 root 20 0 24248 6084 2104 S 0.0 0.2 0:57.18 systemd 3241 phablet 20 0 1175228 5704 3172 S 0.0 0.2 0:52.98 media-hub-serve 22745 phablet 20 0 178464 5524 4212 S 0.0 0.2 0:00.08 mtp-server 1777 polkitd 20 0 386452 5184 2404 S 0.0 0.1 0:54.95 polkitd 712 root 19 -1 83872 4988 4060 S 0.0 0.1 0:52.84 systemd-journal 2465 phablet 20 0 22468 4752 2156 S 0.0 0.1 0:16.53 systemd 1576 message+ 20 0 12588 4696 1608 S 0.3 0.1 5:39.31 dbus-daemon 22738 phablet 20 0 326744 4452 3840 S 0.0 0.1 0:00.27 adbd 22946 phablet 20 0 57096 4436 2280 R 1.6 0.1 0:02.28 top 4685 phablet 20 0 465292 4360 1556 S 0.0 0.1 0:03.44 libertined 22201 phablet 20 0 32.4g 4056 3468 S 0.0 0.1 0:00.34 signal-desktop 4805 phablet 20 0 688412 4040 3228 S 0.0 0.1 0:09.09 lomiri-content-But then if I go to Whatsweb, the Ram of Signal-Desktop decrease, and the ram of Whatsweb or rather QtWebEngineProc starts to increase as I navigate
%Cpu(s): 33.6 us, 8.9 sy, 0.1 ni, 56.2 id, 0.0 wa, 0.0 hi, 1.2 si, 0.0 st MiB Mem : 3498.4 total, 152.5 free, 2967.3 used, 608.5 buff/cache MiB Swap: 1924.1 total, 828.9 free, 1095.2 used. 531.1 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17888 phablet 20 0 14.4g 480044 70424 R 194.7 13.4 6:20.35 QtWebEngineProc 3415 phablet 20 0 6140780 413956 296784 S 21.4 11.6 33:32.10 lomiri 17763 phablet 20 0 3765040 351616 273648 S 67.8 9.8 2:44.29 qmlscene 22255 phablet 20 0 196.3g 286284 143608 S 0.3 8.0 2:21.52 signal-desktop 22131 phablet 20 0 204.3g 111528 18760 S 0.0 3.1 1:15.51 signal-desktop 22974 phablet 20 0 2640604 94748 49620 T 0.0 2.6 0:05.26 lomiri-clock-ap 5222 phablet 20 0 1919932 86168 47080 T 0.0 2.4 0:35.71 lomiri-dialer-a 23243 phablet 20 0 9860.2m 70140 46040 S 0.0 2.0 0:00.82 QtWebEngineProc 4854 phablet 20 0 2283820 67364 56404 S 0.3 1.9 1:40.94 maliit-server 22221 phablet 20 0 32.5g 57860 55796 S 0.0 1.6 1:22.57 signal-desktop 21580 phablet 20 0 277844 53788 51232 S 0.0 1.5 0:27.78 Xwayland 23196 phablet 20 0 917052 48556 40872 S 0.0 1.4 0:00.22 QtWebEngineProc 5103 phablet 20 0 2132348 40060 40060 T 0.0 1.1 1:53.34 lomiri-system-s 1654 root 20 0 2286604 8428 1628 S 0.0 0.2 0:58.64 snapd 2551 1047 20 0 11.6g 8152 1168 S 0.0 0.2 7:02.53 camerahalserver 712 root 19 -1 83872 7700 6760 S 0.0 0.2 0:53.07 systemd-journal 1 root 20 0 24248 7500 3492 S 0.0 0.2 0:57.44 systemd 2148 root 20 0 1214576 6656 4568 S 16.8 0.2 17:11.24 lomiri-system-c 1832 root 20 0 364824 6372 2432 S 0.0 0.2 1:44.27 upowerd 2465 phablet 20 0 22468 5344 2740 S 0.0 0.1 0:16.70 systemd 22201 phablet 20 0 32.4g 5320 4700 S 0.0 0.1 0:00.40 signal-desktop 3241 phablet 20 0 1175228 4824 2144 S 0.0 0.1 0:52.98 media-hub-serve 23134 phablet 20 0 244000 4688 3368 S 0.0 0.1 0:00.06 mtp-server 1576 message+ 20 0 12588 4508 1348 S 0.0 0.1 5:39.96 dbus-daemon 23141 root 20 0 1155836 4452 2264 S 0.0 0.1 0:00.05 adbd-pam-sessio 1777 polkitd 20 0 386452 4116 1348 S 0.0 0.1 0:55.05 polkitd 4685 phablet 20 0 465292 4052 1248 S 0.0 0.1 0:03.44 libertined 23159 phablet 20 0 57096 4028 1876 R 2.0 0.1 0:02.68 topIs all this normal?
-
Ok I think I've found an action that make Ram/Swap disappear magically: import a picture from camera in contentHUB: take a picture and import it directly in contentHub. I've made it various times it a row, until the swap reached 0, and the phone crashed. Here is the last top I got (it was running in background, until it crashed at the same time as the phone interface).
Tasks: 633 total, 1 running, 580 sleeping, 2 stopped, 50 zombie %Cpu(s): 8.8 us, 22.7 sy, 0.0 ni, 32.2 id, 35.1 wa, 0.0 hi, 1.2 si, 0.0 st MiB Mem : 3498.4 total, 359.3 free, 2822.9 used, 250.4 buff/cache MiB Swap: 1924.1 total, 3.1 free, 1921.0 used. 675.5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3415 phablet 20 0 6163912 373664 279220 S 6.8 10.4 34:50.99 lomiri 17763 phablet 20 0 4100244 290944 173152 S 9.0 8.1 4:20.49 qmlscene 17888 phablet 20 0 14.4g 160896 9724 S 9.3 4.5 9:20.91 QtWebEngineProc 24419 phablet 20 0 1867740 83372 49932 S 12.2 2.3 0:07.12 lomiri-camera-a 22255 phablet 20 0 196.3g 68524 24476 S 2.9 1.9 4:10.62 signal-desktop 4854 phablet 20 0 2283820 50280 48904 S 0.0 1.4 1:42.24 maliit-server 5103 phablet 20 0 2132348 40060 40060 T 0.0 1.1 1:53.34 lomiri-system-s 22974 phablet 20 0 2640604 39232 39232 T 0.0 1.1 0:05.26 lomiri-clock-ap 24521 phablet 20 0 9734284 33260 0 S 5.5 0.9 0:00.94 QtWebEngineProc 22221 phablet 20 0 32.5g 24884 23172 S 0.0 0.7 2:05.98 signal-desktop 21580 phablet 20 0 278376 20920 20100 S 0.0 0.6 0:35.24 Xwayland 2551 1047 20 0 11.8g 17060 2724 S 21.5 0.5 8:33.31 camerahalserver 22131 phablet 20 0 204.3g 10172 0 S 0.6 0.3 1:36.64 signal-desktop 3241 phablet 20 0 1888500 3672 88 S 0.0 0.1 0:53.64 media-hub-serve 4857 phablet 20 0 405916 3272 2396 S 1.0 0.1 0:18.51 lomiri-indicato 4858 phablet 20 0 2666196 2948 0 S 0.3 0.1 0:34.58 lomiri-push-ser 2817 phablet 20 0 12140 2752 0 S 0.0 0.1 0:27.39 dbus-daemon 23320 phablet 20 0 57096 2584 1340 R 2.3 0.1 0:12.16 top 1576 message+ 20 0 12588 2040 0 S 0.3 0.1 5:42.26 dbus-daemon 2148 root 20 0 1214388 1948 88 S 3.2 0.1 17:47.24 lomiri-system-c 2807 phablet 9 -11 3893872 1660 84 S 0.0 0.0 3:10.13 pulseaudio 1832 root 20 0 364824 1616 0 S 0.3 0.0 1:45.96 upowerd 4805 phablet 20 0 688412 1528 0 S 0.0 0.0 0:10.97 lomiri-content- 2465 phablet 20 0 22468 1516 56 S 0.0 0.0 0:17.06 systemd 1 root 20 0 23992 1500 0 S 0.3 0.0 0:58.76 systemd 2368 system 20 0 10.5g 1496 96 S 0.3 0.0 0:09.51 mtkpower@1.0-se 1107 root 20 0 10.5g 1196 532 S 1.9 0.0 0:43.22 init 2520 1013 20 0 11.0g 952 132 S 4.5 0.0 0:44.70 minimediaservicAt reboot with the same apps opened I have a lot more "free" swap:
MiB Mem : 3498.4 total, 149.3 free, 2738.0 used, 889.3 buff/cache MiB Swap: 1924.1 total, 1833.6 free, 90.5 used. 760.4 avail Mem -
@pparent said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
196.4g
oh, the Ram eater !
Holmes, the game is afoot
-
@gpatel-fr said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
oh, the Ram eater !
Holmes, the game is afootHolmes would say that Signal-Desktop sure is an easy culprit but the easy culprit rarely is the real culprit!
In this case Signal has an alibi: I've just been able to reproduce the bug, without starting signal-desktop since boot.
It can be done more or less reproducibility by playing with the camera, and importing pictures to Whatsweb directly from the camera app in contentHub a bunch of times, and recording videos.
But also a new clue arrives in the investigation, I've realized that actually if I give the system a lot of time, like 5 - 15 minutes, without rebooting it, it seems to end up recovering (at least sometime, I will see if it is reproductible), and after the recovery free swap recovers to more or less normal levels in top ( 1268 ) .
-
@pparent said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
It can be done more or less reproducibility by playing with the camera, and importing pictures to Whatsweb directly from the camera app in contentHub
could you play a little more with 'top' using 'f' and 's' to sort by VIRT or RES to see what is growing exactly, the default sort (Cpu) not being of much use here. If signal-desktop is not the culprit in this case, it may be whatsweb for example.
Note that camerahalserver (that I don't seem to have on my FP5) is part of the Android container so it's not very instructive by itself: on my FP5 all the Android processes have also high level of VIRT (10-12Gb) without causing any swap use.
-
@gpatel-fr said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
could you play a little more with 'top' using 'f' and 's' to sort by VIRT or RES to see what is growing exactly, the default sort (Cpu) not being of much use here. If signal-desktop is not the culprit in this case, it may be whatsweb for example.
If you look at the last top I posted I sorted with %MEM which is equivalent than sorted by RES. And there is seemingly no process that eats too much RAM. I've never seen in a top any processes consuming outrageous amount of RAM.
I've also done a df -h , to see if a ramfs could use too much space but I've seen nothing.
As you tell me that "the swap handling is not the 'classical' sort, it's the more advanced type of 'compressed Ram'", I wonder if the swap itself could be buggy?
-
@pparent said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
I've never seen in a top any processes consuming outrageous amount of RAM.
That's very strange. I had kept a copy of top when the dialer had crashed my phone and it was clear that there was a culprit:
top - 21:30:13 up 12:42, 2 users, load average: 34,25, 47,73, 35,64 Tasks: 618 total, 16 running, 601 sleeping, 0 stopped, 1 zombie %Cpu(s): 0,5 us, 92,4 sy, 0,0 ni, 1,3 id, 1,7 wa, 2,3 hi, 1,8 si, 0,0 st MiB Mem : 7294,9 total, 62,8 free, 7216,6 used, 64,6 buff/cache MiB Swap: 4096,0 total, 0,0 free, 4096,0 used. 78,3 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4265 phablet 20 0 396220 1340 0 R 60,9 0,0 3:03.44 mission-control 130 root 20 0 0 0 0 R 44,0 0,0 6:10.27 kswapd-1:0 3334 root 20 0 0 0 0 R 38,5 0,0 6:10.26 kswapd0:1 4396 phablet 20 0 4640104 29844 17684 R 34,6 0,4 9:03.45 lomiri 2055 root 20 0 17868 616 0 R 34,3 0,0 1:35.34 wpa_supplicant 4212 phablet 20 0 2043428 1404 0 S 34,3 0,0 3:20.83 nuntium 4888 phablet 20 0 407480 1544 0 R 34,3 0,0 2:04.60 lomiri-indicato 1 root 20 0 25684 3884 0 R 33,0 0,1 2:36.44 systemd 1399 root 20 0 28280 804 0 R 32,4 0,0 1:09.57 systemd-udevd 4914 phablet 20 0 1869152 3080 0 S 31,9 0,0 2:53.33 ciborium 23599 phablet 20 0 9934948 5,0g 1936 R 30,5 70,4 3:16.95 lomiri-dialer-a 4429 phablet 20 0 957680 612 0 R 29,1 0,0 0:48.45 evolution-addre 24185 root 20 0 155816 29852 84 R 28,8 0,4 0:46.14 lomiri-system-cAlso do you see kswapd taking a lot of Cpu like in my top ?
@pparent said in VP22 Upgrade from 24.04-1.0 to 24.04-1.1 failed:
the swap itself could be buggy?
It's probably the same software and setup for all ports since it's not linked to Android.
