Daily Lockups - kswapd & Memory leaks

3T_Ed · 25 Sept 2021, 13:08

I have had exactly the same issues for quite some time using OnePlus 3T, even before using Webber app. As it seems it's not limited to a specific device or brand. The 'lock-ups' occur like 3-4 times a day.

I can't figure out what causes it either, not with 100% certainty anyways.

If I would only know how to capture it I would be happy to contribute to get this matter solved.

StandUpMobile · 27 Sept 2021, 22:00

@3t_ed If you enable USB debugging, next time it happens, you can plug into a computer (with adb tools) and run adb shell. This will bring up a terminal: run "Top" then hit "m" twice to see memory and swap % use.

StandUpMobile · 27 Sept 2021, 22:08

One further update:

sudo cat /proc/sys/vm/swappiness returns 100 ...

Is this parameter used in UT? If so, that doesn't make sense.

The default value of vm.swappiness is 60 and represents the percentage of the free memory before activating swap. The lower the value, the less swapping is used and the more memory pages are kept in physical memory.

I set mine to 20 ... and my test of opening a ton of webapps starts using swap much later.... and I haven't hit lockup yet. Not sure how to persist it though...

sudo mount -o remount,rw /
sudo sysctl vm.swappiness=20

domubpkm · 28 Sept 2021, 08:04

@standupmobile said in Daily Lockups - kswapd & Memory leaks:

sudo mount -o remount,rw /
sudo sysctl vm.swappiness=20

On the Volla the value 20 does not survive a reboot. vm.swappiness returns to its default value of 60.

StandUpMobile · 28 Sept 2021, 17:04

A bit too quick to declare success ... I locked up with 100% swap usage again today while my swappiness was set to 20. Still think it is a beneficial change for the life of the emmc ... however, issue persists.
@domubpkm ... Not sure yet how to persist this across reboots ... going to reach out to porters to see where this setting is established.

StandUpMobile · 29 Sept 2021, 15:45

So turns out the swap showing on our Pixel3's is ZRam (Fake swap on memory that is compressed). The 100 setting is appropriate for this setup. Will keep digging around for some way to prevent 100% use ... (which is the source of my lockups).

StandUpMobile · 4 Oct 2021, 16:32

So I built a routine that allowed me to reproduce the lockups (where the swap use shown in top (zram) hits 100%, and memory is 95+% used) - it basically involves opening a bunch of webapps / apps that I use; but all at once in fast sequence. The camera is one app that needs a bunch of memory ... and can easily push the device over the edge).

I then tweaked memory control parameters as per various suggestions found online and repeated the test. I was able to get a lot further before lockup with the settings below. (The phone uses less zram as I progress through my test, I guess ultimately pushing back the time to lockup). I haven't noticed any performance hit (purely subjective), and I can run many days now without lockup. On the one occasion where I did lockup, recovery was faster (rather than 1-3 mins, it was ~10 seconds)l; however I did notice one of my open apps was killed (white page with green triangle saying refresh). Haven't figured out how to set this across boots - although my reboots are a lot less frequent!

Disclaimer: This is for the pixel3a - that has 4GB of RAM and ~ 1.6GB zram setup. Based on my reading, I think some of these settings are very specific towards hardware (memory, cpu, swap vs. zram) on the device... also, I am a hack wrt linux and have NO other experience with these settings:)

sudo sysctl vm.min_free_kbytes=29620 &&
sudo sysctl vm.vfs_cache_pressure=200 &&
sudo sysctl vm.dirty_background_ratio=3 &&
sudo sysctl vm.dirty_ratio=40 &&
sudo sysctl vm.admin_reserve_kbytes=16384 &&
sudo sysctl vm.swappiness=80

Hope this helps ...

3T_Ed · 31 Mar 2022, 05:57

@standupmobile
Is this problem solved? No more freezes?

fredldotme · 31 Mar 2022, 17:34

@3t_ed yes, at least we found out what happened with apps locking up everything, while in reality there was no low-memory killer active. I enabled it and it's available in devel or even release candidate already.

3T_Ed · 31 Mar 2022, 19:37

@fredldotme
Great! Thank you for letting me know. That's good news so thank you for sorting out this issue. I'm still on stable channel but I assume we'll notice with the next OTA?

domubpkm · 1 Apr 2022, 02:49

Thank you @fredldotme !
What are the gitlab or github links that fix the problem(s). I don't seem to see this in OTA-23 project anyway.

StandUpMobile · 7 Apr 2022, 21:28

@3t_ed To elaborate a bit further: First, Alfred applied those changes to the port. One OR 2 did not stick properly and rather than chasing down the issue further, he remove zram completely. So the dev version of p3a is no longer using any swap. There are no more lockups, but (IMHO) there is a little less bandwidth for multitasking, especially after a few days of uptime (where presumably software memory leaks have chewed into the avail 4GB of ram. All in all a big improvement ... would be good if some other could test and chime in their thoughts.

https://github.com/fredldotme/android_device_google_bonito/commit/cea569eb00838ad9cb6e16471b69510b0f0e0e47

https://github.com/fredldotme/android_device_google_bonito/commit/f4df3991231cf4f7aab62584a9289a7642b88e9c

3T_Ed · 8 Apr 2022, 10:07

@standupmobile
Thanks. I followed up on your advice and am now on RC 673 to check if daily lockups have gone and check for lagging in multitasking. Will report back after the weekend.

fredldotme · 9 Apr 2022, 18:39

The actual fix was this one: https://github.com/fredldotme/android_kernel_google_bonito/commit/944dd258c369ed224b37a9fa439a64c919ad4f27

domubpkm · 9 Apr 2022, 19:09

@fredldotme On which phones does this fix apply? The Volla too ? If yes, on the current RC ? Thanks in advance to clarify.

fredldotme · 10 Apr 2022, 11:42

@domubpkm Well since currently this is a device specific change it doesn't apply as-is to Volla, I actually don't know if that one has a Low Memory Killer activated.

In principle each device maintainer needs to make sure that low-memory situations are dealt with properly, one way or another. We currently don't have a unified approach to that.

domubpkm · 4 Oct 2022, 12:20

@fredldotme Ok. Thank you very much for clarifying this. I am experiencing on the Volla, as already mentioned, problems in multi-tasking. All I can do is to point your remark to the different issues where I mentioned the problem. The rest is out of my knowledge/competence.

Opened/probably concerned issues are :
https://github.com/ubports/morph-browser/issues/495

https://github.com/HelloVolla/ubuntu-touch-beta-tests/issues/101
if you want to put some of your useful comments inside.

3T_Ed · 10 Apr 2022, 17:09

@fredldotme @StandUpMobile @domubpkm

For two days I have been testing this fix on RC and I couldn´t be happier. The daily lockups have gone and the OnePlus 3T is as snappy and smooth like when UT was first installed. Multitasking isn´t a problem either. Only had one freeze, I guess it was just me pushing the device to the limit.

Thanks a ton especially to @fredldotme for making me appreciate my device running UT again as it's supposed to.