Idea: overlayfs for user terminal
-
@pparent trust me, it happened, the issue is way more complicated than you think. If you can think of it, it likely has been tried in the last 8 years (i call this rule 53)
Also i edited my answer a bit with some more tidbits, like you are doing now, feel free to check. But i really reccomend joining us on matrix or telegram
People who are calling Ubuntu Touch "locked down" due to the immutable nature are unfortunately too focussed on
aptthere is at this moment 2 alternative that lets you install the entire repetoire of linux packages all 120,000 of them (apt in contrast has access to about 80,000 of them) if you so wish—- see the links i included in my answer. You have a lot of control under UT, it just a different paradigm that departs from traditional package managers.That is why we're going to solve this with "make apt usable out of the box" but rather with "What are people trying to install, what are they trying to achieve?"
This "i can apt therefore locked down" is a good example of the XY problemThe user is trying to solve a problem, (say use vim) think they have to use apt to install a package to do it (sudo apt install vim) and then fight the system to make apt work
while there may be a better way (crackle install vim) -
@Fuseteam said in Idea: overlayfs for user terminal:
If you can think of it, it likely has been tried in the last 8 years (i call this rule 53)
Strangely a thing like porting Signal-Desktop as a click package, and make its interface responsive, had not been tried in the last 8 years while being relatively easy and fast to do, and many users needing Signal (or even leaving explicitly the ecosystem for the lack of it) !
People who are calling Ubuntu Touch "locked down" due to the immutable nature are unfortunately too focussed on apt there is at this moment 2 alternative that lets you install the entire repetoire of linux packages if you so wish—- see the links i included in my answer. You have a lot of control under UT, it just a different paradigm that departs from traditional package managers
Once again I'm not the one to convince, potential new users are.
The user is trying to solve a problem,
Not necessarily, the user might simply be trying to feel good, and feel at home, knowing they can do the same thing and have the same freedom as in their desktop distro. Feeling can be as important as functionalities for a new user to adopt something.
Well anyway I think what I propose might possibly be tried or even deployed without any system modification. Maybe with an alternative terminal app, (or even simply by modifying bash-rc and keep the same terminal app). So we'll see if I have time to try it out some day.
-
@pparent said in Idea: overlayfs for user terminal:
-Can I interact with files from the host system with tools from the libertine container?
when inside the container you are confined, you can interact with directories shared with the main computer, that means data directories, to the exclusion of configuration of applications external to the container. That may or not be a good thing if the primary concern is the security of hapless users or the liberty of the brave developer (sometimes the 2 types are the same as it is the case for developers caught in supply chain traps
)(py312) phablet@ubuntu-phablet:~$ cd .config (py312) phablet@ubuntu-phablet:~/.config$ ls 8192.alaskalinuxuser chromium evolution google-chrome maliit.org music.ubports steps.s60w79 themeswitch.danfro vivaldi address-book.ubports clock.ubports filemanager.ubports libaccounts-glib messaging-app.ubports notes.ubports sterlingpdf uhome.alaskalinuxuser weather.ubports BraveSoftware connectivity-service forums-ubports-com.webber lomiri microsoft-edge openstore.openstore-team systemd 'Unknown Organization' calendar.ubports dconf gallery.ubports Lomiri mirvncserver.abmyii procps telephony-service user-dirs.dirs camera.ubports dialer-app.ubports glib-2.0 lomiri-ui-toolkit morph-browser pulse terminal.ubports ut-tweak-tool.sverzegnassi (py312) phablet@ubuntu-phablet:~/.config$ libertine-container-manager exec -i noble -c bash groups: impossible de trouver le nom pour l'ID de groupe 32011 Je n'ai pas de nom !@ubuntu-phablet:~/.config$ ls dconf discoverrc okularrc Je n'ai pas de nom !@ubuntu-phablet:~/.config$while '.config' exists in the container and in the main system, they are not the same thing (not shared). 'Documents', 'Videos', 'Pictures', are shared. That's typical for flatpak applications - Libertine use the same tool (bwrap).
-
Example from this review: https://itsfoss.com/news/pinephone-review/
"As a result, users of it are subject to Ubuntu Touch’s “locked-down” style, similar to Android and iOS."" PinePhone owners are generally tinkerers who like control over their device, which is made much harder with Ubuntu Touch."
Actually apt is allowed in the PinePhone, so the article linked is mistaken. Which is typical for that site since it is not technical. The "locked-down style" is particularly wrong and shows the author hasn't done any research at all.
In general, new people coming into UT should expect having to adapt to things being different than their Linux distro. I like to describe the Ubuntu Touch experience as the closest you can get to the familiar GNU/Linux stack while retaining the phone's original functionality. -
The following scripts seems to allow to open a shell that has an editable "/", without modifying the actual rootfs of the phone AND the phone does reboot unmodified without any problem!
Apt (or rather dpkg) does not work just (yet ?) , because of its advanced filesystem usage, but I hope some tricks could be found for that. Anyway this is already useful to experiment modifying configuration files.
WARNING this is very experimental use at your OWN RISKS, I DO NOT TAKE ANY RESPONSABILITY in case of a problem.
#!/bin/sh set -e mkdir -p /tmp/upper /tmp/work /tmp/merged /tmp/merged/.old exec unshare -m --propagation private /bin/sh -c ' mkdir -p /tmp/upper /tmp/work /tmp/merged /tmp/merged/.old mount -t overlay overlay \ -o lowerdir=/,upperdir=/tmp/upper,workdir=/tmp/work \ /tmp/merged mkdir -p /tmp/merged/.old pivot_root /tmp/merged /tmp/merged/.old mount -t proc proc /proc || true mount -t sysfs sysfs /sys || true mount -t tmpfs tmpfs /dev || true cd / exec bash ' -
@Fuseteam said in Idea: overlayfs for user terminal:
But i really reccomend joining us on matrix or telegram
Yes thank you, I may join later, when I have a bit more time!
I find also that sharing infos on the forum is better for making things public and accessible in the long run, and that later we can know precisely what has been tried.
-
Well with this script I seem to be able to apt install at least some packages without modifying the phone system:
sudo ./test.sh [sudo] password for phablet: root@ubuntu-phablet:/# apt install htop Reading package lists... Done Building dependency tree... Done Reading state information... Done Suggested packages: lm-sensors The following NEW packages will be installed: htop 0 upgraded, 1 newly installed, 0 to remove and 63 not upgraded. Need to get 171 kB of archives. After this operation, 455 kB of additional disk space will be used. Get:1 http://ports.ubuntu.com/ubuntu-ports noble/main arm64 htop arm64 3.3.0-4build1 [171 kB] Fetched 171 kB in 0s (421 kB/s) dpkg-preconfigure: unable to re-open stdin: No such file or directory E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory) Selecting previously unselected package htop. (Reading database ... 60848 files and directories currently installed.) Preparing to unpack .../htop_3.3.0-4build1_arm64.deb ... Unpacking htop (3.3.0-4build1) ... Setting up htop (3.3.0-4build1) ... Processing triggers for hicolor-icon-theme (0.17-2) ... W: No sandbox user '_apt' on the system, can not drop privileges root@ubuntu-phablet:/# htop --version htop 3.3.0 root@ubuntu-phablet:/# exit exit phablet@ubuntu-phablet:~$ htop -bash: htop: command not foundtest.sh (WARNING this is very experimental use at your OWN RISKS, I DO NOT TAKE ANY RESPONSABILITY in case of a problem)
#!/bin/sh set -e mkdir -p /tmp/upper /tmp/work /tmp/merged /tmp/merged/.old exec unshare -m --propagation private /bin/sh -c ' mkdir -p /tmp/upper /tmp/work /tmp/merged /tmp/merged/.old mount -t overlay overlay \ -o lowerdir=/,upperdir=/tmp/upper,workdir=/tmp/work \ /tmp/merged mkdir -p /tmp/merged/.old mkdir -p /tmp/upper/etc/ cp /etc/resolv.conf /tmp/upper/etc/ pivot_root /tmp/merged /tmp/merged/.old mkdir -p /usr/share/man mkdir -p /usr/share/doc mkdir -p /usr/share/lintian mount -t tmpfs tmpfs /usr/share/man mount -t tmpfs tmpfs /usr/share/lintian mount -t tmpfs tmpfs /usr/share/doc mount -t proc proc /proc || true mount -t sysfs sysfs /sys || true mount -t tmpfs tmpfs /dev || true cd / exec bash 'It might not be ideal, and will probably not work with any package at all, but this is already quite an achievement to be able to do that.
-
Actually it seems that the only obstacle to this idea is this bug/limitation on overlayfs that exsited in old kernels (that we still have to use in Ubuntu Touch) and that make dpkg fail in some cases, when working with an overlayfs.
Unable to install new version: Invalid cross-device linkhttps://bugs-devel.debian.org/cgi-bin/bugreport.cgi?bug=836211
Patching the kernel(s) seem possible. I've tested with kernel 6.1.0-41, the script above fully works to allow transparent apt install without modifying the actual rootfs.
-
again, don't go the route of enabling apt. It simply not suited due to the way Ubuntu Touch is build. You will run into issues that we cannot help with. Even if you don't personally do, users of whatever you're cooking up will run into those issues. We cannot help with those issues not for lack of technical skills but for lack of time and people, there are other things we can focus our time and energy on. There is plenty we can fix to allow running packages you normally install with apt.
For people who want to install packages, you'd normally use apt for, i would like to suggest crackle instead. I don't want to push to hard on it, as it is something i wrote with the help of some in the community. I don't want to be like those companies praising their own products to sell it to you.
Crackle was born from the need to install packages but the lack of apt. It is the final result from experiments since 2018 which now 7 years and counting. The script itself started 4 years ago, and evolved quite a bit to get where we are now. At first it was wrapped around apt downloading packages and installing them into the home directory via various settings and environment variable, it worked fine for vim, git and even tailscale. But nowhere near the 80,000 packages ubuntu offers. Nobody, none of the people complaining about the lack of a way to install packages, even tried to help adapting more packages. Now 4 years later it uses nix and it works for all the packages i have tried. I even managed to install cargo and pipx with crackle, someone even managed to install flatpak— i have yet to find a package that plainly does not work. And there are over 120,000 packages to test.
I don't want to be over-confident but for this occasion i'll dare say that if a package installed with crackle doesn't work after installation, it would not have worked when installed with apt either— that is, is not an issue with crackle, but something we miss in UT— which is where can then focus our time and energy; improving UT's integration into the rest of the linux ecosystem
Once upon a time Ubuntu Touch used upstart, now we use systemd
Once upon a time Ubuntu Touch had its own display protocol, now we're moving towards Wayland
Once upon a time we had xmir, now we have xwayland
Once upon a time we had only libertine, now we have both snap and nix support
Once upon a time nix couldn't work on UT due to technical limitation, today it just worksStep by step integration work is done, to allow apps like firefox to work seamlessly
wait, did i just say nix just works, why did i then even mention crackle? am i a shill after all? well i can't deny i am biased, but one thing i noticed is that nobody talks about nix as a universal package manager and i think i know why, It is a completely different experience.
Since crackle was already a wrapper around apt, it was already close in experience to apt. So since i just swapped the "backend", it brings an apt-like experience to nix.Yesterday i had a feeling i was forgetting something else you can without apt and without a writable rootfs. And today i know what it was: cargo! nvm! jekyll! all these package managers just work on UT! I completely forgot about it since i haven't touched it in a while but my personal website was made on UT
p.s. installing crackle is a one liner as shown in the readme, on UT that oneliner only works if your rootfs is NOT remounted as readwrite
-
@Fuseteam said in Idea: overlayfs for user terminal:
again, don't go the route of enabling ap
Am I actually free to go the route I want to go with my phone?
You will run into issues that cannot help with.
Can you share any fact that you base your analysis on ?
Everything shows the script I have shared cannot have any impact whatsoever on the rootfs for two distinct reasons, first because it is never remounted RW, so it remains read-only at all time, and second because it mounts an overlay over it, so the real "/" would not be modified even if it was read-write. In practice all the tests shows it actually does that and does not modify at all the phone system.
I'm the kind of person convinced by facts and proofs, if you're saying that what I'm doing is going to generate problems, then I want proof and detailed explanations of why.
-
@pparent I base my analysis on 7 years of helping people in the community debug issues and the time we've wasted because they ignored all warnings given. And not just my experience, but of many in the community.
You are free to do what you want sure, but know that as a developer you're decisions impact users and the community at large. If problems, new and old, will you take personal responsibility for all of them? will you be the one to assist them 24/7? Because the community cannot given our current size and the work that still has to be done.
This path has been taken many times as can be seen in almost a decade of history in the over 50 telegram groups. I have take this path aswell crackle is something that is came to be after all those different attempts. Which again, i don't want to promote too hard— but i do encourage you to at least try it and see if it can fit the usecase you are trying to fill with apt. Why ignore the effort that has already been done, to solve the case of "installing packages via the terminal"? Sure it isn't
aptbut does that have to be? We now have access to over 120,000 packages, which includes everything apt has to offer. Wouldn't it be better to concentrate on finding packages that don't work and figure out why they don't? -
P.s. i have a feeling you haven't been reading my full posts because they are too long, so i again implore you to join either telegram or matrix, the more of the community can jump in to give all the proofs and data that you want, including why
aptis not very suitable for general use. I'd explain the architecture of UT but that would explode my post you'd most likely not read the whole thing and as such not learn much from it.given
aptis such a core system component that UT uses in a very specific way— So if you absolutely insist onaptthen you should at least first learn how updates work on UT. case in point: due to the architecture of Ubuntu Touchsudo apt upgradehas a very real chance of making your phone unbootable. I have seen this happen a couple of times already in the my past 7 years of assisting people in the community.One good way to learn these things is by interacting with the people building the system, our main groups are at t.me/ubports and #ubports:matrix.org. I won't bore you with the tens of telegram groups and tens of matrix rooms i'm in— if you do join, the community will point you to the appropriate groups as necessary
-
@Fuseteam Well I don't want to fight, I have no energy for that, neither here or on telegram. I've not encouraged anybody to do anything, just shared the results of my experiments with all due warnings and disclaimers. Developers out there will be able to read the script an make their own opinion. I've shared all I had to share on that topic, and I don't think I will post anymore in this topic, except if I'm requested to do so. I'm honestly not sure why sharing an interesting idea and experiment triggers a so strong reaction, I did not demand that anything be integrated in the official system without proper validation.
-
@pparent i'm not inviting you to fight, i'm inviting you to help you learn how it all works so you don't have to fight the system and run into issues we don't have the energy to help with.
many people don't read disclaimers, they see something someone publish, they use it. see open-store.io for a good example: clicking the bug icon tells them "don't do a bug report here, do it on click this button to go to the issue tracker. the amount of bug reports on the open store and missing on the actual issue tracker speaks for itself.
-
@pparent the strong reaction around apt is because what you are trying has been tried, we've been around for almost a decade which has yielded results.
for this i invoke "Chesterston's fence", behind that fence is apt, you are looking for a way to tear down the fence. But the real question is, why hasn't this fence been removed in the past decade? Spoiler: it is not due to a lack of trying
