Idea: overlayfs for user terminal

pparent

@gpatel-fr said in Idea: overlayfs for user terminal:

@pparent said in Idea: overlayfs for user terminal:

to do complicated things with libertine

if the goal is installing small utilities, and you know about apt update and apt install, all that is needed is to learn to do things like

libertine-container-manager install-package -p ncdu
libertine-container-manager install-package -p fdisk
libertine-container-manager install-package -p apt-file

and then

libertine-launch -i noble bash

to actually use the installed packages.

I'm sorry but this is way too complicated for a newcomer that just wants to feel at home by using apt, and that first needs to understand the concept that they can't apt and the concept of Libertine before running these commands. Same applies for "just copy the executable from the .deb to .local/" .

By the way the review from @theprivacydad mentions this complexity for a new user of geting used to new concepts and viarous software sources ( Libretine, Waydroid, snap, openstore)

pparent

The thing with libertine container is that you immediately have to ask yourself those questions, which makes it complicated to apprehend, but on top of that I suppose some of these things are not possible. Really I think this makes it way too much overhead for a beginner.

-Can I interact with files from the host system with tools from the libertine container?
-Can I edit system configuration files of the tools that I installed, and if so should it be edited inside the container or on the host?
-Can I execute tools from the host and from the libertine container easily in the same script, including with pipes and STDOOUT redirections?
-Can I link a binary inside the container against a library from host system?
-Can I link a binary from host system against a library installed inside the libertine container?
-If I install a package with a lot of dependencies inside the container will it install in duplicate the libraries that are already installed in the host system?

And things that are certainly not possible with libertine, but are trivial with overlayfs:

-Virtually remove file or package from the host file system
-Virtually edit system configuration files for already existing programs in the host
-Reset a file, a folder, or the whole system to its default value any time.

Fuseteam

Why not mount an overlayfs over "/" and pivot_root in the environment of the user terminal? Like this the user could do whatever they want in their terminal without any risk of breaking the rest of the system (the overlay would not apply outside of the user terminal). On top of that it would reduce the tendency of advanced users (or not so advanced) to remount rw the filesystem to install small utilities with apt. It would also allow for more experimentation on the system with the possibility of a reset at any moment. And it could allow to install deb with less restriction on partition size as the overlay could use more space than the original system partition.

This has already been done aaaaaand it breaks when files underneath change making the phone essentially unbootable.

I think that not being able to install packages with apt in terminal can be frustrating for many new users who expect to escape a locked-down environment like Android, and find complete freedom in Ubuntu Touch. And somehow the first thing we tell them is that they should not do apt commands, it can be frustrating for newcomers and give the impression that it's actually another locked-down system. And we may loose users that way.

Ubuntu Touch is NOT a locked down system, just because apt is unavailable. It is a immutable distro just like the likes of steamos, fedora silverblue, microos among other next generation desktops

Things you can do on UT that you can't on a locked down system like android:

Set up a systemd service to start a services in the background (syncthing is super frustrating to use on android who keeps killing it, works seamlessly on ubuntu touch)
set up a systemd service to start a service only if connected to a socket, (ssh comes to mind)
mount the rootfs readwrite to modify your system for experiments (take that Android! i want a custom keyboard that is of my design)
install packages with snap (granted this is work in progress, you'll notice things you install that are meant for desktop linux, don't work well out of the box on UT, there is a lot of updates still needed, such a better wayland support, xdg portal support, hardware acceleration among other things)
install nix to install over 120,000 packages (psst i've made a post about this)
set up tailscale to access your phone from anywhere
set up openvpn without a need for a dedicated app (did you know you share your vpn connection with other devices?)
set up wireguard without a need for third party software (provided your ports kernel has the wireguard module enabled)
did i mention you can use syncthing to sync directories you choose between your devices yet? (i use this to sync my 2fa app configs between 2 UT devices and 2 linux laptops)
did i mention you can install over 120,000 packages yet? (this firefox click was developed on a Ubuntu Touch device using a crackle a bash/python script monster developed over 4 years also on a ubuntu touch device using vim, yes that vim and ofcourse git)
xforward x11 applications installed using nix, snap to a linux desktop cause why not
did i mention ufirefox yet? that wasn't recompiled for ubuntu touch, it is literally the same binaries used on raspberry pi, straight from mozilla
connect android/ios devices to Ubuntu Touch to use it as bluetooth speaker (yes i have a usecase for this, don't break it)

the basic operation of apt is:

it downloads the package from a repository (stores that in in /var/cache/apt)
extracts that with sudo on top of /

This last part means it any path matches, it will just overwrite that, and this can go very wrong very fast if you don't keep in mind how updates are handled on Ubuntu Touch.

The real question we should asking isn't "how do we make apt work on immutable distros like Ubuntu Touch?" but rather "What am i trying to install with apt?" Chances are you can install it with snap or crackle, yes the same crackle i used to repackage firefox for UT. They will function the same as if you were install them with apt— if they are not functioning as expected, that isn't due to the "lack of apt" but rather that it needs integration work on UT

does this mean apt is useless on UT? Nope, there is little tool called ubports-qa which uses apt to install Merge Request from gitlab directly on your phone for your testing pleasure. Remember when i said "if you don't keep in mind how updates are handled on Ubuntu Touch''? This is a part of that, and to avoid unneccesary breakage you are advised to not run this tool on eithe the stable or rc channels only on the dailies(noble) or devel(focal)

BTW are you on telegram (t.me/ubports) or matrix (#ubports:matrix.org) some of these things are much easier to explain in real time vs a forum like this, there is a lot we can help you learn about ubuntu touch and what crazy experiments have been done— that would this post wayyyy longer than it already is

pparent

@Fuseteam said in Idea: overlayfs for user terminal:

This has already been done aaaaaand it breaks when files underneath change making the phone essentially unbootable

Well I don't see how what I propose could make the phone unbootable if it only applies to user terminal, it is simply not possible. I'm not sure what has already been done, but I guess from what you say it must be mounting the overlayfs directly to "/" for the system which is NOT what I propose. (On top of that, even mounting the overlayfs directly to "/" is possible if you really dig into it and do it properly: openwrt does it and it works like a charm while allowing sysupgrade )

But it's true that there could be problems in the terminal for keeping the data between system upgrades, and that would need to be investigated. But for me it would be useful to have this feature, including if we reset the overlay each time you upgrade, as a temporary solution (it he user is warned properly). Because what this feature would allow is mainly experimenting.

Ubuntu Touch is NOT a locked down system, just because apt is unavailable

I do agree, but that's what new users, that just give it a try for few hours / days, might feel. And this might be enough to loose a user that just will not spend so much time to dig in , before deciding if he wants to go any further with the system or not. People judge fast and don't have much time.

Example from this review: https://itsfoss.com/news/pinephone-review/
"As a result, users of it are subject to Ubuntu Touch’s “locked-down” style, similar to Android and iOS."

" PinePhone owners are generally tinkerers who like control over their device, which is made much harder with Ubuntu Touch."

Fuseteam

@pparent trust me, it happened, the issue is way more complicated than you think. If you can think of it, it likely has been tried in the last 8 years (i call this rule 53)

Also i edited my answer a bit with some more tidbits, like you are doing now, feel free to check. But i really reccomend joining us on matrix or telegram

People who are calling Ubuntu Touch "locked down" due to the immutable nature are unfortunately too focussed on apt there is at this moment 2 alternative that lets you install the entire repetoire of linux packages all 120,000 of them (apt in contrast has access to about 80,000 of them) if you so wish—- see the links i included in my answer. You have a lot of control under UT, it just a different paradigm that departs from traditional package managers.

That is why we're going to solve this with "make apt usable out of the box" but rather with "What are people trying to install, what are they trying to achieve?"
This "i can apt therefore locked down" is a good example of the XY problem

The user is trying to solve a problem, (say use vim) think they have to use apt to install a package to do it (sudo apt install vim) and then fight the system to make apt work
while there may be a better way (crackle install vim)

pparent

@Fuseteam said in Idea: overlayfs for user terminal:

If you can think of it, it likely has been tried in the last 8 years (i call this rule 53)

Strangely a thing like porting Signal-Desktop as a click package, and make its interface responsive, had not been tried in the last 8 years while being relatively easy and fast to do, and many users needing Signal (or even leaving explicitly the ecosystem for the lack of it) !

People who are calling Ubuntu Touch "locked down" due to the immutable nature are unfortunately too focussed on apt there is at this moment 2 alternative that lets you install the entire repetoire of linux packages if you so wish—- see the links i included in my answer. You have a lot of control under UT, it just a different paradigm that departs from traditional package managers

Once again I'm not the one to convince, potential new users are.

The user is trying to solve a problem,

Not necessarily, the user might simply be trying to feel good, and feel at home, knowing they can do the same thing and have the same freedom as in their desktop distro. Feeling can be as important as functionalities for a new user to adopt something.

Well anyway I think what I propose might possibly be tried or even deployed without any system modification. Maybe with an alternative terminal app, (or even simply by modifying bash-rc and keep the same terminal app). So we'll see if I have time to try it out some day.

gpatel-fr

@pparent said in Idea: overlayfs for user terminal:

-Can I interact with files from the host system with tools from the libertine container?

when inside the container you are confined, you can interact with directories shared with the main computer, that means data directories, to the exclusion of configuration of applications external to the container. That may or not be a good thing if the primary concern is the security of hapless users or the liberty of the brave developer (sometimes the 2 types are the same as it is the case for developers caught in supply chain traps )

(py312) phablet@ubuntu-phablet:~$ cd .config
(py312) phablet@ubuntu-phablet:~/.config$ ls
 8192.alaskalinuxuser   chromium               evolution                   google-chrome       maliit.org              music.ubports              steps.s60w79        themeswitch.danfro           vivaldi
 address-book.ubports   clock.ubports          filemanager.ubports         libaccounts-glib    messaging-app.ubports   notes.ubports              sterlingpdf         uhome.alaskalinuxuser        weather.ubports
 BraveSoftware          connectivity-service   forums-ubports-com.webber   lomiri              microsoft-edge          openstore.openstore-team   systemd            'Unknown Organization'
 calendar.ubports       dconf                  gallery.ubports             Lomiri              mirvncserver.abmyii     procps                     telephony-service   user-dirs.dirs
 camera.ubports         dialer-app.ubports     glib-2.0                    lomiri-ui-toolkit   morph-browser           pulse                      terminal.ubports    ut-tweak-tool.sverzegnassi
(py312) phablet@ubuntu-phablet:~/.config$ libertine-container-manager exec -i noble -c bash
groups: impossible de trouver le nom pour l'ID de groupe 32011
Je n'ai pas de nom !@ubuntu-phablet:~/.config$ ls
dconf  discoverrc  okularrc
Je n'ai pas de nom !@ubuntu-phablet:~/.config$

while '.config' exists in the container and in the main system, they are not the same thing (not shared). 'Documents', 'Videos', 'Pictures', are shared. That's typical for flatpak applications - Libertine use the same tool (bwrap).

ikoz

@pparent

Example from this review: https://itsfoss.com/news/pinephone-review/
"As a result, users of it are subject to Ubuntu Touch’s “locked-down” style, similar to Android and iOS."

" PinePhone owners are generally tinkerers who like control over their device, which is made much harder with Ubuntu Touch."

Actually apt is allowed in the PinePhone, so the article linked is mistaken. Which is typical for that site since it is not technical. The "locked-down style" is particularly wrong and shows the author hasn't done any research at all.
In general, new people coming into UT should expect having to adapt to things being different than their Linux distro. I like to describe the Ubuntu Touch experience as the closest you can get to the familiar GNU/Linux stack while retaining the phone's original functionality.

pparent

The following scripts seems to allow to open a shell that has an editable "/", without modifying the actual rootfs of the phone AND the phone does reboot unmodified without any problem!

Apt (or rather dpkg) does not work just (yet ?) , because of its advanced filesystem usage, but I hope some tricks could be found for that. Anyway this is already useful to experiment modifying configuration files.

WARNING this is very experimental use at your OWN RISKS, I DO NOT TAKE ANY RESPONSABILITY in case of a problem.

#!/bin/sh
set -e

mkdir -p /tmp/upper /tmp/work /tmp/merged /tmp/merged/.old

exec unshare -m --propagation private /bin/sh -c '
  mkdir -p /tmp/upper /tmp/work /tmp/merged /tmp/merged/.old

  mount -t overlay overlay \
    -o lowerdir=/,upperdir=/tmp/upper,workdir=/tmp/work \
    /tmp/merged

  mkdir -p /tmp/merged/.old
  pivot_root /tmp/merged /tmp/merged/.old

  mount -t proc proc /proc || true
  mount -t sysfs sysfs /sys || true
  mount -t tmpfs tmpfs /dev || true

  cd /
  exec bash
'

pparent

@Fuseteam said in Idea: overlayfs for user terminal:

But i really reccomend joining us on matrix or telegram

Yes thank you, I may join later, when I have a bit more time!

I find also that sharing infos on the forum is better for making things public and accessible in the long run, and that later we can know precisely what has been tried.