Snap Crackle and Pop! Readwrite rootfs is overrated

Fuseteam

Hi my name is fuseteam but some of you may know from telegram as Tobiyo Kuujikai. this will be my first topic, i've been thinking how to write this up for a while. ~~no, i'm not japanese~~ i suppose i should start at the beginning.

When i first heard of ubuntu touch i was pretty excited to see a OS i could be in control of, as i never like how android worked and such. heh i only switched to android from symbian when my trusty old nokia 5800 just didn't hold enough battery anymore. like many the idea of using my phone as a "PC" really fascinated me, especially since at the time i had a laptop with 3GB of RAM and smartphone were beginning to get just as much RAM. you can imagine my disappointment when canonical announced the drop..... as i was a information technology and having a (fairly amateurish) understanding of programming i grew desperate and began searching for how to get in contact and do something. i eventually stumbled upon the unity8 telegram group and from there i found the other ubports groups. after some activity and getting to know the community i was eventually offered a device to get to know the os and the rest is (telegram) history.

now why am I talking about all this? what up with the title?
my expectation was that this will enable a "PC in my pocket" a "palmtop" at this point i had some experience with ubuntu on the desktop, i attempted to apply my meager experience to ubuntu touch and see what i can do. you can imagine my surprise when things didn't work as i had come to know it. as i got to know more of the community i found way to do what i wanted to do and have had a lot of fun with it. i even ended up editing the keyboard on the device with vim, breaking and fixing things. one of my goto method to restore things when things go "really bad" has become ssh in and run sudo system-image-cli -b 0 --progress=dots. i have used this more times then i can count. and that on its allowed the readonly rootfs to grow on me, even to such a degree i somewhat want it on desktop xD

but as i was doing some things in a unsupported way there were some caveats, every time i refreshed my rootfs i lost all my favorite commandline utilities! but i can't complain right? apt is not supported, i get to keep the pieces..... as time went on however it does start to become tedious to reinstall things but hey i can't complain i am doing unsupported things, enter nix. someway somehow i stumbled on nix and from there i talked with quite a few regular contributors such as NotKit, Jonatan, Myii, Kugi, Dalton, Kugi and probably many more and learn quite a lot
did you know that while the rootfs is readonly, some paths are actually writable: for example /home is completely writable. in hindsight it has to be, how else would our userdata survive refreshes and ota updates! it turns out there is a file (that isn't writable) that mounts basically keeps track of all those paths in /etc/system-image/writable_paths this file is processed in dash mounting the folders and thus preserving /userdata! from there i discovered nix is actually pretty easy to set up!

remount readwrite
create the directories /nix and /etc/nix
add the paths to /etc/system-image/writable
remount readonly
reboot
everytime it disappears just readd the paths, recreate the directories and reboot, easy yes? wait nix is missing title, this post isn't about nix........is there caveat? yes zero packages for armhf! nix would compile the packages which could take hours, and lots of space. heck i have this sneaking suspicion it even trying to compile a new kernel! surely there must a different way! and there is, from there there's the possibilities to package the apps in a click and ship everything it needs, but that that worked! but it still felt messy, it would create a bunch of stub apps which literally just tells the user to add a certain path to PATH to be able to run the application........ it works! but we can do better! at that point i began toying with the idea to build an app but then it hit me. i learn deb packages are archives, that apt has a download-only option that apt can be configured to work independent from the rootfs and that dpkg can extract debs to any arbitrary directory, crackle was born.

orginally i intended to propose a path like /home/pkgs to be a standard PATH as to have some sort of eventual multi user config, but with growing security concerns and little push from some in the community it now installs in the xdg home directories: ~/.cache, ~/.config and ~/.local/share

so will this be a replacement for libertine? no. a lot of work has been put into libertine, years of work. a little bash script surely can't replace that! so what is my goal? in my experience and understanding libertine doesn't cut it..... why not? libertine is a whole desktop ubuntu container 1.5 GB just to run little old vim and given i use vim to experiment on device, the isolation from rootfs is little much. so my initial goal was to be able to install the utilities i use to hack on the phone and preserve them all on a readonly rootfs. But given how similar many utitliies function this extends beyond vim but probably not all the way to gui applications, those are really complicated. it is however possible to fetch a package and all its dependencies with this tool however to make it "self-contained" possibly make a crude click of out of it but that a whole other beast. well that was a mouth full lemme try to keep the rest short with the question i can already hear:

sooo what packages already work?

according reports and my own (crude) tests the following packages appear to work:

vim
git
nmap
htop
mc
transmission-cli
shellcheck
xz-utils
curl
tmux
byobu
patchelf
tree
sshfs

packages that don't work

man
locate
many others i have yet to test

does it really not work with gui applications?

well it can work but it very fiddly and mostly unusable
it would be better to try those in libertine or at least bundled in click and access properly through xmir (xwayland one day)

wait slow down why is it called crackle?

the name is play on physics which itself is reference to a very old commercial
in physics snap is a derivative of jerk which is a derivative of acceleration, i guess the "acceleration" of acceleration
pop is the "acceleration" of snap
but wait about the "speed" of snap? well that's crackle!
what? is all greek? no worries have this nice video instead: https://www.youtube.com/watch?v=q6TIsxTdrCU

will this work for all commandline utitlities?

no idea, probably not. it comes down to what the utitlity needs, sshfs for example fails because of "operation not permitted" (likely related to the way fuse should be setup) while man fails because it just has to have its config file in /etc it just refuses otherwise and probably the way it wants to be setup, locate also has a similar issue with it just wanting the certain files in certain locations. i'm only one little guy i can only test and use so many applications

where is the sauce?

the source is at https://github.com/tuxecure/crackle, its also on gitlab at https://gitlab.com/tuxecure/crackle-apt/crackle. the github is a mirror of gitlab. the scripts used to setup the environment and such is at https://gitlab.com/tuxecure/crackle-apt/cracklebin

so can i do anything i can do with apt with it?

uhhhhhhhh no at this stage the following commands are available

crackle install <package>
crackle download <package>
crackle clean
crackle show <package>
crackle search <package>
crackle update
yes that complete when it comes to apt like commands, there is no way to upgrade or remove packages at this stage, tho i suppose you could use crackle install to upgrade single packages it also has a couple of unique commands
crackle setup this is used to "install" crackle and setup up its initial environment more details in the next section
crackle crack <package> this can be used to extract a package and its dependencies in /home/phablet/packages/<package>

How do i get crackle?

currently you can get a hold of it by downloading the literally downloading the code (as a zip) from github or gitlab and extract it. then you cd into the folder and run ./crackle setup note this will use crackle (recursive!) to install both git and patchelf
git to download the scripts from cracklebin and patchelf so those scripts can patch binaries where needed. git can also be used to fetch newer version with git pull later

much bugs, where report?

for now bugs can be reported on gitlab
or you can contact me on telegram @fuseteam

so wait why is readwrite rootfs overrated?

for me i made it unnecessary to remount the rootfs. As crackle made it possible to "install" 90% of the utilities i tend to use in the terminal without remounting the rootfs, so for me a readwrite rootfs is overrated granted i do still remount to readwrite to hack on the os crackle might not be suited for your use, but it doesn't mean there is not way, there is always libertine and if that doesn't suit your fancy i would like to encourage you to either help improve libertine, or to explore what options are out there and how to get it working on Ubuntu Touch I believe there are a lot of possibilties and with that this post is way too long, so i'll sign off here

jezek

@fuseteam Bravo! I haven't tried your scripts yet, but will definitely try. I had this thought, that apt is just a zipped file and I have somewhere seen somewhat to install into home directory from apt, so theoretically some utilities could be installed into home. I have golang installed in home like this (just unpacked and set paths). Vim I couldn't figure out, so I use a neovim standalone executable as vim. But you are a lot further before me. Bravo again and thank you for your crackle. I look forward to toy with it. I've said it twice, I'll say it again. Bravo!

Fuseteam

@jezek actually have vim figured out actually so yeahfeel free to toy with crackle :3 the more people toy with it the more we can tests the limits of it xD
to infinity and beyond! Plus Ultra!

dtarrant

Wow, amazing post! I have to admit that 90% of this is way over my head, but I know you understand what you are presenting. I am a physics graduate and this is the first time I have heard of "snap" and "pop". Awesome.
Respect

Fuseteam

@dtarrant haha that's surprising, granted i have heard snap, crackle and pop aren't used that often, afaik they are occasionally used, though "sometimes somewhat facetiously" hehe in physics terms it's the 4th, 5th and 6th derivatives of position if i'm counting right

Fuseteam

wheeeee sshfs now works /o/

sven

When I first read about Crackle in June, I found a useful introduction page (on github?). Now, I cannot find it anymore. What is its current URL?

matteo

@fuseteam great thread and fantastic job on this. I think this very useful, definitely looking forward to install crackle and exploit its potential. Will follow its development

Fuseteam

@sven i haven't written an introduction page yet. i think i'll make one after i've figure out how to make it remove packages it installed in a sane way. the closest to a introduction page that i have is this forum post, feel free to ask me anything about it
the github and gitlab link are as mention in the post

Fuseteam

@matteo haha glad to hear, tho install is a bit of a overstatement it's really just a script making use of what is available on UT out of the box

Fuseteam

Major limitation #1: seahorse the gnome keyring frontend thingie
attempts to launch seahorse results in the following error:

(process:9315): GLib-GIO-ERROR **: Settings schema 'org.gnome.seahorse' is not installed

Trace/breakpoint trap

Fuseteam

@fuseteam

@fuseteam said in Snap Crackle and Pop! Readwrite rootfs is overrated:

Major limitation #1: seahorse the gnome keyring frontend thingie
attempts to launch seahorse results in the following error:
(process:9315): GLib-GIO-ERROR **: Settings schema 'org.gnome.seahorse' is not installed

Trace/breakpoint trap

looks like these schema's are installed in /usr/share/glib-2.0/schemas i wonder if this can be controlled somehow with packages as to tell the program where to look for these schemas

Fuseteam

i have no publish a ready to use zip file in the release section of the github mirror along with some rudimentary usage instructions

nero355

@fuseteam said in Snap Crackle and Pop! Readwrite rootfs is overrated:

When i first heard of ubuntu touch i was pretty excited to see a OS i could be in control of, as i never like how android worked and such.

like many the idea of using my phone as a "PC" really fascinated me, especially since at the time i had a laptop with 3GB of RAM and smartphone were beginning to get just as much RAM.
you can imagine my disappointment when canonical announced the drop.....

+1 my brother!

sooo what packages already work?

according reports and my own (crude) tests the following packages appear to work:

vim

nmap

htop

mc

curl

tmux

tree

sshfs

NICE!

Thank you for this awesome tool/project and I will keep an eye on it in the future!