Set partition sizes when flashing

L-00117

This post is deleted!

myii

Apologies to all for the mass call-outs used in this post -- I wouldn't usually do this. I've encountered a situation where extending rootfs and all subsequent customisations are lost. I'm sharing this as a warning for those who have already resized, a caveat for those considering to do so, and as a discussion point for how to move forwards.

Switching the release channel loses the extended `rootfs`

With hindsight, there's nothing surprising about this since the process of switching channels replaces /userdata/ubuntu.img.

@haveaniceday above:

Of course, this will probably break OTA updates...

@UniSuperBox above:

The image file can be resized, and I'm considering ways to make ubports-qa do this in case the user would prefer to use apt and turn off system-image upgrades.

I never had any issues with OTA updates, even after some fairly extensive hacking and customisations. Other than switching the channel, what other system-image upgrades could lead to this outcome?

Moving forward

@haveaniceday above:

I'll just repeat the steps for each release.

Essentially, this is all that can be done. But I managed to do some decent testing and found that this reconfiguration process can be made much easier. In my case, I made promising progress with SaltStack configuration management. Solutions involving tools such as this become viable once rootfs has been extended (in that the necessary packages can be installed). It wouldn't be too difficult to have a repo for collecting formulas that help automate this.

@vandys @arubislander @doniks Since you all appear to have gone through the resizing process, I hope you don't mind me drawing your attention to this post.

myii

One strange experience to share about extending rootfs after switching the release channel. Following the instructions above, using 6000 worked fine for the rc channel. However, after switching to the dev channel, I couldn't get this to work for that value, no matter how many times I resized and rebooted. I did manage to get other sizes to work, though. Ultimately, I settled for 6144 (6 * 1024) to get 6.0GB, i.e.:

dd if=/dev/null of=/userdata/ubuntu.img bs=1M seek=6144 count=0

UniSuperBox

Most system-image upgrades are differential, where only the changes from one image to the next are downloaded and applied. So, any updates in upstream Ubuntu as well as our own packages are changed. If you run sudo apt dist-upgrade, you will already have these newer packages and won't see any changes when the upgrade happens. This could be seen as a bug or a feature depending on who you are. It's a bug in that an image may not be in a known good state after an upgrade.

Upgrades which download a whole system image will always rewrite all of your changes. Full updates will happen in the following scenarios:

You specify a full upgrade to system-image (There's a way to do this, but I'm not sure how at this time)
The system-image-server does not have a diff from the image you are using to the new image (you've skipped a few updates, for example I went from devel 2018-12-21 to 2018-12-28 today and had to download the full image)
You are switching release channels

This is why I said that ubports-qa should disable system-image updates. After you've changed your system image, it is no longer in a known good state and may behave in unexpected ways.

myii

@unisuperbox Fantastic, detailed explanation -- appreciate you taking the time to go through that.

The system-image-server does not have a diff from the image you are using to the new image (you've skipped a few updates, for example I went from devel 2018-12-21 to 2018-12-28 today and had to download the full image)

This leaves me with some food for thought since it appears that I've increased my risk of this happening by switching to the dev channel.

This is why I said that ubports-qa should disable system-image updates. After you've changed your system image, it is no longer in a known good state and may behave in unexpected ways.

Is there a simple way of testing this out in the meantime?

KrisJace

I have been rad with the Nexus5 where rootfs is not constrained by a small physical partition size. I wish all devices could be converted to that layout.
So far I enabled very easy resizing of rootfs via atuTools4UT: https://sourceforge.net/projects/all-things-ubuntu-library/files/releases/

And the related portion in the sourcecode: https://sourceforge.net/p/all-things-ubuntu-library/code/HEAD/tree/trunk/ATUpackages/ATU/ATU-src/atulib.pas#l1329

Currently working on a GUI for also resizing rootfs based on physical partition layout, ie. on Meizu PRO 5. Not sure if I will be able to do it on-device, but am still exploring this possibility, even using pivot chroot method.

rooty

Hi,

Please help me, I'm confused...

So I'm trying to increase my root, did following:

dd if=/dev/null of=/userdata/ubuntu.img bs=1M seek=6144 count=0
e2fsck -p /userdata/ubuntu.img
resize2fs /userdata/ubuntu.img

got this

root@ubuntu-phablet:~# ls -lahi /userdata/
total 3.9G
      2 drwxrwx--x.  5 system      system           4.0K Jul  1 12:08 .
      2 drwxr-xr-x  23 lxc-dnsmasq systemd-timesync 4.0K Apr 30 19:54 ..
1310721 drwxrwx--x  23 system      system           4.0K Jul  1 13:21 android-data
     22 -rw-r--r--   1 root        root                2 Oct 13  1970 .last_ubuntu-build
     21 -rw-rw-rw-   1 root        root                0 Oct 13  1970 .last_update
     13 -rw-------.  1 root        root                2 Jan  7  1970 .layout_version
     11 -rw-------   1 root        root              32M Oct 13  1970 SWAP.img
 131073 drwxr-xr-x   5 root        root             4.0K Oct 13  1970 system-data
     19 -rw-------   2 root        root             6.0G Oct 14  1970 system.img
     19 -rw-------   2 root        root             6.0G Oct 14  1970 ubuntu.img
 655361 drwxr-xr-x   3 root        root             4.0K Oct 13  1970 user-data

root@ubuntu-phablet:/userdata# parted ubuntu.img
GNU Parted 3.2
Using /userdata/ubuntu.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model:  (file)
Disk /userdata/ubuntu.img: 6442MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  6442MB  6442MB  ext2

already did reboot couple times and still getting

root@ubuntu-phablet:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      2.0G  1.8G   82M  96% /

my specs:
nexus 5
Ubuntu 16.04.6 LTS

p.s.
All Things Ubuntu Library does not work for me

root@ubuntu-phablet:~# ./ATU_arm-linux
./ATU_arm-linux: error while loading shared libraries: libgdk-x11-2.0.so.0: cannot open shared object file: No such file or directory
root@ubuntu-phablet:~# ldd ./ATU_arm-linux
	libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6f01000)
	libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xb6eee000)
	libgdk-x11-2.0.so.0 => not found
	libX11.so.6 => /usr/lib/arm-linux-gnueabihf/libX11.so.6 (0xb6dfd000)
	libgdk_pixbuf-2.0.so.0 => /usr/lib/arm-linux-gnueabihf/libgdk_pixbuf-2.0.so.0 (0xb6dd4000)
	libgtk-x11-2.0.so.0 => not found
	libgobject-2.0.so.0 => /usr/lib/arm-linux-gnueabihf/libgobject-2.0.so.0 (0xb6d8b000)
	libglib-2.0.so.0 => /lib/arm-linux-gnueabihf/libglib-2.0.so.0 (0xb6cb3000)
	libgthread-2.0.so.0 => /usr/lib/arm-linux-gnueabihf/libgthread-2.0.so.0 (0xb6ca1000)
	libgmodule-2.0.so.0 => /usr/lib/arm-linux-gnueabihf/libgmodule-2.0.so.0 (0xb6c8e000)
	libpango-1.0.so.0 => /usr/lib/arm-linux-gnueabihf/libpango-1.0.so.0 (0xb6c48000)
	libcairo.so.2 => /usr/lib/arm-linux-gnueabihf/libcairo.so.2 (0xb6b90000)
	libatk-1.0.so.0 => /usr/lib/arm-linux-gnueabihf/libatk-1.0.so.0 (0xb6b69000)
	libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6a7d000)
	/lib/ld-linux-armhf.so.3 (0xb6f38000)
	libxcb.so.1 => /usr/lib/arm-linux-gnueabihf/libxcb.so.1 (0xb6a59000)
	libgio-2.0.so.0 => /usr/lib/arm-linux-gnueabihf/libgio-2.0.so.0 (0xb6956000)
	libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xb68de000)
	libffi.so.6 => /usr/lib/arm-linux-gnueabihf/libffi.so.6 (0xb68c8000)
	libpcre.so.3 => /lib/arm-linux-gnueabihf/libpcre.so.3 (0xb686b000)
	libthai.so.0 => /usr/lib/arm-linux-gnueabihf/libthai.so.0 (0xb6855000)
	libpixman-1.so.0 => /usr/lib/arm-linux-gnueabihf/libpixman-1.so.0 (0xb67cb000)
	libfontconfig.so.1 => /usr/lib/arm-linux-gnueabihf/libfontconfig.so.1 (0xb678e000)
	libfreetype.so.6 => /usr/lib/arm-linux-gnueabihf/libfreetype.so.6 (0xb6717000)
	libpng12.so.0 => /lib/arm-linux-gnueabihf/libpng12.so.0 (0xb66ef000)
	libxcb-shm.so.0 => /usr/lib/arm-linux-gnueabihf/libxcb-shm.so.0 (0xb66dc000)
	libxcb-render.so.0 => /usr/lib/arm-linux-gnueabihf/libxcb-render.so.0 (0xb66c4000)
	libXrender.so.1 => /usr/lib/arm-linux-gnueabihf/libXrender.so.1 (0xb66ad000)
	libXext.so.6 => /usr/lib/arm-linux-gnueabihf/libXext.so.6 (0xb6692000)
	libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xb6670000)
	librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xb665a000)
	libXau.so.6 => /usr/lib/arm-linux-gnueabihf/libXau.so.6 (0xb664e000)
	libXdmcp.so.6 => /usr/lib/arm-linux-gnueabihf/libXdmcp.so.6 (0xb663a000)
	libselinux.so.1 => /lib/arm-linux-gnueabihf/libselinux.so.1 (0xb6615000)
	libresolv.so.2 => /lib/arm-linux-gnueabihf/libresolv.so.2 (0xb65f5000)
	libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb65cd000)
	libdatrie.so.1 => /usr/lib/arm-linux-gnueabihf/libdatrie.so.1 (0xb65b8000)
	libexpat.so.1 => /lib/arm-linux-gnueabihf/libexpat.so.1 (0xb6590000)

rooty

ok, so I was not able to accomplish this task from inside my phone, for some reason modified filesystem's size shrinks back to 2GB even tho I run resize2fs few times. Partition was fine tho, it just FS.

For those who's trying to solve this issue:
Was able to change the file system size from a host, connecting the phone to my laptop and running

adb shell  # phone in recovery mode
e2fsck -f -y system.img
resize2fs -fp system.img

simbion

@rooty
I managed to do it to expand the system partition from 2GB to 6GB. Went into recovery mode UBports:
simbion@simbion:~/1$ adb devices
simbion@simbion:~/1$ adb pull /data/system.img
simbion@simbion:~/1$ dd if=/dev/null of=system.img bs=1M seek=6144 count=0
simbion@simbion:~/1$ e2fsck -p system.img
simbion@simbion:~/1$ resize2fs system.img
simbion@simbion:~/1$ adb push system.img /data
The process is not fast. After that, I have all the updates and applications began to be installed. It would be nice if did it in UBports when you install to select how much you need to under system.

dobey

@simbion said in Set partition sizes when flashing:

It would be nice if did it in UBports when you install to select how much you need to under system.

Installing packages with apt and resizing the rootfs is not supported. Nobody needs it, though you might want it.

L-00327

@dobey then what can you do with ubuntu touch can you run snapd on my oneplus one, it is open source how can I get full space for my apt-get not just reserved for storage? already mounted r/w permanently with one liner $ adb shell

$ sudo touch /userdata/.writable_image

$ sudo reboot
If i could manage to somehow free up this space I'd be gucci for installing full web apps again, this didn't happen when I used to install it with multirom. Does anyone know if deleting the lock file will help thanks

dobey

@errorcodevortex No you can't run snapd on Ubuntu touch.

As stated, if you want to resize the root partition and use apt to install things, it is not supported. You can do it if you want, but it will not be supported.

The goal of Ubuntu Touch is to provide an operating system for phones based on Ubuntu. It is not to provide a traditional PC Linux distro, because phones are not built this way. If what you want is a traditional PC Linux distribution on your phone, with latest kernel and not all hardware working, but enough to do the few specific things you want to run on your device, perhaps you should look at https://postmarketos.org instead.

L-00327

@dobey Space used to be supported before lol enabled by default, did it change when it turned to ubports or something? now i can't figure out yet how to get more then an extra few gbs of my 64gb available. I had a lot of programs running sucessfully before. https://www.youtube.com/watch?v=NHE5THEWq7c this guy managed to install snapd but I'm assuming it was a 64bit device. Tried to do this https://www.linuxuprising.com/2018/07/anbox-how-to-install-google-play-store.html but couldn't install the install.sh it seemed only because of snapd.socket error. Heard of plasmamobile but not postmarket os but not trying to build anything from scratch that wont support calling. Might checkit out tho. You can install full linux on the worst computers there really aint no reason it cant be a linux distro and phone on my device. I've ran full kali-linux on this phone. Maybe this the attitude why it hasn't taken off yet, the whole point of linux is open source freedom it seems like developments gone backwards. Am I strictly supposed to use libertine for desktop stuff somehow is that "supported"? I haven't looked into this quite yet just got anbox going.

dobey

@errorcodevortex Your attitude is the problem here, demanding that this small community bend to support your own personal desires. That is not at all what Linux, Open Source, nor most of the communities around them, are about.

Libertine is the supported (albeit still an experimental feature) method of creating a container for installing legacy applications, yes.

If you want an insecure classic Linux distribution on your phone, then Ubuntu Touch is not it, and it never would have been, especially if Canonical had continued developing it in the direction they were going with.

There are plenty of reasons why it can't be a "full linux distro" by your standards, despite your refusal to accept how Android phones are designed and built.

On the other hand, if you do want something that is more like a traditional PC, where things could theoretically work more like how you want, and still want to run UT, then you could look at buying a PinePhone or Librem 5, both of which are not Android based, and therefore much more flexible in how things can work; though using apt to install additional packages and upgrade the system will still not be actively supported, it will work more reliably on such a system where partitioning can be better controlled by us, rather than having to conform to Android designs.

L-00327

This post is deleted!

L-00327

@dobey All I want is my space back I used to have, I don't think I'm asking for much here thats it essentially. I had installed a ubports version before also even with the multirom method as well didnt face the space issue same device. I understand how android works lol, I've been working with android/ computers for years. Bummed multirom app died out I had sailfish, touch, Tugapower 9, and cm13.1.2 all bootable before. This isn't even my main device right now but I'd like it to be daily driver like before not interested in buying another phone hence why I'm here revamping my old one. I understand its not exactly the same bootloader/ efi,partitition setup etc of course but end of the day it has the power to at the least to give me my space back. To say that not supported is bunk. I've installed entire operating systems on flash drives with less space. I just can't figure out how to symlink it right yet never had to deal with this issue before. I just can't see how it can expect to attract free linux developers this way under pretenses like that, it's putting me off already even though I do really like it, but it simply seems more cannonocized now than before. Not all ubports models have this specific space issue either it seems just some but nowhere have I seen anyone actually fix the issue.

Lakotaubp

@errorcodevortex

Firstly let me state I have little idea of what you are trying to achieve and can offer no technical help at all. From reading quickly through this thread (as I guess you have to) this comment sticks out

UniSuperBox 15 Oct 2018, 16:58

The image file can be resized, and I'm considering ways to make ubports-qa do this in case the user would prefer to use apt and turn off system-image upgrades. Of course we wouldn't be able to help people if they broke things in this state, but we can give them just enough rope to shoot themselves in the foot...

from this point on people have either succeeded or hit a wall in some way.
A couple of things. What you want to do may or may not be possible (I have no idea way over my head) but it is not one of the prime aims of this this project so with the small team of mainly volunteer developers we have is not going to be a priority. Booting multirom is a similar thing, possible but not supported.
As this thread was started nearly a year ago you may be better off asking a specific question relating to symlink in a new post and maybe linking back to this one for reference.

dobey

@errorcodevortex said in Set partition sizes when flashing:

All I want is my space back I used to have,

I have no idea what you're talking about, but resizing the rootfs has never been a supported feature of Ubuntu Touch. You're making an assumption based on a false memory, as far as I can tell. The rootfs has always been roughly 2GB loopback mount. MultiROM was also never officially supported. It's still possible to use it, but it is not officially supported, and you need a patched version to point at the correct image server.

Installing anything from apt in rootfs was never really supported either. Yes, it's somewhat possible with some hacking, but creating the .writable_image file explicitly disables system updates, leaving you with an unsupported system. You are literally on your own at that point. Also, due to how some files are bind mounted, to enable readonly rootfs, simply running apt-get upgrade will break at some point.

The OS is built around security and privacy. Therefore, yes, certain things which subvert those principles are not actually supported methods of using the system, even if you can do it with some hacking. Just like how rooting Android is not supported by Google. But I don't see you having a rant at them for it.

doniks

@dobey said in Set partition sizes when flashing:

@errorcodevortex said in Set partition sizes when flashing:
The rootfs has always been roughly 2GB loopback mount.

Still, I think there are two kinds of rootfs' those that reside inside the "system" partition, which are not (really) resizable and those residing inside a "system.img" file in /data/system.img. Even though resizing still isn't supported it is relatively easy to do with the image.

The Nexus 7 deb is installed in such an image file. I think multirom capitalized on the support for installing into a file and did all installations in that way. I don't know though what makes some installations go into a partition and some into a file.

dobey

@doniks said in Set partition sizes when flashing:

Even though resizing still isn't supported it is relatively easy to do with the image.

I never stated it couldn't be done. But it is not supported. and switching channels, or some in-channel updates, can/will overwrite it. I've only stated that it is not supported, and extremely problematic, but if you want to do it, the tools are there. We simply cannot support it as a normal feature of the system.

Also, ideally, we'd be able to ship the rootfs in the system partition, rather than in data, and have it truly be read-only without the option to mount it r/w at runtime, so that we could do boot verification and sign images to allow bootloader to be locked, and enable proper encryption of data partition. But unfortunately we aren't there yet. This would also be ideal on a PC, but alas we are still being held back there too.

Set partition sizes when flashing

Switching the release channel loses the extended rootfs

Moving forward

Switching the release channel loses the extended `rootfs`