Language packs as click packages: call to arms!

mardy

Hi all,
I've written a length e-mail to the development mailing list here.

Your input is very appreciated, either in the ML or as a reply to this forum message.

Ciao,
Alberto

dobey

A) There's a development list?

Please nooooooo. We don't need to have langpacks at all. it's an outdated anti-feature in Ubuntu, created to fit the live ISO on smaller CD-ROMs. We really do not need them. We really should just include support for all languages we support, by default, and optimize irrelevant things out of the image if there are space concerns.

mardy

@dobey said in Language packs as click packages: call to arms!:

A) There's a development list?

Yes, but it feels like I'm the only one using it

Please nooooooo. We don't need to have langpacks at all. it's an outdated anti-feature in Ubuntu, created to fit the live ISO on smaller CD-ROMs. We really do not need them. We really should just include support for all languages we support, by default, and optimize irrelevant things out of the image if there are space concerns.

But we do have this size problem. Saving 300 megabytes is a huge thing. And someday we might want to include text to speech or speech recognition programs, or other language-related features (AFAICT, word prediction is now enabled only for a few languages), so it's easy to predict that this size will grow even more. What exactly is the problem with click language packs?

I suspect that your adversion to the language packs stems from the way they are used in Ubuntu; if so, I fully agree. I do not understand why we need to have locale data in /usr/share/locale-langpack/ in addition to /usr/share/locale/; and even worse, in some cases we do have the translation files in both directories for the same language/program, where the translations files are different, and I have no clue which one is the authoritative one. So yes, it's a mess. But if you read the code in my rootstock-ng branch you'll understand that I'm not depending on the Ubuntu language packs feature: I need to deal with it since it's there, but my solution would work even if the traditional Ubuntu language packs were not there (and actually, it would work better).

What I'm doing is just taking the final rootfs and moving data out into clicks. It's a solution that works regardless of how translations end up in the rootfs in the first place.

mymike

I fully agree with @mardy regarding the space problem on the rootfs, unfortunately I don't know about technical matters so cannot comment about it...

A Former User

Speech to Text would be a valuable feature, down the line.

dobey

@mardy said in Language packs as click packages: call to arms!:

@dobey said in Language packs as click packages: call to arms!:
But we do have this size problem. Saving 300 megabytes is a huge thing. And someday we might want to include text to speech or speech recognition programs, or other language-related features (AFAICT, word prediction is now enabled only for a few languages), so it's easy to predict that this size will grow even more. What exactly is the problem with click language packs?

We just got rid of gdbserver/libc-dev, which freed up a huge amount of space. The only reason we have size problems with the image now, is because it's never really been optimized, and we just keep adding new things to it.

I suspect that your adversion to the language packs stems from the way they are used in Ubuntu; if so, I fully agree. I do not understand why we need to have locale data in /usr/share/locale-langpack/ in addition to /usr/share/locale/; and even worse, in some cases we do have the translation files in both directories for the same language/program, where the translations files are different, and I have no clue which one is the authoritative one. So yes, it's a mess. But if you read the code in my rootstock-ng branch you'll understand that I'm not depending on the Ubuntu language packs feature: I need to deal with it since it's there, but my solution would work even if the traditional Ubuntu language packs were not there (and actually, it would work better).

My aversion to langpacks stems from the fact that they are an anti-feature, and a premature optimization. How they are done is implementation details. One doesn't have to install any extra packages on Android to get foreign language keyboards. Core apps may be fully translated, but vast majority of 3rd party apps will be missing translations. To me, langpacks are just another way to sweep some problems under the carpet, so we end up ignoring them, rather than fixing the real problems.

mardy

@dobey said in Language packs as click packages: call to arms!:

We just got rid of gdbserver/libc-dev, which freed up a huge amount of space. The only reason we have size problems with the image now, is because it's never really been optimized, and we just keep adding new things to it.

Sure, and soon we'll get rid of Oxide, I guess, so the future is at least a bit rosy. But having this lot of data which I don't need in the rootfs is a waste, you'll agree.

My aversion to langpacks stems from the fact that they are an anti-feature, and a premature optimization. How they are done is implementation details. One doesn't have to install any extra packages on Android to get foreign language keyboards. Core apps may be fully translated, but vast majority of 3rd party apps will be missing translations. To me, langpacks are just another way to sweep some problems under the carpet, so we end up ignoring them, rather than fixing the real problems.

It's not a premature optimization, it's a way to get rid of 307 MB (and more, in the future). The way that this feature is exposed in the user interface matters a lot: it can range from the ideal situation (the user doesn't notice, and all the langauges he needs are already on the device) from the worst one (the user needs to open a terminal and type some obscure commands to install the language packs, after which the device enters a reboot loop).

The way I see it, we are close to ideal situation, for the very simple reason that (unless everybody agrees otherwise) downloadable images having all the language data preinstalled are not going away. As long as the device can accomodate them and users don't complain about lack of space, that's undoubtly the best solution.

Now, what I want to do is part of a greater and much more evil plan: customizable images (and also, possibly, changing the way that clicks are preinstalled on the device). The idea would be that the flasher tool would offer you an option:

Download and install default image
Download minimal image + select individual click packages

Case #1 is like nowadays: all languages will be preinstalled. In case #2, the flasher tool would let the user select among click packages to be added on top of the base image: language packs, sure, but also ordinary applications and other click packages types that will appear in the future (I've a plan to add click packages with regional map data and wifi information for location lookup). I haven't yet thought how these will be installed on the device, but thinking out loud, I see these possibilities:

The flasher tool downloads and install the clicks itself
The flasher tool simply writes the list of clicks into a configuration file on the device, which will download and install them during the first boot
Something in between (downloading the clicks from the PC, copying them over, but installing them on first boot)

The nice thing about this, no matter how we implement it, is that the flasher tool could remember the list of clicks the user has selected, and use it as a default selection for the next time. Or it could also try to read the list of the click packages installed on the connected device and prompt to use that, instead. And I bet there are more possibilities for making users' lives easier that I haven't thought of.

Anyway, the main idea is to let the user select the extra packages, and remember this list across installations and backups. And I believe that if this is done in the right way, users will on the contrary appreciate the feature (and they could see the device boot up in their own language on the first use, imagine that! )

dobey

@mardy Are we discussing language packs as clicks, or a more general solution to "what apps are installed by default" when flashing a phone? Because those seem like two separate things, and we should not use one of them as a reason to do another thing (or to not do some other different thing instead), I think.

I also disagree with the idea of anecdotal argumentation of well, it's a waste in the rootfs for me, so why should it be there for others. I think it would be best to not go down this route. There are plenty of things in the rootfs which are meaningless for me too. Also, the rootfs is meant to be read-only, so supporting having more room in it to install things via apt, I think shouldn't be a primary concern here. Of course it will be nice to have a smaller rootfs, but I think this is also not too big a concern, as we already have a set limit for what size the rootfs should be. Then again, many devices we support have external storage via microSD slots, but we do not support installing apps or storing application data on them. Another way to greatly increase available storage for customers who want to install more apps than internal storage allows for.

Maybe having langpacks is something we will need to do in the future. But I think right now, we should not rush into this as a solution to the problems we have, as there are plenty of larger concerns at the moment. And if we rush into it, we will be stuck for a long time, even if we discover some other solutions to whatever problem this is meant to solve, instead.

mardy

@dobey said in Language packs as click packages: call to arms!:

@mardy Are we discussing language packs as clicks, or a more general solution to "what apps are installed by default" when flashing a phone? Because those seem like two separate things, and we should not use one of them as a reason to do another thing (or to not do some other different thing instead), I think.

I'm not using one as a reason to do the other, but rather I'm explaining how I imagine this evolving into, to convince you that we can get a nice user experience even with selectable language packs.

I also disagree with the idea of anecdotal argumentation of well, it's a waste in the rootfs for me, so why should it be there for others. I think it would be best to not go down this route. There are plenty of things in the rootfs which are meaningless for me too.

And what I'm suggesting is that, if those things could easily be installed as clicks, and if doing so would save a considerable amount of space, then it's something we might want to try out. It's a question on whether it's worth the effort.
I did start with language packs because I thought that it would be an easy pick, since they are data-only. And so far it looks like it's something easily doable.

[...]

Maybe having langpacks is something we will need to do in the future. But I think right now, we should not rush into this as a solution to the problems we have, as there are plenty of larger concerns at the moment. And if we rush into it, we will be stuck for a long time, even if we discover some other solutions to whatever problem this is meant to solve, instead.

I'm not requesting that people spend their time on this. If @bhdouglass told me that he has no time to review the changes I'll eventually submit to the Open Store, then I certainly wouldn't force him, but it looks like he likes the idea as well.
And I'll try not to rush: I do want to see a good implementation of this (especially because, as I said, I plan to apply a similar scheme for navigation data, later on).

Could I spend my time in a better way? Probably, but I'm doing this as a hobby in my spare time, so I tend to work on those things that interest me first