Gotta release fast

neopar

@unisuperbox said in Gotta release fast:

Who is your ideal user for the fourth channel? Right now, the split is fairly clear:

I try to clarify my point of view and map suggested channels to the current channels we have:

devel -> current devel
testing -> current rc
rc -> current stable
stable -> kind of LTS release; for users who don't want to download the entire footfs image every week, but want to receive security updates

It is not relevant ATM, since most users are kind of beta testers. But it could be important when the community grows.

kugiigi

I don't know how much effort is needed every stable release but perhaps we can have a regular OTA and a some kind of a mini-OTA midway. This mini-OTA will include small bug fixes that won't most likely create regressions. An example of this are the layout issues in core apps. Regular OTA will be for bigger fixes and new features.

alan_g

@kugiigi said in Gotta release fast:

I don't know how much effort is needed every stable release but perhaps we can have a regular OTA and a some kind of a mini-OTA midway. This mini-OTA will include small bug fixes that won't most likely create regressions. An example of this are the layout issues in core apps. Regular OTA will be for bigger fixes and new features.

That sounds plausible, but I've worked (or consulted) on a lot of software projects and this is hard to make work. It is very difficult to manage updating both a released version and a development version without more than doubling the workload, It turns out there can be unexpected interactions between apparently small, safe changes.

It is better to be in a position where there's no need to manage two sets of changes. This does have risks (which is why devel breaks) and we have to mitigate those risks. One way is to release often so that the number of changes in each release is small, another is to have easy ways to revert to a previous good release, another is to release to a small group before rollout.

With the proposed use of rc a group of "canaries" try changes on rc first to identify unexpected issues.

Which reminds me: one essential test is that rc can revert back to stable in the event of problems.

dobey

@alan_g One should always be able to revert to the previous image in the same channel as well. The main place where this becomes problematic, is for things which are shipped as clicks, rather than as the more immutable system image. Reverting to previous clicks is nigh impossible (without already having the old click), and we don't have stable/testing/devel channels for things in the app store.

alan_g

@dobey said in Gotta release fast:

@alan_g One should always be able to revert to the previous image in the same channel as well. The main place where this becomes problematic, is for things which are shipped as clicks, rather than as the more immutable system image. Reverting to previous clicks is nigh impossible (without already having the old click), and we don't have stable/testing/devel channels for things in the app store.

That is good both for the user and for the channel owner. Is that supported?

dobey

@alan_g said in Gotta release fast:

That is good both for the user and for the channel owner. Is that supported?

I would say so, though there isn't UI for it at the moment. We could probably make some changes to the UI to make it easier to select any of the available images in a channel, to switch to (assuming we only keep N images in a channel, rather than all images ever built).

Currently one needs to do the revert either using ubuntu-device-flash or by using system-image-cli directly on the device via an adb/ssh shell.

Lakotaubp

@dobey Would be good to have an easy UI rather than having to resort to terminal etc. Only I don't know how difficult it would be to include.

UniSuperBox

For clicks:

The OpenStore can (and does) downgrade Clicks when it sees fit, but there is currently no way in the UI to select an older app version to download and install.

For images:

There is currently no way to select which image in a channel you wish to use. You have the latest, always.

dobey

@unisuperbox said in Gotta release fast:

There is currently no way to select which image in a channel you wish to use. You have the latest, always.

You can specify which revision of an image to install, and from which channel, with the two tools I mentioned. I presume the installer always uses the latest image and has no way to specify a build number, but I do not know with certainty, so I didn't mention it.

I think it would be pretty easy to add something to the System Settings updates/channels panel though.

UniSuperBox

As we near the OTA-6 release, I would like to gather what I believe are the most important points from this post:

Everyone wants "release faster", without a doubt.
"Release faster" depends on having enough confidence that software being released to stable is well-tested, which we do not have. Some things which would translate into more release confidence include:
- Automated tests on all system components
- Integration tests between system components
- Automated full-device testing, such as Canonical's fabled "Frankenstein device lab"
- Formalized manual testing by users who can confirm stability
Even with full release confidence (but especially without), an automatic release model requires the ability to roll back to a previous version of apps and the full system image
- Aside: I want to experiment with one of the new atomic release models like OSTree which allows agnostic packaging formats to be installed on top. From my quick foray, I see:
  - Benefits: Automatic and manual rollbacks, automatic diffing, and switchable roots
  - Drawbacks that make this a long-term or impossible idea for our existing devices
    - Probably requires newer kernel versions than 3.4 or 3.10
    - We don't have enough engineering power to concoct a solution which would allow converting a system-image system to OSTree in-flight, so we'd probably require users to manually switch.

I think that bringing up the edge channel as a place to do a large migrations out-of-band with the normal release cadence is going to have a huge impact on how quickly we can release in the future. Previously, @mariogrip's work on bringing us to upstream Libhybris would have had to wait until after OTA-6 lands, and then it would have delayed OTA-7 until we had proper confidence in it. With edge, we can have people help us test these huge changes with the ability to roll back and without disrupting current users.

To respond to my earlier hindsight: OTA-6 was not (yet ) taken away by TDS. We did not hit the original deadline set, but we will still be within our 6-8 week cycle when it releases. This includes our testing and release admin stage, which we are currently in. All of this while I'm not sweating bullets from pushing a release on ourselves before we're sure we're ready.

We have ended up in "Gotta release slower" in this cycle without a doubt. I hope to make the improvements we've identified here so "gotta release fast" can become a reality. I also hope to be able to take our release management up so that we hit 2-4 weeks development, 2 weeks testing. This means we'll get faster releases, but we still won't hit fast releases.

Important note for anyone interested in contributing to... basically anything I've said here: I'll be holding an OTA-7 development meeting at the start of the cycle where we'll discuss how we want to improve during the OTA-7 cycle and taking bugs from the tracker for the release. Only assigned tickets will be added to this milestone, so if you have a pet bug that you want fixed now is the time to get involved to help fix it! Subscribe to this GitLab issue for more information as the day draws nearer.