Gotta release fast

UniSuperBox

The current OTA cadence isn't working with our development tooling and workflow. Let's fix that.

The problem(s)

The release of OTA-5 was a bit rough.

The decision to release came too late and didn't appropriately echo through every team involved.
There wasn't enough time to appropriately review all release material and make it the best it could be
It wasn't well communicated that the release would be phased over 5 days -- it'll be in everyone's hands on Wednesday

All seems like communication problems, right?

The hindsight

Knowing what we know now (even this early after the release), I think I can speak for the entire team here:

We allowed ourselves to be taken away by Target Date Syndrome. That is, while hitting your target date is important, great software is more important.
We should have delayed the release rather than pushing it exactly when it said on the GitHub milestone. Even with the Ubuntu release clouding the news for the next couple of weeks, another week of development (and another of hard testing) would have been preferred to a hard push out the door.
We should have made a new framework version. The OpenStore (starting today) is able to hold updates back from people who don't have a framework version. This would have solved the pain people are experiencing with the Notes app.

But the release still had some good points:

It was on time according to the GitHub milestone (we had pushed the release back a week earlier in the cycle)
The new features are impressive. Our browser is faster and more secure than it's ever been. New users are hearing better sounds (thanks @malditobastardo!). Upgraders are being met with a guided setup. Our Korean users have a keyboard all their own.
- That doesn't mean that OTA-5 is perfect. It has its bugs, and we'd appreciate if you report any you find.

So what now?

Some of the issues and hindsight I mentioned above are just lessons that we needed to learn sooner or later. Now that we've been bitten, we'll let the scar remind us to do right.

We have a couple of options that I'll present here that should help us improve the pre-, during-, and post-release process.

Gotta release slower...

Our first option, which we were previously leaning toward, is to release on an 8-week cadence (with the option to delay if needed). We'd split out every milestone (and associated Stable release) like this:

Week 1: Introspection
- Watch for critical issues coming in as the release phases in, be ready to stop
- Discussion of what went well, what didn't go well, and what is looking promising but could improve from our last cycle (also called "roses, thorns, and buds")
- Accept bugs into the current milestone by what developers say they will take during the cycle
Weeks 2-6: development
- Merge larger changes earlier in cycle
- bug fixes and performance improvements
- Whenever anything happens, write it down to prepare for release announcements
Week 7: Does it test okay?
- Required "Release OKs" are entered based on each team's testing and completed work:
  - Core development (Either @mariogrip or myself)
  - App development (@bhdouglass for sure)
  - Quality Assurance head (@neothethird or another leader of the QA team)
- If any "Release OK" is not received or is denied, the release moves back to development to mend the problem.
Week 8: It's done, prep for release!
- Make release pre-announcements heard throughout all the halls of UBports. No one should be surprised that an OTA is coming, unless they're living under a rock and their ISP is buried under a mountain of rubble
- Continue testing just in case
- Prepare release blog posts
- Inform press that a release is coming

As always, devel ships nightly, rc (Release Candidate) ships every week, and stable ships upon completion of all of this.

The reason this would be pinned to an 8-week cycle is simple... with our set of (highly skilled and totally appreciated, I must say) contributors, it isn't feasible to promise any less than this without being in a constant state of "crunch time." That isn't safe for contributors, paid or otherwise.

As I type this, I find there is no reason that the week numbers must be hardcoded, though... at any point during the cycle where all major teams OK the release, it is ready to ship. When all these OKs occur, a full week is needed to complete the release.

Possible benefits

Formalizes our current release method
Encourages involvement of community members (OK'ers need to get their team's blessing to put in their Release OK)

Possible problems

Security fixes are getting to users of Stable only every 8+ weeks. That is widely regarded as unacceptable in today's climate.
Real emphasis on testing isn't applied until week 7. Ideally we'd be hitting testing hard on weeks 1-8, but we are humans after all...
- RC also doesn't make much sense until week 7, unless you're a developer
Since many changes can happen in 6 weeks, we've found this can be scary for upgrading users
The "Release OK" process could be a cause for contention among teams and their leaders
- I really don't expect this, but I've been proven wrong before

Gotta release fast!!!

I hear you say: Ah, so that's where the title came from...

This is the polar opposite of "Gotta release slower". If you must wave the buzzword flag, this is closer to an "Agile" method. Instead of slowing down the cadence, we ensure that there is one release to the stable channel almost every week. The release of a stable image looks like this:

Devel images built nightly
On Thursday (example October 18th), the devel image from Tuesday (example October 16th) is promoted to RC (unless there is an issue on ubports/ubuntu-touch labeled critical-devel)
The following Thursday (example October 25th), the RC image is promoted to Stable (unless there is an issue labeled critical-rc). The image is phased in over a period of 4-5 days.

At most, users of Stable will be 15 days behind the current devel (Please correct me if I'm wrong... I'm not always the best with date logic). I'm open to adjusting the dates slightly as needed.

If we felt it important, we could have the QA team OK these releases too. Nothing severe, just a "type the version number, check the box if no one's phone is broke".

Our milestone system would move to milestones for each major feature rather than each major release, followed by Epics on GitLab for the same purpose as we slowly transition over. As parts of features land, the milestone becomes more complete.

I'm not 100% sure how we'd label releases at this point... @mariogrip insists on using semantic versioning based on the API stability of the development framework. I say that we should use the week number of the devel build that creates the stable build, so someone getting the build on Thursday, the 18th of October, 2018, would receive 2018-W41. You can discuss that over here.

Possible benefits

We are never more than 15 days (again, correct me if I'm wrong) behind upstream Ubuntu security fixes on the Stable channel
There is always a clearly testable image (you get new shiny things faster on RC)
There is always a clear need for testing (if you're on RC, you're helping protect users on Stable)
People don't press the big red button
- Without the "OK!" process, there is less chance for major disagreement harming users
- Less chance for mistakes while releasing or phasing
Bugfixes get to users faster
Developers (who should be using devel, glare) are using software closer to our users. When we were nearing the end of the OTA-4 cycle, the Ubuntu Touch we were using was very different from what most people were using.

Possible problems

We're asking users to update a lot
- Not necessarily a bad thing. Most desktop distributions I've used default to notifying users of updates every week. LineageOS also checks once per week.
Faster releases mean less potential testing (bugs hit users faster)
Higher chance of SystemImage failure (since we update more)

So... what now?

Personally, I'm leaning toward releasing fast... I think that the best way to release as a community is to make the computers do it instead.

Depending on the methodology we decide as a community (and how fast we do it!), OTA-6 may be released under either option. If we decide in fewer than 2 weeks, OTA-6 will definitely use whatever we decide. If OTA-6 is ready before we decide, it will be the last release before we move to the new method.

elastic

@unisuperbox thanks for sharing the thoughts of the team... I second the release every fortnight as updates should come fast and predictibal - the weekly release of lineage is highly anticipated by their user base...
of course one can follow the xfce path as well it's ready when it's ready

Luksus

Yes, do a fast release.
Currently (before last OTA) there are so much fixes, that the stable users have to wait for one or two months.
I think it's better to get fixes to stable channel quick.

alan_g

In the ideal, Agile, world software should always be in a releasable state and the timing of releases should be determined by user readiness and the cost of releasing (not dates or feature completion).

The costs of releasing here are administrative time and bandwidth. Correct me if I'm wrong, but I don't think either is a reason to block "release faster".

I think I can speak for all the users here and say: we are always ready to receive the latest working version.

The difference between this ideal world is that we can't be sure that things are in a releasable state just because the automated tests pass. Not all the projects that make up UT have adequate automated tests to guarantee their correctness and we don't have test coverage that everything works together on every device.

So, I'm all for "release fast" and trying to mitigate any issues that throws up...

Having a couple of days to identify critical issues on devel before moving to rc sounds sensible to me.

The criteria for moving from rc to stable needs to be clearer or stricter. I suggest that any new issues on rc (i.e. ones that don't also exist on stable) should default to critical-rc unless there is an explicit group decision otherwise.

Also, I think we have enthusiasts around that will happily try rc. Maybe there could be a place where they can sign off that they have "used it for 24 hours on XXX device" to feed into the release-to-stable decision? Delaying a day (or two) for better empirical testing shouldn't affect the cadence too much.

dobey

So, in the Canonical days, there was a lot more testing, with more automation, along with more visibility into things like unit test coverage, and crash reports.

Could we perhaps document where we differ now from what processes and features were in place at that time, so we can perhaps bring back some of those in a manner suitable for UBports, and then go from there?

In particular, the tests situation is really bad now with UBports, as many packages just have tests disabled during the deb builds, autopilot tests aren't being run anywhere, and we don't have the infrastructure set up to be building with coverage enabled.

rogier.oudshoorn

My vote is for FAST. Smaller changes at every release usually means it's easier to fix, and said fix is always only 2 weeks away (unless you also want to be able to do hotfixes to stable, which might mean days away).

Given all the above, i would suggest having a manual step before RC gets promoted to stable; just to have someone look around to check how many people have actually used the RC to address stability. It also means that, in case of vacation times, nothing gets promoted automatically.

I do have a question: can you actually OTA the entire population within a day?

dobey

Everyone would obviously vote for fast. But we already have that, so why are they not just using the devel channel?

We don't need faster (or slower) releases. We need verifiable releases. Once a month for stable channel is plenty fast enough a cadence, and we could do interim security fix releases if necessary. If you're on stable, that's plenty fast for releases, and people who want faster can switch to rc, or even devel. System updates being released too often on stable channel just makes the product seem immature, while not enough updates makes it look unmaintained. We need to find the happy medium there.

But if we can't verify the reliability of the software, any release cadence is going to be problematic.

zx81

I prefer the fast option but I am happy to go with the flow if the slower option is preferred by the majority.

At the end of the day you guys are doing a fantastic job but can't be expected to devote your entire time to it .

vandys

I have to say that the transition to Xenial was a huge step, almost a defining milestone for the project. I think you made the right call after that big hurdle to get it out, even with some imperfections. Maybe from here on onward you might choose to hold back, but I think for something as big as your new release, it was right to get it out in the world rather than letting it drag on like one of those never-released "1.0"'s.

neopar

Great to see there is public discussion about this topic!

I think the point is to pick up almost every UT user, since there are not many of them. There are still users using Canonical version out there.

In some projects we used more layers to solve that kind of problems. In UT world the approach could be to introduce one more release channel (maybe that would be the happy medium metioned by @dobey

devel → will likely break something on your phone
testing → kinda mix of current devel and rc
rc → stable enough to use as daily driver, well tested, introducing new feauters frequently
stable → rock solid version, for users who don’t care much about new features they don’t need ATM.

The user could decide to use rc or stable without being afraid that something will beak with next update.

But there should be a way to deliver critical bug fixes on any channel instantly.

A Former User

@elastic: "of course one can follow the xfce path as well it's ready when it's ready"

I think there's a lot to be said for that, actually. It would give things a more natural flow.

UniSuperBox

I love the discussion that this has garnered. I think we can take all of the points to bring together a better plan soon.

@alan_g:

The criteria for moving from rc to stable needs to be clearer or stricter. I suggest that any new issues on rc (i.e. ones that don't also exist on stable) should default to critical-rc unless there is an explicit group decision otherwise.

Good choice. It would require some very on-the-ball triagers keeping track of bugs and being able to mark anything new in this RC as a regression. I'm not sure GitHub or GitLab automation would have the tools we need to make that automatic.

Maybe there could be a place where they can sign off that they have "used it for 24 hours on XXX device" to feed into the release-to-stable decision?

Another good choice. Something as simple as a form where they enter their device, the version they're using, and say "yes" or "no" could work. Having trusted testers in our QA group with ultimate sign-off powers is what I'd come back with, though.

@dobey:

Could we perhaps document where we differ now from what processes and features were in place at that [Canonical] time, so we can perhaps bring back some of those in a manner suitable for UBports, and then go from there?

Hmm. I don't have a clear understanding of what those processes were, to be honest. I was just getting started with UBports when Canonical stopped releasing updates altogether. Could you give a clearer picture of what you're looking for, maybe some wiki.ubuntu.com pages?

This post is my first try for getting our processes compiled into a coherent place. Later, a fully fleshed out release document would be placed on docs.ubports.com.

Once a month for stable channel is plenty fast enough a cadence, and we could do interim security fix releases if necessary.

A good plan on its face, but the rolling development nature might make the "Interim security releases" part sticky. What if there's an in-image security fix at week 2 but we merged in a feature at week 1 that is still undergoing full QA (since we have a month, right?) prior to release? I think this situation would be fairly common and I wouldn't be comfortable pulling the release trigger in that case.

the tests situation is really bad now with UBports, as many packages just have tests disabled during the deb builds, autopilot tests aren't being run anywhere, and we don't have the infrastructure set up to be building with coverage enabled.

Correct on all counts. While Autopilot might be a problem (no one is maintaining it), autopkgtests should definitely be running.

System updates being released too often on stable channel just makes the product seem immature, while not enough updates makes it look unmaintained. We need to find the happy medium there.

Maybe if not weekly, then every 2 weeks. My primary concern is discouraging a mindset of "bring it in for QA, the release is 4 weeks away" and encouraging "QA, can you look at this a lot before we bring it in?" (this applies to @3arn0wl's quote from @elastic, too)

@neopar:

I think the point is to pick up almost every UT user, since there are not many of them. There are still users using Canonical version out there.

Not really. The only way we'll reach the people on Canonical builds is with marketing and word of mouth. The release cadence doesn't relate to that.

In some projects we used more layers to solve that kind of problems. In UT world the approach could be to introduce one more release channel

Who is your ideal user for the fourth channel? Right now, the split is fairly clear:

developers who are merging things into the image should have devel on at least one device which they update extremely often.
Users who want faster updates and QA testers should have rc on all their devices
Users who want only the most tested software should have stable on all their devices.

I'm not sure where "testing" would fit in here.

But there should be a way to deliver critical bug fixes on any channel instantly.

The rolling release model shoots this as far as I can see. That's why I'd prefer a faster, but scheduled, release.

alan_g

@unisuperbox said in Gotta release fast:

Good choice. It would require some very on-the-ball triagers keeping track of bugs and being able to mark anything new in this RC as a regression. I'm not sure GitHub or GitLab automation would have the tools we need to make that automatic.

UBports has an unpaid (or even paying!) community who want to get involved but may not have the skills or resources to help with writing the code. Reviewing issue reports is an essential task with a modest learning curve and can be done with limited resources such as time.

All that is required is to:

Ensure the report describes a problem
Ensure there is enough information to reproduce the problem
Ensure the affected device(s), channel(s) and version(s) are mentioned
Mark the issue as "checked"

It should be clear (but probably isn't) that this is something anyone with "5 minutes" can help with. It amounts to putting the issue into this outline:

Title
How to reproduce
Expected result
Actual result
Devices, channels & versions
Confirmed by

The above has to happen before additional steps, that requires more knowledge or skills, can be taken:

Ensure the bug is reported against the right project
Assess the impact
Prioritize the work to fix
Do the work to fix

neopar

@unisuperbox said in Gotta release fast:

Who is your ideal user for the fourth channel? Right now, the split is fairly clear:

I try to clarify my point of view and map suggested channels to the current channels we have:

devel -> current devel
testing -> current rc
rc -> current stable
stable -> kind of LTS release; for users who don't want to download the entire footfs image every week, but want to receive security updates

It is not relevant ATM, since most users are kind of beta testers. But it could be important when the community grows.

kugiigi

I don't know how much effort is needed every stable release but perhaps we can have a regular OTA and a some kind of a mini-OTA midway. This mini-OTA will include small bug fixes that won't most likely create regressions. An example of this are the layout issues in core apps. Regular OTA will be for bigger fixes and new features.

alan_g

@kugiigi said in Gotta release fast:

I don't know how much effort is needed every stable release but perhaps we can have a regular OTA and a some kind of a mini-OTA midway. This mini-OTA will include small bug fixes that won't most likely create regressions. An example of this are the layout issues in core apps. Regular OTA will be for bigger fixes and new features.

That sounds plausible, but I've worked (or consulted) on a lot of software projects and this is hard to make work. It is very difficult to manage updating both a released version and a development version without more than doubling the workload, It turns out there can be unexpected interactions between apparently small, safe changes.

It is better to be in a position where there's no need to manage two sets of changes. This does have risks (which is why devel breaks) and we have to mitigate those risks. One way is to release often so that the number of changes in each release is small, another is to have easy ways to revert to a previous good release, another is to release to a small group before rollout.

With the proposed use of rc a group of "canaries" try changes on rc first to identify unexpected issues.

Which reminds me: one essential test is that rc can revert back to stable in the event of problems.

dobey

@alan_g One should always be able to revert to the previous image in the same channel as well. The main place where this becomes problematic, is for things which are shipped as clicks, rather than as the more immutable system image. Reverting to previous clicks is nigh impossible (without already having the old click), and we don't have stable/testing/devel channels for things in the app store.

alan_g

@dobey said in Gotta release fast:

@alan_g One should always be able to revert to the previous image in the same channel as well. The main place where this becomes problematic, is for things which are shipped as clicks, rather than as the more immutable system image. Reverting to previous clicks is nigh impossible (without already having the old click), and we don't have stable/testing/devel channels for things in the app store.

That is good both for the user and for the channel owner. Is that supported?

dobey

@alan_g said in Gotta release fast:

That is good both for the user and for the channel owner. Is that supported?

I would say so, though there isn't UI for it at the moment. We could probably make some changes to the UI to make it easier to select any of the available images in a channel, to switch to (assuming we only keep N images in a channel, rather than all images ever built).

Currently one needs to do the revert either using ubuntu-device-flash or by using system-image-cli directly on the device via an adb/ssh shell.

Lakotaubp

@dobey Would be good to have an easy UI rather than having to resort to terminal etc. Only I don't know how difficult it would be to include.