The current OTA cadence isn't working with our development tooling and workflow. Let's fix that.
The release of OTA-5 was a bit rough.
- The decision to release came too late and didn't appropriately echo through every team involved.
- There wasn't enough time to appropriately review all release material and make it the best it could be
- It wasn't well communicated that the release would be phased over 5 days -- it'll be in everyone's hands on Wednesday
All seems like communication problems, right?
Knowing what we know now (even this early after the release), I think I can speak for the entire team here:
- We allowed ourselves to be taken away by Target Date Syndrome. That is, while hitting your target date is important, great software is more important.
- We should have delayed the release rather than pushing it exactly when it said on the GitHub milestone. Even with the Ubuntu release clouding the news for the next couple of weeks, another week of development (and another of hard testing) would have been preferred to a hard push out the door.
- We should have made a new framework version. The OpenStore (starting today) is able to hold updates back from people who don't have a framework version. This would have solved the pain people are experiencing with the Notes app.
But the release still had some good points:
- It was on time according to the GitHub milestone (we had pushed the release back a week earlier in the cycle)
- The new features are impressive. Our browser is faster and more secure than it's ever been. New users are hearing better sounds (thanks @malditobastardo!). Upgraders are being met with a guided setup. Our Korean users have a keyboard all their own.
- That doesn't mean that OTA-5 is perfect. It has its bugs, and we'd appreciate if you report any you find.
So what now?
Some of the issues and hindsight I mentioned above are just lessons that we needed to learn sooner or later. Now that we've been bitten, we'll let the scar remind us to do right.
We have a couple of options that I'll present here that should help us improve the pre-, during-, and post-release process.
Gotta release slower...
Our first option, which we were previously leaning toward, is to release on an 8-week cadence (with the option to delay if needed). We'd split out every milestone (and associated Stable release) like this:
- Week 1: Introspection
- Watch for critical issues coming in as the release phases in, be ready to stop
- Discussion of what went well, what didn't go well, and what is looking promising but could improve from our last cycle (also called "roses, thorns, and buds")
- Accept bugs into the current milestone by what developers say they will take during the cycle
- Weeks 2-6: development
- Merge larger changes earlier in cycle
- bug fixes and performance improvements
- Whenever anything happens, write it down to prepare for release announcements
- Week 7: Does it test okay?
- Required "Release OKs" are entered based on each team's testing and completed work:
- Core development (Either @mariogrip or myself)
- App development (@bhdouglass for sure)
- Quality Assurance head (@neothethird or another leader of the QA team)
- If any "Release OK" is not received or is denied, the release moves back to development to mend the problem.
- Week 8: It's done, prep for release!
- Make release pre-announcements heard throughout all the halls of UBports. No one should be surprised that an OTA is coming, unless they're living under a rock and their ISP is buried under a mountain of rubble
- Continue testing just in case
- Prepare release blog posts
- Inform press that a release is coming
devel ships nightly,
rc (Release Candidate) ships every week, and
stable ships upon completion of all of this.
The reason this would be pinned to an 8-week cycle is simple... with our set of (highly skilled and totally appreciated, I must say) contributors, it isn't feasible to promise any less than this without being in a constant state of "crunch time." That isn't safe for contributors, paid or otherwise.
As I type this, I find there is no reason that the week numbers must be hardcoded, though... at any point during the cycle where all major teams OK the release, it is ready to ship. When all these OKs occur, a full week is needed to complete the release.
- Formalizes our current release method
- Encourages involvement of community members (OK'ers need to get their team's blessing to put in their Release OK)
- Security fixes are getting to users of Stable only every 8+ weeks. That is widely regarded as unacceptable in today's climate.
- Real emphasis on testing isn't applied until week 7. Ideally we'd be hitting testing hard on weeks 1-8, but we are humans after all...
- RC also doesn't make much sense until week 7, unless you're a developer
- Since many changes can happen in 6 weeks, we've found this can be scary for upgrading users
- The "Release OK" process could be a cause for contention among teams and their leaders
- I really don't expect this, but I've been proven wrong before
Gotta release fast!!!
I hear you say: Ah, so that's where the title came from...
This is the polar opposite of "Gotta release slower". If you must wave the buzzword flag, this is closer to an "Agile" method. Instead of slowing down the cadence, we ensure that there is one release to the stable channel almost every week. The release of a stable image looks like this:
- Devel images built nightly
- On Thursday (example October 18th), the devel image from Tuesday (example October 16th) is promoted to RC (unless there is an issue on ubports/ubuntu-touch labeled
- The following Thursday (example October 25th), the RC image is promoted to Stable (unless there is an issue labeled
critical-rc). The image is phased in over a period of 4-5 days.
At most, users of Stable will be 15 days behind the current devel (Please correct me if I'm wrong... I'm not always the best with date logic). I'm open to adjusting the dates slightly as needed.
If we felt it important, we could have the QA team OK these releases too. Nothing severe, just a "type the version number, check the box if no one's phone is broke".
Our milestone system would move to milestones for each major feature rather than each major release, followed by Epics on GitLab for the same purpose as we slowly transition over. As parts of features land, the milestone becomes more complete.
I'm not 100% sure how we'd label releases at this point... @mariogrip insists on using semantic versioning based on the API stability of the development framework. I say that we should use the week number of the devel build that creates the stable build, so someone getting the build on Thursday, the 18th of October, 2018, would receive 2018-W41. You can discuss that over here.
- We are never more than 15 days (again, correct me if I'm wrong) behind upstream Ubuntu security fixes on the Stable channel
- There is always a clearly testable image (you get new shiny things faster on RC)
- There is always a clear need for testing (if you're on RC, you're helping protect users on Stable)
- People don't press the big red button
- Without the "OK!" process, there is less chance for major disagreement harming users
- Less chance for mistakes while releasing or phasing
- Bugfixes get to users faster
- Developers (who should be using devel, glare) are using software closer to our users. When we were nearing the end of the OTA-4 cycle, the Ubuntu Touch we were using was very different from what most people were using.
- We're asking users to update a lot
- Not necessarily a bad thing. Most desktop distributions I've used default to notifying users of updates every week. LineageOS also checks once per week.
- Faster releases mean less potential testing (bugs hit users faster)
- Higher chance of SystemImage failure (since we update more)
So... what now?
Personally, I'm leaning toward releasing fast... I think that the best way to release as a community is to make the computers do it instead.
Depending on the methodology we decide as a community (and how fast we do it!), OTA-6 may be released under either option. If we decide in fewer than 2 weeks, OTA-6 will definitely use whatever we decide. If OTA-6 is ready before we decide, it will be the last release before we move to the new method.