Freedesktop sdk aka 'tiny base runtime' project

[Laurence Urhegyi]

Hi all,

So i wanted to kick off this mailing list by trying to round up a few
things which have been discussed recently, at GUADEC, in f2f
conversations afterwards, or shot over emails between a few people. It's
been discussed for a while that we want to create a tiny base runtime
with bst. The upshot is this will homogenize the metadata used in the
build process for Flatpak, GNOME Continuous and eventually many other
projects too, we hope.

Once we have something solid that works, we want to get it used by as
many people as possible, so we'll need to do some publicity stuff. I
have CC'd ***@lists.freedesktop.org and
***@lists.freedesktop.org on this mail to try to raise awareness of what
we're planning to do and involve them in the discussion.

We've set up on gitlab.com, for now [1]. There you can see the
conversion script from JSON to bst [2] and [3], which is part of our
first milestone.

So below is the summary of things discussed recently. All feedback /
additions / corrections are welcome.

## Project Precis

A strap-line we came up with...

'This project maintains a compatible set of git repos and build
instructions to provide other projects with a neutral stable base for
operating system development'.

## Objectives

* Create and maintain a minimal base runtime using BuildStream definitions.
* Establish neutrally located infrastructure to host the base runtime.
* Implement an effective strategy for security updates to the runtime.
* Ensure that the base runtime works with the relevant tooling in
Flatpak and GNOME Continuous.
* Replace flatpak builder with BuildStream to generate flatpak SDKs.
* Replace Freedesktop Base Runtimes (based on YOCTO) with the base runtime.
* Lastly, replace the GNOME Continuous (based on YOCTO) with the base
runtime.

## Roadmap

1) Make Buildtream-based SDK a dropin-replacement of the current one
2) Replace BaseSDK completely with BuildStream's Freedesktop SDK
3) Migrate to Freedesktop Infra
4) Convert GNOME Continuous to use the base runtime

See milestones on gitlab for further details and tasks under each
milestone [4].

Some points on the above:
* We're on gitlab.com for now as an interim solution before we migrate
to freedesktop infra, or freedesktop migrates to gitlab.
* Hardware will required at some point for the infrastructure.
* All infrastructure we establish must be repeatable and trivial to set
up on another server.
* Architectures to aim for: armv7 (hard float only), aarch64, x86, and
x86_64, all little endian.
* We'll aim first for the intel arches and hope for community help on
the ARM side of things.

## Licence

X / MIT

No CLA to sign: All contributors hold their own copyright.

### List of uploaders

We'll choose 4 or 5 people to begin with (we'd discussed said Javier,
Alex, Rob, Adam, Tristan). This list should be kept small: always less
than 10. List people who can merge in git. Eventually set-up a post-job
to update gitlab config automatically.

## Policy to become an uploader

An invite is required from someone already on the list of uploaders and
then approval from a second person already on the list.

## Policy for maintenance of the list of the uploaders

Any changes to the list of uploaders requires review and approval by a
second person who must also be from from the uploader list and is not
the original submitter.

## Allowing merges and review process

For non-trivial changes, review is always encouraged. People with merge
permissions can lose access due to inactivity, if the list of uploaders
choose so. Access can be re-granted if required as described above.

## Release

We've discussed a rolling release: the aim is to continuously update the
runtime while keeping the ABI backward compatible for as long as
reasonably possible.

For the base runtime we expect multiple years of ABI stability to be
feasible (parallel installation of multiple versions may occasionally
be necessary for a few libraries). When an ABI break is required
(e.g., C++ ABI break), the plan is to keep the previous branch
maintained (at least for security updates) for two years.

This should result in a rather small maintenance effort (1-2 branches at
a time).

## Code of Conduct

We've discussed a simple but effective code of conduct, along the lines
of: use common sense: don't abuse others and don't misbehave. When
anyone does, folk should tell the mergers, who will be generally annoyed
at the miscreants, and may take actions.

OK, so I think this covers all the initial important points, if I've
missed anything then please feel free to add that.

Thanks,
Laurence


[1] https://gitlab.com/freedesktop-sdk/freedesktop-sdk
[2] https://gitlab.com/freedesktop-sdk/freedesktop-sdk/tree/test-conversions
[3] https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/1
[4] https://gitlab.com/freedesktop-sdk/freedesktop-sdk/milestones

[Thiago Macieira]

Post by Laurence Urhegyi
So i wanted to kick off this mailing list by trying to round up a few
things which have been discussed recently, at GUADEC, in f2f
conversations afterwards, or shot over emails between a few people. It's
been discussed for a while that we want to create a tiny base runtime
with bst. The upshot is this will homogenize the metadata used in the
build process for Flatpak, GNOME Continuous and eventually many other
projects too, we hope.

Can you give some background on what the problem is in the first place? What is
a tiny base runtime? What is bst? Why was it chosen? Your text talks about
replacing Yocto (a well-known, well-established project with 10 years of
history) with bst, so you need to justify. Or I misunderstood you.

If there have been previous discussions on this subject on other mailing
lists, can you point them out?

I think I can guess why you want this in freedesktop.org, but can you write it
out please? The bug report[1] asking for the mailing list creation did not
include the justification.

[1] https://bugs.freedesktop.org/show_bug.cgi?id=103254

[Laurence Urhegyi]

Hi Thiago,

<snip>

Post by Thiago Macieira
Can you give some background on what the problem is in the first place? What is
a tiny base runtime? What is bst? Why was it chosen? Your text talks about
replacing Yocto (a well-known, well-established project with 10 years of
history) with bst, so you need to justify. Or I misunderstood you.
If there have been previous discussions on this subject on other mailing
lists, can you point them out?

I believe that Emmet and Tristan between them have addressed all of your

Post by Thiago Macieira
I think I can guess why you want this in freedesktop.org, but can you write it
out please? The bug report[1] asking for the mailing list creation did not
include the justification.

To summarise, then:

* Freedesktop.org is a neutral location, not tied to one community and
we would like the base runtime to remain distribution / desktop neutral
and accessible to anyone or any project potentially interested.
* One specific thing we intend to amend is the current freedesktop
flatpak sdk, so freedesktop seems to be the right place to host this effort.

Let me know if I need to add this justification somewhere else as well
as just these mailing lists.

Laurence

[Thiago Macieira]

Post by Laurence Urhegyi
I believe that Emmet and Tristan between them have addressed all of your

Where have they addressed the questions? I have not received any email from
them and they don't appear to have posted to xdg.

[Laurence Urhegyi]

Thiago, see Emmet's earlier response below.

I'll also send Tristan's now.

Post by Thiago Macieira

Post by Laurence Urhegyi
So i wanted to kick off this mailing list by trying to round up a few
things which have been discussed recently, at GUADEC, in f2f
conversations afterwards, or shot over emails between a few people. It's
been discussed for a while that we want to create a tiny base runtime
with bst. The upshot is this will homogenize the metadata used in the
build process for Flatpak, GNOME Continuous and eventually many other
projects too, we hope.

Can you give some background on what the problem is in the first place? What is
a tiny base runtime? What is bst? Why was it chosen? Your text talks about
replacing Yocto (a well-known, well-established project with 10 years of
history) with bst, so you need to justify. Or I misunderstood you.

    My memory of the justification was the result of raising the idea
of using BuildStream (1) to build Flatpaks (2) for FlatHub (3) in a
session at the GUADEC 2017 Unconference (4).  As consensus was
approached to consider BuildStream for this application, two possible
migration blockers were raised, being A) whether BuildStream had been
ported to run on a Yocto runtime, and B) whether the current maintainer
of the Yocto runtime being used for FlatHub Flatpaks wanted to keep
maintaining the Yocto runtime, or update it to include BuildStream
dependencies.
    For A), there were no volunteers to do the porting, although some
folk talked about some of the work they had done for other projects to
bootstrap systems using LFS (5) techniques with BuildStream.  Given B),
discussion quickly moved to whether using a distribution runtime would
be suitable: this was fairly quickly determined to be unsustainable, as
FlatHub has an explicit policy of distribution neutrality.  At this
point, there was a volunteer who offered to construct BuildStream
definitions for the base runtime, which offer was generally accepted.
        My understanding is that this project is not intended to
replace Yocto for general application, as Yocto has extensive support
tooling to help with many use cases that neither BuildStream nor any
bootstrap with BuildStream definitions would cover, but only in the
specific cases of building a base for FlatHub Flatpaks and support for
GNOME continuous (which has been unable to publish an image for some
months, for various reasons, not least a lack of folk actively
involved).  While both of these are primarily GNOME-facing uses, I
imagine that a low-disk-space low-memory base runtime may also be useful
for automated testing and validation of other desktop environments or
construction of small chroots for various automation support
applications, and so support the idea of hosting with Freedesktop,
rather than with GNOME or somewhere else.  I also understand there was a
plan to have a companion project within GNOME, that would handle
BuildStream definitions for GNOME-specific common dependencies, to
ensure that the mooted Freedesktop project would remain desktop neutral,
although I’ve heard nothing about this since GUADEC, so it may remain
theoretical, or be dependent on this project.
    For clarity, my memory of the discussions at GUADEC include no
clear strategy for wholesale replacement of Yocto, but niche replacement
for certain use cases, and in a way that may be useful for similar niche
use cases, where a rolling-release model and focus on minimal tooling to
build desktop applications is the primary focus, rather than end-user
polish, robust platform support, rich developer tooling, or similar.

Post by Thiago Macieira
If there have been previous discussions on this subject on other mailing
lists, can you point them out?

    I have not seen any.  I remember the discussion at GUADEC, and have
seen some IRC chatter since then, but these are the first real posts
I’ve seen on the topic.
1: https://buildstream.gitlab.io/buildstream/
2: http://flatpak.org/
3: https://flathub.org/
4: https://wiki.gnome.org/GUADEC/2017/Unconference/FlatpakBOF
5: http://www.linuxfromscratch.org/
—
Emmet HIKORY
_______________________________________________
Freedesktop-sdk mailing list
https://lists.freedesktop.org/mailman/listinfo/freedesktop-sdk

[Daniel Stone]

Hi Laurence,

On 27 October 2017 at 15:45, Laurence Urhegyi

So i wanted to kick off this mailing list by trying to round up a few things
which have been discussed recently, at GUADEC, in f2f conversations
afterwards, or shot over emails between a few people. It's been discussed
for a while that we want to create a tiny base runtime with bst. The upshot
is this will homogenize the metadata used in the build process for Flatpak,
GNOME Continuous and eventually many other projects too, we hope.

Thiago's questions would be good to have an elaborated answer on. In
particular, whether BuildStream - which seems like it's only a few
months old - is a core part of the 'API' for the SDK, or whether it's
just incidentally used as part of the build process. I hadn't heard of
it until now, and would be curious to see if it's really used at all
outside of GNOME and Codethink.

## Objectives
* Create and maintain a minimal base runtime using BuildStream definitions.

Same question as above: is the goal of the SDK to use BuildStream
definitions, or is BuildStream just something which happens to be a
part of it?

* Establish neutrally located infrastructure to host the base runtime.
* Implement an effective strategy for security updates to the runtime.
* Ensure that the base runtime works with the relevant tooling in Flatpak
and GNOME Continuous.
* Replace flatpak builder with BuildStream to generate flatpak SDKs.
* Replace Freedesktop Base Runtimes (based on YOCTO) with the base runtime.
* Lastly, replace the GNOME Continuous (based on YOCTO) with the base
runtime.

I'd ideally suggest establishing the goals of the SDK in terms of what
it's meant to provide to external users. That's something which is
actually compelling and useful: fd.o doesn't usually get involved in
projects whose sole goal is to _use_ a new build system.

* We're on gitlab.com for now as an interim solution before we migrate to
freedesktop infra, or freedesktop migrates to gitlab.

For the record, I would very much like fd.o to migrate to GitLab.

* Hardware will required at some point for the infrastructure.

What kind of hardware? Is that something you'd expect from fd.o, or
would you reuse, e.g. flathub and GNOME Continuous? Do you need
hosting, or build machines, or test runners, or ... ?

## Licence
X / MIT
No CLA to sign: All contributors hold their own copyright.

Is this just for the SDK - which I understand at this point to be a
bunch of build recipes and a conversion script from JSON/Yocto to
YAML/BuildStream - or are you talking about BuildStream itself?

## Code of Conduct
use common sense: don't abuse others and don't misbehave. When anyone does,
folk should tell the mergers, who will be generally annoyed at the
miscreants, and may take actions.

fd.o already has a Code of Conduct here, which we expect all new
projects to abide by:
https://www.freedesktop.org/wiki/CodeOfConduct/

If you have any specific input or suggestions, these would be welcome.

Cheers,
Daniel

[Laurence Urhegyi]

Thiago,

Seems the below mail from Tristan also didn't reach the xdg list.

Hi,
I can provide some background.

Post by Daniel Stone
Hi Laurence,
On 27 October 2017 at 15:45, Laurence Urhegyi

So i wanted to kick off this mailing list by trying to round up a few things
which have been discussed recently, at GUADEC, in f2f conversations
afterwards, or shot over emails between a few people. It's been discussed
for a while that we want to create a tiny base runtime with bst. The upshot
is this will homogenize the metadata used in the build process for Flatpak,
GNOME Continuous and eventually many other projects too, we hope.

Thiago's questions would be good to have an elaborated answer on. In
particular, whether BuildStream - which seems like it's only a few
months old - is a core part of the 'API' for the SDK, or whether it's
just incidentally used as part of the build process. I hadn't heard of
it until now, and would be curious to see if it's really used at all
outside of GNOME and Codethink.

Yes BuildStream is new, it's one year in the making and close to our
1.0 first stable release.
Before starting, we discussed this with the GNOME release team and the
Flatpak maintainer, Alex - one of our main goals when starting the
BuildStream project was to unify the build metadata used to deliver
GNOME releases, run GNOME continuous integration and deliver Flatpak
SDKs and Flatpaks.
Consider that for GNOME, we currently had 3 formats which the release
o JHBuild XML
o flatpak-builder JSON
o GNOME continuous JSON
All of which represent... basically different ways to build the same
things.
Now we have the ball rolling and will start by migrating the existing
JHBuild releases to use buildstream, consensus on that is pretty much
reached in this thread[0], aside from talks we had face to face at
GUADEC.
Next, we will be delivering the GNOME Flatpak SDK with BuildStream,
from the *same* build data maintained by the GNOME release team.
At the same time, we will be migrating builds of the
org.freedesktop.Sdk and org.freedesktop.BaseSdk Flatpak runtimes to use
BuildStream build metadata and tooling - overall we will be maintaining
less tooling, and having multiple BuildStream projects which logically
depend on eachother helps to leverage the feature set (shared artifact
caches, avoid export/import dances, etc).
This will include a tiny runtime for the base SDK, and something a bit
bigger to match the Flatpak org.freedesktop.Sdk runtime.
In order to tie up the loop, it will make sense to have a minimal
kernel build so that we can create bootable images - this will help us
tie up the loop and allow GNOME developers to boot VMs with full GNOME
builds and test the nitty gritty stuff like GDM and gnome-session.
So the bottom line here, is that we have a shared runtime to build with
BuildStream, and neither the GNOME release team mailing list, or
Flatpak mailing lists are suitable venues for discussion and planning
of this work - since *both* projects have a common interest in making
this happen.
Of course, we hope that this base runtime and potentially bootable
minimal system layer can be of use to other projects besides the GNOME
and Flatpak projects, so we will welcome other participants to
contribute and be flexible - so long as the topic remains about
building approximately the same stack using BuildStream (for instance,
some parties might be interested in generating Docker images as output
or other kinds of deployments of pretty much the same builds).

Post by Daniel Stone

## Objectives
* Create and maintain a minimal base runtime using BuildStream definitions.

Same question as above: is the goal of the SDK to use BuildStream
definitions, or is BuildStream just something which happens to be a
part of it?

The goal is to have a common base built with BuildStream, and the
project itself will consist of BuildStream YAML.
There are other ways of building parts of stacks and deploying those,
however tying them together to the same tooling is certainly the big
picture here - so that we do have a unified picture of everything we
build, and so they can all leverage the same features (e.g. projects
which depend on other projects, and sharing of artifacts/build output
using the same artifact sharing mechanisms).
Development of this is underway, we just need a proper venue to discuss
it. Since this particular project should remain desktop/distro
agnostic, and since our first goals are to build the freedesktop
flatpak SDKs with BuildStream, freedesktop.org seems to be the logical
venue for this.

Post by Daniel Stone

* Establish neutrally located infrastructure to host the base runtime.
* Implement an effective strategy for security updates to the runtime.
* Ensure that the base runtime works with the relevant tooling in Flatpak
and GNOME Continuous.
* Replace flatpak builder with BuildStream to generate flatpak SDKs.
* Replace Freedesktop Base Runtimes (based on YOCTO) with the base runtime.
* Lastly, replace the GNOME Continuous (based on YOCTO) with the base
runtime.

I'd ideally suggest establishing the goals of the SDK in terms of what
it's meant to provide to external users. That's something which is
actually compelling and useful: fd.o doesn't usually get involved in
projects whose sole goal is to _use_ a new build system.

I dont think that continuing to build GNOME Continuous images and base
Flatpak runtimes with Yocto is an option.
It makes no sense to use many different tools which produce
approximately same things, we're just tripling our maintenance burden
for no reason with that thinking.
I don't mind really how people want to phrase this, but if it sounds
"The purpose of this project is rather to provide build metadata for
building a commonly maintained base runtime for both GNOME and
Flatpak projects.
Others are welcome to join and consume the same metadata and add to
this, so long as there is not too much scope creep in terms of what
we build."

Post by Daniel Stone

* We're on gitlab.com for now as an interim solution before we migrate to
freedesktop infra, or freedesktop migrates to gitlab.

For the record, I would very much like fd.o to migrate to GitLab.

* Hardware will required at some point for the infrastructure.

What kind of hardware? Is that something you'd expect from fd.o, or
would you reuse, e.g. flathub and GNOME Continuous? Do you need
hosting, or build machines, or test runners, or ... ?

I'll try not to get too deep on this one, my assumption is that
Laurence is referring to resources needed to run gitlab and runners to
run CI, or at least continuous builds, on a variety of supported
hardware - For this, I dont want to make an official statement but I'm
quite certain that we can reuse existing hardware (we currently build
the org.freedesktop.Sdk and friends on some arm machines donated to the
GNOME project by Codethink[1], and I believe the Intel machine(s) used
build the same are a Red Hat donation).
Asides from this, I'll leave it to others to reply, I suppose a simple
static website could be nice (but gitlab's "pages" gives us this if we
have fd.o migrate to gitlab anyway).

Post by Daniel Stone

## Licence
X / MIT
No CLA to sign: All contributors hold their own copyright.

Is this just for the SDK - which I understand at this point to be a
bunch of build recipes and a conversion script from JSON/Yocto to
YAML/BuildStream - or are you talking about BuildStream itself?

This point is about the actual YAML build metadata.
I.e. we would hope to avoid copyright / legal notices all over the
place and instead have a single, ultra-permissive license, we think the
X / MIT license is good for this mostly because it's popular and well
understood.
Hope I've cleared some things up here.
Best Regards,
-Tristan
[0]: https://mail.gnome.org/archives/release-team/2017-October/msg00018.html
[1]: https://blogs.gnome.org/tvb/2016/05/23/endless-and-codethink-team-up-for-gnome-on-arm/

[Laurence Urhegyi]

Hi Daniel,

On 30/10/17 10:49, Daniel Stone wrote:
<snip>

Post by Daniel Stone
Thiago's questions would be good to have an elaborated answer on.

Let me know if Emmet and Tristan's respective explanations leave
anything uncovered.

<snip>

Post by Daniel Stone
I'd ideally suggest establishing the goals of the SDK in terms of what
it's meant to provide to external users.

Good point. We should emphasize the benefits to users better than I
have. Tristan's point is pertinent here: the benefit is the reduced
overhead from maintaining only one set of metadata.

<snip>

Post by Daniel Stone
What kind of hardware? Is that something you'd expect from fd.o, or
would you reuse, e.g. flathub and GNOME Continuous? Do you need
hosting, or build machines, or test runners, or ... ?

Setting up a runner for each architecture will be required. I expect
that we can re-use what's already in use, yes, rather than needing to
acquire anything new. Therefore we're not expecting anything from fd.o
to this end.

<snip>

Post by Daniel Stone
fd.o already has a Code of Conduct here, which we expect all new
https://www.freedesktop.org/wiki/CodeOfConduct/

Thanks. These notes are just capturing discussions had: we would of
course stick to that Code of Conduct.

Post by Daniel Stone
If you have any specific input or suggestions, these would be welcome.

Thanks,
Laurence