settingsLogin | Registersettings

[openstack-dev] [puppet][tripleo][release] puppet module versioning

0 votes

Ahoy folks,

I would like to bring attention to deriving version numbers for puppet
modules (for packaging) as part of the release process. Today we
uncovered an issue[0] in the way that version numbers were being
derived for packages which ultimately broke the TripleO CI upgrade
jobs because the master version was older than the last released Ocata
version.

As for a bit of history around the puppet version numbers, PEP-440 is
not supported in puppet and they are using a form of SemVer[1] . So
we've historically chosen to just use X.Y.Z for our version numbers
and we usually end up bumping them prior to cutting a new release.
That being said, for the first time we've run into the issue where the
master branches metadata.json contained a version less than the
stable/ocata branch which resulted in the upgrade jobs failing.

In this case, we could have avoided the version issue by pushing a
version bump to master after cutting the stable/ocata branch to ensure
that master would be the next version. We've talked about improving
the automation around the puppet versions in the past and I'm
wondering if this is something that would be best to be done in the
release process. Now because puppet supports SemVer, we could
designate the next version as a X.Y.Z-dev version in the metadata.json
and would expect packagers to use the metadata.json file as the source
of truth for deriving version information and not using something like
pbr or git describe. Right now as it sits for puppet modules, we
generally aren't touching the version information until we're ready to
release the next version which doesn't seem right for knowing what the
current version of the code actually is.

So what I'm proposing is to use a "-dev" pre-release identifier
between version releases for puppet modules. As part of the tagging
process we could propose the next release version with "-dev" to the
repository. The flow would look something like...

MASTER
1.0.0-dev
+
| TAG: M1
+---> 1.0.0
|
v
MASTER
1.1.0-dev
+
| TAG: M2
+---> 1.1.0
|
v
MASTER
1.2.0-dev
+
| TAG: M3
+---> 1.2.0
|
v
MASTER
1.3.0-dev
+
| TAG: RC1
+---> 1.3.0
|
v
RELEASE
+
| STABLE BRANCH
+---> 1.4.0-dev
|
v
MASTER
2.0.0-dev

Currently since the metadata.json file is in the repository and for
the release process we're providing a git hash for the version to tag,
I'm not sure how the dropping of the "-dev" would work. Since we
basically want the release process to take the hash, mangle the
metadata.json, commit it and use this new hash as the point to tag.
thoughts?

Thanks,
-Alex

[0] https://bugs.launchpad.net/tripleo/+bug/1669462
[1] https://github.com/puppetlabs/puppet/blob/master/lib/semver.rb#L10


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 9, 2017 in openstack-dev by aschultz_at_redhat.c (5,800 points)   2 2 4

5 Responses

0 votes

Excerpts from Alex Schultz's message of 2017-03-07 12:56:17 -0700:

Ahoy folks,

I would like to bring attention to deriving version numbers for puppet
modules (for packaging) as part of the release process. Today we
uncovered an issue[0] in the way that version numbers were being
derived for packages which ultimately broke the TripleO CI upgrade
jobs because the master version was older than the last released Ocata
version.

As for a bit of history around the puppet version numbers, PEP-440 is
not supported in puppet and they are using a form of SemVer[1] . So
we've historically chosen to just use X.Y.Z for our version numbers
and we usually end up bumping them prior to cutting a new release.
That being said, for the first time we've run into the issue where the
master branches metadata.json contained a version less than the
stable/ocata branch which resulted in the upgrade jobs failing.

In this case, we could have avoided the version issue by pushing a
version bump to master after cutting the stable/ocata branch to ensure
that master would be the next version. We've talked about improving
the automation around the puppet versions in the past and I'm
wondering if this is something that would be best to be done in the
release process. Now because puppet supports SemVer, we could
designate the next version as a X.Y.Z-dev version in the metadata.json
and would expect packagers to use the metadata.json file as the source
of truth for deriving version information and not using something like
pbr or git describe. Right now as it sits for puppet modules, we
generally aren't touching the version information until we're ready to
release the next version which doesn't seem right for knowing what the
current version of the code actually is.

So what I'm proposing is to use a "-dev" pre-release identifier
between version releases for puppet modules. As part of the tagging
process we could propose the next release version with "-dev" to the
repository. The flow would look something like...

MASTER
1.0.0-dev
+
| TAG: M1
+---> 1.0.0
|
v
MASTER
1.1.0-dev
+
| TAG: M2
+---> 1.1.0
|
v
MASTER
1.2.0-dev
+
| TAG: M3
+---> 1.2.0
|
v
MASTER
1.3.0-dev
+
| TAG: RC1
+---> 1.3.0
|
v
RELEASE
+
| STABLE BRANCH
+---> 1.4.0-dev
|
v
MASTER
2.0.0-dev

Currently since the metadata.json file is in the repository and for
the release process we're providing a git hash for the version to tag,
I'm not sure how the dropping of the "-dev" would work. Since we
basically want the release process to take the hash, mangle the
metadata.json, commit it and use this new hash as the point to tag.
thoughts?

The release scripts don't have permission to merge a commit, so it would
work better if that step was done manually and then the resulting SHA
can be used in the tag request.

FWIW, we used to do something very similar to this for Python projects,
and ended up dropping it because it causes extra synchronization pains
around releases. A big part of scaling up release support was making
them easier by dropping any references to version numbers in files that
have to be checked in. I'm guessing there's no way to do that for
puppet?

Doug

Thanks,
-Alex

[0] https://bugs.launchpad.net/tripleo/+bug/1669462
[1] https://github.com/puppetlabs/puppet/blob/master/lib/semver.rb#L10


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 9, 2017 by Doug_Hellmann (87,520 points)   3 4 12
0 votes

On Thu, Mar 9, 2017 at 10:54 AM, Doug Hellmann doug@doughellmann.com wrote:
Excerpts from Alex Schultz's message of 2017-03-07 12:56:17 -0700:

Ahoy folks,

I would like to bring attention to deriving version numbers for puppet
modules (for packaging) as part of the release process. Today we
uncovered an issue[0] in the way that version numbers were being
derived for packages which ultimately broke the TripleO CI upgrade
jobs because the master version was older than the last released Ocata
version.

As for a bit of history around the puppet version numbers, PEP-440 is
not supported in puppet and they are using a form of SemVer[1] . So
we've historically chosen to just use X.Y.Z for our version numbers
and we usually end up bumping them prior to cutting a new release.
That being said, for the first time we've run into the issue where the
master branches metadata.json contained a version less than the
stable/ocata branch which resulted in the upgrade jobs failing.

In this case, we could have avoided the version issue by pushing a
version bump to master after cutting the stable/ocata branch to ensure
that master would be the next version. We've talked about improving
the automation around the puppet versions in the past and I'm
wondering if this is something that would be best to be done in the
release process. Now because puppet supports SemVer, we could
designate the next version as a X.Y.Z-dev version in the metadata.json
and would expect packagers to use the metadata.json file as the source
of truth for deriving version information and not using something like
pbr or git describe. Right now as it sits for puppet modules, we
generally aren't touching the version information until we're ready to
release the next version which doesn't seem right for knowing what the
current version of the code actually is.

So what I'm proposing is to use a "-dev" pre-release identifier
between version releases for puppet modules. As part of the tagging
process we could propose the next release version with "-dev" to the
repository. The flow would look something like...

MASTER
1.0.0-dev
+
| TAG: M1
+---> 1.0.0
|
v
MASTER
1.1.0-dev
+
| TAG: M2
+---> 1.1.0
|
v
MASTER
1.2.0-dev
+
| TAG: M3
+---> 1.2.0
|
v
MASTER
1.3.0-dev
+
| TAG: RC1
+---> 1.3.0
|
v
RELEASE
+
| STABLE BRANCH
+---> 1.4.0-dev
|
v
MASTER
2.0.0-dev

Currently since the metadata.json file is in the repository and for
the release process we're providing a git hash for the version to tag,
I'm not sure how the dropping of the "-dev" would work. Since we
basically want the release process to take the hash, mangle the
metadata.json, commit it and use this new hash as the point to tag.
thoughts?

The release scripts don't have permission to merge a commit, so it would
work better if that step was done manually and then the resulting SHA
can be used in the tag request.

Ok then I guess as part of release prep we can just handle the
dropping of the -dev. Would it be possible for the release process to
at least propose the next '-dev' version via the release tools?

FWIW, we used to do something very similar to this for Python projects,
and ended up dropping it because it causes extra synchronization pains
around releases. A big part of scaling up release support was making
them easier by dropping any references to version numbers in files that
have to be checked in. I'm guessing there's no way to do that for
puppet?

No because the metadata.json is the source of truth for the version
information for the puppet modules so it either has to either exist or
be generated and committed so that when people checkout from git
puppet knows what versions are there. Technically if you're only
doing source deployments you could manage the versions yourself and
just ignore the warnings from puppet but it also makes it hard to know
what your environment has version why. 'puppet module list' pulls the
version information out of the metadata.json. If we automate (and
document) some of this workflow it's not so bad as we've been handling
all this version mangling already.

for module in $list; do
pushd $module; sed -ie 's/-dev//' metadata.json; git add
metadata.json; git commit -m "release prep"; git review; popd
done

Thanks,
-Alex

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 9, 2017 by aschultz_at_redhat.c (5,800 points)   2 2 4
0 votes

Excerpts from Alex Schultz's message of 2017-03-09 11:29:28 -0700:

On Thu, Mar 9, 2017 at 10:54 AM, Doug Hellmann doug@doughellmann.com wrote:

Excerpts from Alex Schultz's message of 2017-03-07 12:56:17 -0700:

Ahoy folks,

I would like to bring attention to deriving version numbers for puppet
modules (for packaging) as part of the release process. Today we
uncovered an issue[0] in the way that version numbers were being
derived for packages which ultimately broke the TripleO CI upgrade
jobs because the master version was older than the last released Ocata
version.

As for a bit of history around the puppet version numbers, PEP-440 is
not supported in puppet and they are using a form of SemVer[1] . So
we've historically chosen to just use X.Y.Z for our version numbers
and we usually end up bumping them prior to cutting a new release.
That being said, for the first time we've run into the issue where the
master branches metadata.json contained a version less than the
stable/ocata branch which resulted in the upgrade jobs failing.

In this case, we could have avoided the version issue by pushing a
version bump to master after cutting the stable/ocata branch to ensure
that master would be the next version. We've talked about improving
the automation around the puppet versions in the past and I'm
wondering if this is something that would be best to be done in the
release process. Now because puppet supports SemVer, we could
designate the next version as a X.Y.Z-dev version in the metadata.json
and would expect packagers to use the metadata.json file as the source
of truth for deriving version information and not using something like
pbr or git describe. Right now as it sits for puppet modules, we
generally aren't touching the version information until we're ready to
release the next version which doesn't seem right for knowing what the
current version of the code actually is.

So what I'm proposing is to use a "-dev" pre-release identifier
between version releases for puppet modules. As part of the tagging
process we could propose the next release version with "-dev" to the
repository. The flow would look something like...

MASTER
1.0.0-dev
+
| TAG: M1
+---> 1.0.0
|
v
MASTER
1.1.0-dev
+
| TAG: M2
+---> 1.1.0
|
v
MASTER
1.2.0-dev
+
| TAG: M3
+---> 1.2.0
|
v
MASTER
1.3.0-dev
+
| TAG: RC1
+---> 1.3.0
|
v
RELEASE
+
| STABLE BRANCH
+---> 1.4.0-dev
|
v
MASTER
2.0.0-dev

Currently since the metadata.json file is in the repository and for
the release process we're providing a git hash for the version to tag,
I'm not sure how the dropping of the "-dev" would work. Since we
basically want the release process to take the hash, mangle the
metadata.json, commit it and use this new hash as the point to tag.
thoughts?

The release scripts don't have permission to merge a commit, so it would
work better if that step was done manually and then the resulting SHA
can be used in the tag request.

Ok then I guess as part of release prep we can just handle the
dropping of the -dev. Would it be possible for the release process to
at least propose the next '-dev' version via the release tools?

That seems like a reasonable thing to add. The natural way to do
it is to add a new job to the tag pipeline for the repositories.
You'll likely want to copy some of the configuration from the
tag-releases job, because that runs on a node with permission to
propose new patches.

FWIW, we used to do something very similar to this for Python projects,
and ended up dropping it because it causes extra synchronization pains
around releases. A big part of scaling up release support was making
them easier by dropping any references to version numbers in files that
have to be checked in. I'm guessing there's no way to do that for
puppet?

No because the metadata.json is the source of truth for the version
information for the puppet modules so it either has to either exist or
be generated and committed so that when people checkout from git
puppet knows what versions are there. Technically if you're only

Yes, that's what I was afraid of.

doing source deployments you could manage the versions yourself and
just ignore the warnings from puppet but it also makes it hard to know
what your environment has version why. 'puppet module list' pulls the
version information out of the metadata.json. If we automate (and
document) some of this workflow it's not so bad as we've been handling
all this version mangling already.

for module in $list; do
pushd $module; sed -ie 's/-dev//' metadata.json; git add
metadata.json; git commit -m "release prep"; git review; popd
done

Thanks,
-Alex

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 9, 2017 by Doug_Hellmann (87,520 points)   3 4 12
0 votes

Doug Hellmann wrote:
Excerpts from Alex Schultz's message of 2017-03-09 11:29:28 -0700:

On Thu, Mar 9, 2017 at 10:54 AM, Doug Hellmann doug@doughellmann.com wrote:

Excerpts from Alex Schultz's message of 2017-03-07 12:56:17 -0700:

So what I'm proposing is to use a "-dev" pre-release identifier
between version releases for puppet modules. As part of the tagging
process we could propose the next release version with "-dev" to the
repository. The flow would look something like...
[...]

As an aside, we used to do that for pre-versioned deliverables (drop the
"2015.1.0" as the "next" version in setup.cfg as the very first commit
in master after a branch point). The main issue with it was that it
required a lock on master to make it work (avoid other commits to make
it into master before the version bump).

--
Thierry Carrez (ttx)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 13, 2017 by Thierry_Carrez (57,480 points)   3 8 13
0 votes

Excerpts from Thierry Carrez's message of 2017-03-13 15:19:40 +0100:

Doug Hellmann wrote:

Excerpts from Alex Schultz's message of 2017-03-09 11:29:28 -0700:

On Thu, Mar 9, 2017 at 10:54 AM, Doug Hellmann doug@doughellmann.com wrote:

Excerpts from Alex Schultz's message of 2017-03-07 12:56:17 -0700:

So what I'm proposing is to use a "-dev" pre-release identifier
between version releases for puppet modules. As part of the tagging
process we could propose the next release version with "-dev" to the
repository. The flow would look something like...
[...]

As an aside, we used to do that for pre-versioned deliverables (drop the
"2015.1.0" as the "next" version in setup.cfg as the very first commit
in master after a branch point). The main issue with it was that it
required a lock on master to make it work (avoid other commits to make
it into master before the version bump).

It's not perfect, but since we can't make puppet take its version
from the git tag (like we could with pbr), and the jobs would apply
only to the puppet repositories, it seems like a reasonable approach.

Doug


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 13, 2017 by Doug_Hellmann (87,520 points)   3 4 12
...