settingsLogin | Registersettings

[openstack-dev] [devstack] etcd v3.2.0?

0 votes

Hi All,
I just push a review [1] to bump the minimum etcd version to
3.2.0 which works on intel and ppc64le. I know we're pretty late in the
cycle to be making changes like this but releasing pike with a dependacy
on 3.1.x make it harder for users on ppc64le (not many but a few :D)

Yours Tony.

[1] https://review.openstack.org/474825


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

asked Sep 25, 2017 in openstack-dev by Tony_Breeds (19,660 points)   3 6 10

9 Responses

0 votes

On 06/15/2017 10:06 PM, Tony Breeds wrote:
Hi All,
I just push a review [1] to bump the minimum etcd version to
3.2.0 which works on intel and ppc64le. I know we're pretty late in the
cycle to be making changes like this but releasing pike with a dependacy
on 3.1.x make it harder for users on ppc64le (not many but a few :D)

Yours Tony.

[1] https://review.openstack.org/474825

It should be fine, no one is really using these much at this point.
However it looks like mirroring is not happening automatically? The
patch fails on not existing in the infra mirror.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 16, 2017 by Sean_Dague (66,200 points)   4 8 14
0 votes

On Fri, Jun 16, 2017 at 6:01 AM, Sean Dague sean@dague.net wrote:
On 06/15/2017 10:06 PM, Tony Breeds wrote:

Hi All,
I just push a review [1] to bump the minimum etcd version to
3.2.0 which works on intel and ppc64le. I know we're pretty late in the
cycle to be making changes like this but releasing pike with a dependacy
on 3.1.x make it harder for users on ppc64le (not many but a few :D)

Yours Tony.

[1] https://review.openstack.org/474825

It should be fine, no one is really using these much at this point.
However it looks like mirroring is not happening automatically? The
patch fails on not existing in the infra mirror.

    -Sean

It appears so. Also, IIRC, infra mirror would only host x86 binaries.
Right now PowerKVM CI works by patching devstack-gate to override
infra etcd download url. The fix [2] still needs to get merged to make
it a bit easier to use d-g with your own etcd mirror.

[2] https://review.openstack.org/#/c/467437/


Mikhail Medvedev
IBM OpenStack CI for KVM on Power


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 16, 2017 by Mikhail_Medvedev (1,180 points)   1
0 votes

Mikhail,

I have a TODO on my list - " adding a job that looks for new releases
and uploads them to tarballs periodically "

Thanks,
-- Dims

On Fri, Jun 16, 2017 at 3:32 PM, Mikhail Medvedev mihailmed@gmail.com wrote:
On Fri, Jun 16, 2017 at 6:01 AM, Sean Dague sean@dague.net wrote:

On 06/15/2017 10:06 PM, Tony Breeds wrote:

Hi All,
I just push a review [1] to bump the minimum etcd version to
3.2.0 which works on intel and ppc64le. I know we're pretty late in the
cycle to be making changes like this but releasing pike with a dependacy
on 3.1.x make it harder for users on ppc64le (not many but a few :D)

Yours Tony.

[1] https://review.openstack.org/474825

It should be fine, no one is really using these much at this point.
However it looks like mirroring is not happening automatically? The
patch fails on not existing in the infra mirror.

    -Sean

It appears so. Also, IIRC, infra mirror would only host x86 binaries.
Right now PowerKVM CI works by patching devstack-gate to override
infra etcd download url. The fix [2] still needs to get merged to make
it a bit easier to use d-g with your own etcd mirror.

[2] https://review.openstack.org/#/c/467437/


Mikhail Medvedev
IBM OpenStack CI for KVM on Power


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Davanum Srinivas :: https://twitter.com/dims


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 16, 2017 by Davanum_Srinivas (35,920 points)   2 4 7
0 votes

On Fri, Jun 16, 2017 at 03:59:22PM -0400, Davanum Srinivas wrote:
Mikhail,

I have a TODO on my list - " adding a job that looks for new releases
and uploads them to tarballs periodically "

If you point me to how things are added to that mirror I can work
towards that.

Tony.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jun 18, 2017 by Tony_Breeds (19,660 points)   3 6 10
0 votes

On Sun, Jun 18, 2017 at 7:36 PM, Tony Breeds tony@bakeyournoodle.com wrote:
On Fri, Jun 16, 2017 at 03:59:22PM -0400, Davanum Srinivas wrote:

Mikhail,

I have a TODO on my list - " adding a job that looks for new releases
and uploads them to tarballs periodically "

If you point me to how things are added to that mirror I can work
towards that.

Awesome! thanks Tony, some kolla jobs do that for example, but i think
this job is a better one to key off of:
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381

Outline of the work is - check if there are any new releases in github
downloads, if so download them using wget and then delegate to the scp
publisher (with keep-hierarchy) to create the new directories and
upload the file(s).

Thanks,
Dims

Tony.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Davanum Srinivas :: https://twitter.com/dims


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 19, 2017 by Davanum_Srinivas (35,920 points)   2 4 7
0 votes

On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:

Awesome! thanks Tony, some kolla jobs do that for example, but i think
this job is a better one to key off of:
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381

Outline of the work is - check if there are any new releases in github
downloads, if so download them using wget and then delegate to the scp
publisher (with keep-hierarchy) to create the new directories and
upload the file(s).

So perhaps I'm dense but I can't see an easy way to get a list of
release artefacts from github in a form that wget can consume. The best
I can see is via the API. I've knocked up a quick'n'dirty mirror
script[1] but I really feel like I've gone off into the weeds.

You basically need to do:

git clone && cd
virtualenv .venv
.venv/bin/pip install -U pip setuptools wheel
.venv/bin/pip install -r ./requirements.txt # [2]
.venv/bin/python ./mirror-github-releases.py \
'coreos/etcd::.linux.gz:etcd' \
'coreos/etcd:6225411:.linux.gz:etcd'

This will in theory from the 3.2.0 (latest) release and look at the
3.1.7 release, see that it's already publically mirrored and move on.

It wouldn't be too hard to incorporate into a job. Thoughts?

Yours Tony.

[1] https://github.com/tbreeds/mirror-github-releases
[2] Yes of course I could publish it on pypi if we want to go down this
path


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jun 19, 2017 by Tony_Breeds (19,660 points)   3 6 10
0 votes

Tony,

On Sun, Jun 18, 2017 at 11:34 PM, Tony Breeds tony@bakeyournoodle.com wrote:
On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:

Awesome! thanks Tony, some kolla jobs do that for example, but i think
this job is a better one to key off of:
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381

Outline of the work is - check if there are any new releases in github
downloads, if so download them using wget and then delegate to the scp
publisher (with keep-hierarchy) to create the new directories and
upload the file(s).

So perhaps I'm dense but I can't see an easy way to get a list of
release artefacts from github in a form that wget can consume. The best
I can see is via the API. I've knocked up a quick'n'dirty mirror
script[1] but I really feel like I've gone off into the weeds.

You basically need to do:

git clone && cd
virtualenv .venv
.venv/bin/pip install -U pip setuptools wheel
.venv/bin/pip install -r ./requirements.txt # [2]
.venv/bin/python ./mirror-github-releases.py \
'coreos/etcd::.linux.gz:etcd' \
'coreos/etcd:6225411:.linux.gz:etcd'

Works for me!

This will in theory from the 3.2.0 (latest) release and look at the
3.1.7 release, see that it's already publically mirrored and move on.

It wouldn't be too hard to incorporate into a job. Thoughts?

Yours Tony.

[1] https://github.com/tbreeds/mirror-github-releases
[2] Yes of course I could publish it on pypi if we want to go down this
path


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Davanum Srinivas :: https://twitter.com/dims


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Jun 19, 2017 by Davanum_Srinivas (35,920 points)   2 4 7
0 votes

On Mon, Jun 19, 2017 at 08:17:53AM -0400, Davanum Srinivas wrote:
Tony,

On Sun, Jun 18, 2017 at 11:34 PM, Tony Breeds tony@bakeyournoodle.com wrote:

On Sun, Jun 18, 2017 at 08:19:16PM -0400, Davanum Srinivas wrote:

Awesome! thanks Tony, some kolla jobs do that for example, but i think
this job is a better one to key off of:
http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/infra.yaml#n381

Outline of the work is - check if there are any new releases in github
downloads, if so download them using wget and then delegate to the scp
publisher (with keep-hierarchy) to create the new directories and
upload the file(s).

So perhaps I'm dense but I can't see an easy way to get a list of
release artefacts from github in a form that wget can consume. The best
I can see is via the API. I've knocked up a quick'n'dirty mirror
script[1] but I really feel like I've gone off into the weeds.

You basically need to do:

git clone && cd
virtualenv .venv
.venv/bin/pip install -U pip setuptools wheel
.venv/bin/pip install -r ./requirements.txt # [2]
.venv/bin/python ./mirror-github-releases.py \
'coreos/etcd::.linux.gz:etcd' \
'coreos/etcd:6225411:.linux.gz:etcd'

Works for me!

Okay I'll put something more compleet together for infra review.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Jun 20, 2017 by Tony_Breeds (19,660 points)   3 6 10
0 votes

On Fri, Jun 16, 2017 at 12:06:47PM +1000, Tony Breeds wrote:
Hi All,
I just push a review [1] to bump the minimum etcd version to
3.2.0 which works on intel and ppc64le. I know we're pretty late in the
cycle to be making changes like this but releasing pike with a dependacy
on 3.1.x make it harder for users on ppc64le (not many but a few :D)

Yours Tony.

[1] https://review.openstack.org/474825

So this came up at the PTG and the current plan is:

Interim solution:
1. Get the mirroring tool to the point it can be consumed by infra.
2. setup a new zuulv3 job to run this and do the mirroring.

Middle term solution:
1. Get etcd3 packages updated in debian/ubuntu create a PPA (or
similar) for infra to consume.

Both of these are intended to be done during Queens.

Long term plan:
1. Ensue the packages above are just there for 18.04 so we can put this
behind us.

With hindsight I think we need to add something like "Current packages
can be consumed in our CI" as a requirement for anything added as a base
service.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Yours Tony.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Sep 25, 2017 by Tony_Breeds (19,660 points)   3 6 10
...