settingsLogin | Registersettings

[openstack-dev] [tripleo] Roadmap for Container CI work

0 votes

After our weekly meeting of today, I found useful to share and discuss
our roadmap for Container CI jobs in TripleO.
They are ordered by priority from the highest to lowest:

  1. Swap ovb-nonha job with ovb-containers, enable introspection on the
    container job and shuffle other coverage (e.g ssl) to other jobs
    (HA?). It will help us to get coverage for ovb-containers scenario
    again, without consuming more rh1 resources and keep existing
    coverage.
  2. Get multinode coverage of deployments - this should integrate with
    the scenarios we already have defined for non-container deployment.
    This is super important to cover all overcloud services, like we did
    with classic deployments. It should be non voting to start and then
    voting once it works. We should find a way to keep the same templates
    as we have now, and just include the docker environment. In other
    words, find a way to keep using:
    https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-multinode.yaml
    so we don't duplicate scenario environments.
  3. Implement container upgrade job, which for Pike will be deploy a
    baremetal overcloud, then migrate on upgrade to containers. Use
    multinode jobs for this task. Start with a non-voting job and move to
    the gate once it work. I also suggest to use scenarios framework, so
    we keep good coverage.
  4. After we implement the workflow for minor updates, have a job with
    tests container-to-container updates for minor (rolling) updates, this
    ideally should add some coverage to ensure no downtime of APIs and
    possibly checks for service restarts (ref recent bugs about bouncing
    services on minor updates)
  5. Once Pike is released and Queens starts, let's work on container to
    containers upgrade job.

Any feedback or question is highly welcome,

Note: The proposal comes from shardy's notes on
https://etherpad.openstack.org/p/tripleo-container-ci - feel free to
contribute to the etherpad or mailing list.

Thanks,
--
Emilien Macchi


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Apr 6, 2017 in openstack-dev by emilien_at_redhat.co (36,940 points)   3 8 13

4 Responses

0 votes

On Tue, 2017-04-04 at 16:01 -0400, Emilien Macchi wrote:
After our weekly meeting of today, I found useful to share and
discuss
our roadmap for Container CI jobs in TripleO.
They are ordered by priority from the highest to lowest:

  1. Swap ovb-nonha job with ovb-containers, enable introspection on
    the
    container job and shuffle other coverage (e.g ssl) to other jobs
    (HA?). It will help us to get coverage for ovb-containers scenario
    again, without consuming more rh1 resources and keep existing
    coverage.

The existing containers job already had introspection enabled I think.
So this is largely a swap (containers for nonha). We may move some of
the SSL features into the existing HA job if that makes more sense to
keep coverage on par with what the nonha job was already covering.

  1. Get multinode coverage of deployments - this should integrate with
    the scenarios we already have defined for non-container deployment.
    This is super important to cover all overcloud services, like we did
    with classic deployments. It should be non voting to start and then
    voting once it works. We should find a way to keep the same templates
    as we have now, and just include the docker environment. In other
    words, find a way to keep using:
    https://github.com/openstack/tripleo-heat-templates/blob/master/ci/en
    vironments/scenario001-multinode.yaml
    so we don't duplicate scenario environments.
  2. Implement container upgrade job, which for Pike will be deploy a
    baremetal overcloud, then migrate on upgrade to containers. Use
    multinode jobs for this task. Start with a non-voting job and move to
    the gate once it work. I also suggest to use scenarios framework, so
    we keep good coverage.

Using multinode here is a great start. We might also at some point
decide to swap out the existing OVB job to be a baremetal -> containers
version so we have end to end coverage there as well.

  1. After we implement the workflow for minor updates, have a job with
    tests container-to-container updates for minor (rolling) updates,
    this
    ideally should add some coverage to ensure no downtime of APIs and
    possibly checks for service restarts (ref recent bugs about bouncing
    services on minor updates)
  2. Once Pike is released and Queens starts, let's work on container
    to
    containers upgrade job.

Any feedback or question is highly welcome,

Note: The proposal comes from shardy's notes on
https://etherpad.openstack.org/p/tripleo-container-ci - feel free to
contribute to the etherpad or mailing list.

Thanks,


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 4, 2017 by Dan_Prince (8,160 points)   1 5 8
0 votes

On Tue, Apr 04, 2017 at 04:01:48PM -0400, Emilien Macchi wrote:
After our weekly meeting of today, I found useful to share and discuss
our roadmap for Container CI jobs in TripleO.
They are ordered by priority from the highest to lowest:

  1. Swap ovb-nonha job with ovb-containers, enable introspection on the
    container job and shuffle other coverage (e.g ssl) to other jobs
    (HA?). It will help us to get coverage for ovb-containers scenario
    again, without consuming more rh1 resources and keep existing
    coverage.
  2. Get multinode coverage of deployments - this should integrate with
    the scenarios we already have defined for non-container deployment.
    This is super important to cover all overcloud services, like we did
    with classic deployments. It should be non voting to start and then
    voting once it works. We should find a way to keep the same templates
    as we have now, and just include the docker environment. In other
    words, find a way to keep using:
    https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-multinode.yaml
    so we don't duplicate scenario environments.
  3. Implement container upgrade job, which for Pike will be deploy a
    baremetal overcloud, then migrate on upgrade to containers. Use
    multinode jobs for this task. Start with a non-voting job and move to
    the gate once it work. I also suggest to use scenarios framework, so
    we keep good coverage.
  4. After we implement the workflow for minor updates, have a job with
    tests container-to-container updates for minor (rolling) updates, this
    ideally should add some coverage to ensure no downtime of APIs and
    possibly checks for service restarts (ref recent bugs about bouncing
    services on minor updates)
  5. Once Pike is released and Queens starts, let's work on container to
    containers upgrade job.

Any feedback or question is highly welcome,

+1, I think this makes sense and is well aligned with what we discussed in
the meeting.

I agree the priority is roughly in the order listed above, but provided we
have sufficient folks willing to help we can probably work on some of these
tasks in parallel, as really we need at least (1), (2) and (3) ASAP.

I've started looking at (4) but there is significant work required to
enable this as our current breakpoint based update workflow won't work, and
it looks like we also can't use the rolling update feature of
SoftwareDeploymentGroup, because we want each node to be fully updated
before moving on to the next.

Steve


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 5, 2017 by Steven_Hardy (16,900 points)   2 7 13
0 votes

On 4.4.2017 22:01, Emilien Macchi wrote:
After our weekly meeting of today, I found useful to share and discuss
our roadmap for Container CI jobs in TripleO.
They are ordered by priority from the highest to lowest:

  1. Swap ovb-nonha job with ovb-containers, enable introspection on the
    container job and shuffle other coverage (e.g ssl) to other jobs
    (HA?). It will help us to get coverage for ovb-containers scenario
    again, without consuming more rh1 resources and keep existing
    coverage.
  2. Get multinode coverage of deployments - this should integrate with
    the scenarios we already have defined for non-container deployment.
    This is super important to cover all overcloud services, like we did
    with classic deployments. It should be non voting to start and then
    voting once it works. We should find a way to keep the same templates
    as we have now, and just include the docker environment. In other
    words, find a way to keep using:
    https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-multinode.yaml
    so we don't duplicate scenario environments.
  3. Implement container upgrade job, which for Pike will be deploy a
    baremetal overcloud, then migrate on upgrade to containers. Use
    multinode jobs for this task. Start with a non-voting job and move to
    the gate once it work. I also suggest to use scenarios framework, so
    we keep good coverage.

The first iteration of this job is ready to be reviewed and landed.
Please see the patches here [1].

The latest job execution didn't go all the way to success yet, it failed
during Ansible upgrade steps execution [2], but i think the patches are
now far enough that they would be good to merge anyway, and issues can
be ironed out subsequently, as well as making the job actually
Ocata->master rather than master->master (currently just switching from
non-containers to containers).

[1] https://review.openstack.org/#/q/topic:container-upgrade
[2]
http://logs.openstack.org/84/450784/8/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/a1850f7/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz

  1. After we implement the workflow for minor updates, have a job with
    tests container-to-container updates for minor (rolling) updates, this
    ideally should add some coverage to ensure no downtime of APIs and
    possibly checks for service restarts (ref recent bugs about bouncing
    services on minor updates)
  2. Once Pike is released and Queens starts, let's work on container to
    containers upgrade job.

Any feedback or question is highly welcome,

Note: The proposal comes from shardy's notes on
https://etherpad.openstack.org/p/tripleo-container-ci - feel free to
contribute to the etherpad or mailing list.

Thanks,


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 6, 2017 by =?UTF-8?B?SmnFmcOtIF (3,860 points)   2 3
0 votes

On 6.4.2017 12:46, Jiří Stránský wrote:
On 4.4.2017 22:01, Emilien Macchi wrote:

After our weekly meeting of today, I found useful to share and discuss
our roadmap for Container CI jobs in TripleO.
They are ordered by priority from the highest to lowest:

  1. Swap ovb-nonha job with ovb-containers, enable introspection on the
    container job and shuffle other coverage (e.g ssl) to other jobs
    (HA?). It will help us to get coverage for ovb-containers scenario
    again, without consuming more rh1 resources and keep existing
    coverage.
  2. Get multinode coverage of deployments - this should integrate with
    the scenarios we already have defined for non-container deployment.
    This is super important to cover all overcloud services, like we did
    with classic deployments. It should be non voting to start and then
    voting once it works. We should find a way to keep the same templates
    as we have now, and just include the docker environment. In other
    words, find a way to keep using:
    https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-multinode.yaml
    so we don't duplicate scenario environments.
  3. Implement container upgrade job, which for Pike will be deploy a
    baremetal overcloud, then migrate on upgrade to containers. Use
    multinode jobs for this task. Start with a non-voting job and move to
    the gate once it work. I also suggest to use scenarios framework, so
    we keep good coverage.

The first iteration of this job is ready to be reviewed and landed.
Please see the patches here [1].

The latest job execution didn't go all the way to success yet, it failed
during Ansible upgrade steps execution [2], but i think the patches are
now far enough that they would be good to merge anyway, and issues can
be ironed out subsequently, as well as making the job actually
Ocata->master rather than master->master (currently just switching from
non-containers to containers).

[1] https://review.openstack.org/#/q/topic:container-upgrade
[2]
http://logs.openstack.org/84/450784/8/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/a1850f7/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz

Sorry the [2] link was incorrect, this is the right one:

http://logs.openstack.org/84/450784/9/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/23f9190/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz

  1. After we implement the workflow for minor updates, have a job with
    tests container-to-container updates for minor (rolling) updates, this
    ideally should add some coverage to ensure no downtime of APIs and
    possibly checks for service restarts (ref recent bugs about bouncing
    services on minor updates)
  2. Once Pike is released and Queens starts, let's work on container to
    containers upgrade job.

Any feedback or question is highly welcome,

Note: The proposal comes from shardy's notes on
https://etherpad.openstack.org/p/tripleo-container-ci - feel free to
contribute to the etherpad or mailing list.

Thanks,


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 6, 2017 by =?UTF-8?B?SmnFmcOtIF (3,860 points)   2 3
...