settingsLogin | Registersettings

[openstack-dev] [dib] diskimage-builder v2 RC1 release; request for test

0 votes

Hi

diskimage-builder has been working on a feature/v2 branch
incorporating some largely internal changes to the way it finds and
calls elements, enhancements to partitioning and removal of some
long-deprecated elements. We have just tagged 2.0.0rc1 and are
requesting testing by interested parties.

You can view the branch at

https://git.openstack.org/cgit/openstack/diskimage-builder/log/?h=feature/v2

For any issues it is probably best to file a bug. Developers are
standing by in #openstack-dib.

=== User facing changes

If you use dib exclusively via the command-line disk-image-create
installed from a package or via pypi you are unlikely to notice any
difference (if you run it directly from a git-tree checkout, you're in
development territory so read on).

v2 includes a new framework for partitioning contributed by Andreas
Florath. This should allow for creating multiple partitions, images
with encryption, LVM support and flexibility for multiple-devices, all
of which are currently not supported. Please check the v2
documentation, specs and reach out if these features interest you
(some parts still in review).

Element override is now supported. If you have an element of the same
name earlier in the ELEMENTS_PATH, it will override later instances
(previously, the behaviour was undefined).

A number of long-deprecated elements have been removed in v2, which
are to the best of our knowledge unused.

  • partitioning-sfdisk
  • deploy-ironic-element
  • ironc-discovered-ramdisk
  • serial-console-element
  • map-services

=== Developer changes

Developers, or those that are doing more "interesting" things in
deployment or CI, might like to read below to follow the history of
the branch and hopefully do some testing with the feature/v2 branch
ASAP so we can address any concerns.

dib started out mostly as a shell script collection that used setup.py
to install itself. Over time it has made more and more use of Python.
The current instructions say to checkout the source tree and run
"./bin/disk-image-create"; this causes a number of issues for the
internal Python components. We carry hacks to try and figure out of
we're being called uninstalled from a source tree, or installed in pip
develop mode, or installed on the system.

For purposes of both users and development we want dib to be as
"pythonic" as possible and behave like all other projects. Two major
visible changes are:

  • command-line scripts are entry points (i.e. need to be installed)
  • elements have moved under diskimage_create module

The result of the first is that "./bin/disk-image-create" from the
source tree is no longer there. Like all other projects, you should
install dib into a virtualenv (if you're developing, use pip -e) and
"disk-image-create" will "just work". FYI, on the back-end, this
entry-point essentially exec()s the old shell-script driver; but we
reserve the right to move more of the logic into python over time.
Also, instead of having to os.system(/path/disk-image-create), we can
allow direct calls like a real library.

A side-effect of this is that we have removed and deprecated the
dib-utils package. This was intended to be a more generic repository
of tools that might be useful outside dib, but that did not eventuate
and it has been folded back into dib for simplicity.

The second change, moving the inbuilt elements under the
diskimagecreate module, is a simplification so we always have a
canonical path to our elements. Currently the elements are installed
via "data
files" in setup.cfg, which means setup.py puts the elements
in /usr/share... when installing, or leaves them in the tree when
installed via pip -e. Again we have path guessing hacks trying to
guess what situation we're in. Elements have moved from elements/ to
diskimage_builder/elements in the source tree (we did try to minimise
this with symlinks, but setuptools doesn't like it, it complicates
things when git merging across the branches and is legacy cruft to go
wrong in the future. It was better to make a clean break).

Since we always know where elements are relative to the imported
diskimage_builder module we can drop all the path guessing complexity.
This has other good flow-on effects such as testr being able to find
unit-tests for elements in the normal fashion and having imports work
as usual.

We are aware there are a number of tools that like to take dib
elements and do things with them. This is fine, I guess, although of
course if we don't know about how you use elements we're liable to
break them. Thus your use may be confused that the elements have
moved.

Reading some of the dib source you may find there is a canonical way
to find out the included dib elements path -- ask dib itself,
something like:

DIBELEMENTS=$(python -c '
import diskimage
builder.paths;
diskimagebuilder.paths.showpath("elements")')

Note you probably do not want this. As mentioned, another feature of
v2 is override elements -- an element that appears first in the
element path-list will override any built-in one (just like $PATH).
There is a function, diskimagebuilder.getelements(), which will
correctly process the element path, calculate overrides and return a
canonical list of elements, their dependencies and correct paths.

THAT said, you probably do not want this either! There are a number
of elements that do things on behalf of other elements -- they look
for a file in the included elements, say, and use that as a manifest
for something. Previously, these would just have to make up their own
element processing via inspection of the command-line arguments. dib
now exports pre-computed variables that an element can walk for all
the current build elements -- a YAML list for easy python decoding and
a function that builds an array for Bash elements.

=== Merge plans

Barring any new developments, I would think that 1.28.0 would be the
last release before the v2 branch is merged back to master. I imagine
this would be in about one to two weeks.

The future is obviously dependent on what contributions appear. It
seems the main area of interest currently is in creation of
containers; if this interests you keep an eye on the review queue.

Thanks,

-i


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Mar 20, 2017 in openstack-dev by Ian_Wienand (3,620 points)   4 5

5 Responses

0 votes

Hello Ian,

I think I found a regression: since [1] only the first
package specified using '-p' is picked up.
You can find a proposal for a fix under [2].
Maybe it's worth to include this in V2?

Kind regards

Andre

[1] https://review.openstack.org/#/c/396702
[2] https://review.openstack.org/#/c/438010


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Feb 24, 2017 by Andre_Florath (620 points)  
0 votes

On Thu, Feb 9, 2017 at 12:22 AM, Ian Wienand iwienand@redhat.com wrote:
Hi

diskimage-builder has been working on a feature/v2 branch
incorporating some largely internal changes to the way it finds and
calls elements, enhancements to partitioning and removal of some
long-deprecated elements. We have just tagged 2.0.0rc1 and are
requesting testing by interested parties.

I am too late, but it is a good place to mention that 2.0.0 introduced
a regression that ends up breaking image build on ppc64 platform
somehwere within bootloader element.

Error trace:

2017-03-17 15:54:37,317 INFO nodepool.image.build.devstack-xenial: +
[[ ppc64el =~ ppc ]]
2017-03-17 15:54:37,317 INFO nodepool.image.build.devstack-xenial: +
/usr/sbin/grub-install --modules=part_msdos --force /dev/loop0
--no-nvram
2017-03-17 15:54:37,369 INFO nodepool.image.build.devstack-xenial:
Installing for powerpc-ieee1275 platform.
2017-03-17 15:54:45,912 INFO nodepool.image.build.devstack-xenial:
/usr/sbin/grub-install: warning: unknown device type loop0p1
2017-03-17 15:54:45,913 INFO nodepool.image.build.devstack-xenial: .
2017-03-17 15:54:45,987 INFO nodepool.image.build.devstack-xenial:
/usr/sbin/grub-install: error: the chosen partition is not a PReP
partition.

On 1.2.8, the cmdline was /usr/sbin/grub-install --modules=part_msdos --force /dev/loop0p1 --no-nvram. I do not know
yet why grub-install thinks it should still use loop0p1 on 2.1.0.

I have not dug too deep into it, ended up downgrading
diskimage-builder to 1.2.8. Simple downgrade would not work, because
1.2.8 would now fail with dib-run-parts binary missing. So I also had
to copy dib-run-parts from 2.1.0. Maybe installing from source and
pinning 1.2.8 would have been a better strategy.


Mikhail Medvedev (mmedvede)
IBM


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 17, 2017 by Mikhail_Medvedev (1,180 points)   1
0 votes

Hello!

Thanks for the bug report. Can you please file this as a bug?

There is a very high probability that I introduced a change that
leads to the failure [1] - even if this is fixed now there is a
high probability that it will fail again when [2] is merged.

The reason is, that there are no test cases because there is no
nodepool CI job running on PPC. (Or do I miss something here?)

We are only a very few people at diskimage-builder with limited
resources and had to concentrate on the 'main-line' (i.e.: that
what can be tested by us). A discussion about what to support
or test was already started some time ago [3].

Looks that you are from IBM: would it be possible to provide
PPC hardware for testing and the man-power to integrate
this into the CI?
This would really help finding such problems during development
phase and would put me into the situation to be able to fix your
problem.

Kind regards

Andre

[1] https://review.openstack.org/#/c/375261/
[2] https://review.openstack.org/#/c/444586/
[3] https://review.openstack.org/#/c/418204/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 17, 2017 by Andre_Florath (620 points)  
0 votes

On Fri, Mar 17, 2017 at 3:23 PM, Andre Florath andre@florath.net wrote:
Hello!

Thanks for the bug report. Can you please file this as a bug?

Hi Andre,
Submitted the bug https://bugs.launchpad.net/diskimage-builder/+bug/1674402

There is a very high probability that I introduced a change that
leads to the failure [1] - even if this is fixed now there is a
high probability that it will fail again when [2] is merged.

The reason is, that there are no test cases because there is no
nodepool CI job running on PPC. (Or do I miss something here?)

Correct, there isn't a ppc CI running on diskimage-builder patches.

We are only a very few people at diskimage-builder with limited
resources and had to concentrate on the 'main-line' (i.e.: that
what can be tested by us). A discussion about what to support
or test was already started some time ago [3].

Looks that you are from IBM: would it be possible to provide
PPC hardware for testing and the man-power to integrate
this into the CI?
This would really help finding such problems during development
phase and would put me into the situation to be able to fix your
problem.

Agreed, there is little can be done without being able to test the failure case.

Would adding a third-party CI job help? I can put together a
functional job on ppc64. I assume we want a job based on
gate-dib-dsvm-functests-*?

Kind regards

Andre

[1] https://review.openstack.org/#/c/375261/
[2] https://review.openstack.org/#/c/444586/
[3] https://review.openstack.org/#/c/418204/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 20, 2017 by Mikhail_Medvedev (1,180 points)   1
0 votes

On 03/21/2017 03:10 AM, Mikhail Medvedev wrote:
On Fri, Mar 17, 2017 at 3:23 PM, Andre Florath andre@florath.net wrote:
Submitted the bug https://bugs.launchpad.net/diskimage-builder/+bug/1674402

Thanks; some updates there.

Would adding a third-party CI job help? I can put together a
functional job on ppc64. I assume we want a job based on
gate-dib-dsvm-functests-*?

As discussed in #openstack-dib we have this reporting on a group of
the functional tests. My only concern is biting off more than we can
chew initially and essentially training people that the results are
unreliable. Once we get over this initial hurdle we can look at
expanding it and voting.

-i


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Mar 21, 2017 by Ian_Wienand (3,620 points)   4 5
...