settingsLogin | Registersettings

[Openstack-operators] Upgrade OpenStack Juno to Mitaka

0 votes

Hi,

my name is Michael, I am working at the Baden-Wuerttemberg Cooperative State
University Mannheim.

We have an installation of OpenStack with roundabout 14 nodes each 32 cores (1
controller (glance,keystone, etc.), 1 neutron, 2 objectstores, 1 blockstore, 9
compute nodes) the blockstore, the objectstore and the compute nodes uses a
storage node over iSCSI with multipath to store the data, virtual machines etc.
The glance image service uses the objectstore as storage for the images.

At the moment we are running Juno release and we want to upgrade the
installation to Mitaka without loosing any data (users, images, volumes, virtual
machines, etc.). I already try to find documentation how such an upgrade should
be performed but I didn't find any well described how to do this.

So the following questions has arised:

What is the best way to performe an upgrade from Juno to Mitaka?
Is the best way to upgrade from Juno -> Kilo -> Liberty -> Mitaka or is it
possible to migrate directly to mitaka?
Is it better to perform an inplace upgrade or is the beter solution to setup a
new environment?
What is the best way to save the existing data to import it in an new
environment?
Is there any well described how to or best practise guide for such an upgrade?

Any ideas or help would be welcome :-)

Kind regards,
Michael

Michael Stang
Laboringenieur, Dipl. Inf. (FH)

Duale Hochschule Baden-Württemberg Mannheim
Baden-Wuerttemberg Cooperative State University Mannheim
ZeMath Zentrum für mathematisch-naturwissenschaftliches Basiswissen
Fachbereich Informatik, Fakultät Technik
Coblitzallee 1-9
68163 Mannheim

zemath@dhbw-mannheim.de
http://www.dhbw-mannheim.de_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

asked Jun 15, 2016 in openstack-operators by Michael_Stang (1,200 points)   1 1 4

21 Responses

0 votes

Hello,

first of all I suggest you read this article:
http://superuser.openstack.org/articles/openstack-upgrading-tutorial-11-pitfalls-and-solutions

What is the best way to performe an upgrade from Juno to Mitaka?

I would go for the in place upgrade, but I always upgraded without
jumping version, that AFAIK is not supported.

The main problem I see, is that database migrations are supported when
you upgrade to the next release, but if you jump form Juno to Mitaka I
have to idea how the database upgrade cloud be done.

Saverio


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 15, 2016 by Saverio_Proto (5,480 points)   1 4 6
0 votes

+1 on Saverio's response. You will want to upgrade to Mitaka by NOT jumping
releases; J - K - L - M not J - M

On Wed, Jun 15, 2016 at 4:13 AM, Saverio Proto zioproto@gmail.com wrote:

Hello,

first of all I suggest you read this article:

http://superuser.openstack.org/articles/openstack-upgrading-tutorial-11-pitfalls-and-solutions

What is the best way to performe an upgrade from Juno to Mitaka?

I would go for the in place upgrade, but I always upgraded without
jumping version, that AFAIK is not supported.

The main problem I see, is that database migrations are supported when
you upgrade to the next release, but if you jump form Juno to Mitaka I
have to idea how the database upgrade cloud be done.

Saverio


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 15, 2016 by Melvin_Hillsman (4,480 points)   1 2 2
0 votes

We¹re almost complete with this process, we¹re moving from icehouse to
kilo.

Certainly for nova, I was able to jump directly from I->K however, I did
have to deal with lots of weird issues that ended up with me having to
update the computes at the same time as the controllers.

With regards to upgrade levels, I found the docs to be a little
incomplete, I also found that certain components (e.g nova-cert) didn¹t
take a release name as per the docs and required an RPC version number.

Watch out, you¹ll need to refactor your configs quite a bit, things have
been moved into different stanza¹s, new options added.

From: Melvin Hillsman mrhillsman@gmail.com
Date: Wednesday, June 15, 2016 at 8:07 AM
To: Saverio Proto zioproto@gmail.com
Cc: OpenStack Operators openstack-operators@lists.openstack.org
Subject: Re: [Openstack-operators] Upgrade OpenStack Juno to Mitaka

+1 on Saverio's response. You will want to upgrade to Mitaka by NOT
jumping releases; J - K - L - M not J - M

On Wed, Jun 15, 2016 at 4:13 AM, Saverio Proto
zioproto@gmail.com wrote:

Hello,

first of all I suggest you read this article:
http://superuser.openstack.org/articles/openstack-upgrading-tutorial-11-pit
falls-and-solutions

What is the best way to performe an upgrade from Juno to Mitaka?

I would go for the in place upgrade, but I always upgraded without
jumping version, that AFAIK is not supported.

The main problem I see, is that database migrations are supported when
you upgrade to the next release, but if you jump form Juno to Mitaka I
have to idea how the database upgrade cloud be done.

Saverio


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 15, 2016 by Kingshott,_Daniel (480 points)  
0 votes


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 15, 2016 by omgjlk_at_us.ibm.com (380 points)  
0 votes

On Wed, Jun 15, 2016 at 03:19:28PM +0000, Jesse Keating wrote:
I'll offer a counter point.

We're not doing Juno to Mitaka, however we are doing Kilo to Mitaka, skipping
over Liberty.

The database migrations to get from Kilo to Mitaka have ran smoothly for us.

While it is great that it /appeared/ to work correctly for you, that is in
no way guaranteed. There also might be data that has silently been incorrectly
migrated due to missing the intermediate release that could cause problems
at some indeterminate point down the road. While we do test the N -> N+1
upgrade path, there is no CI testing of the N -> N + 2 upgrade path. IOW
while it may have worked for you between these 2 particular releases, there
is again no guarantee it'll work for any future pair of N, N+2 releases.
If you want to accept the risks, that's fine, but I'd certainly not suggest
it is a reasonable thing todo for deployments in general. Also what happened
to work for you may just as easily not work for other people, depending on
characteristics of their deployment configuration & data set.

Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 15, 2016 by Daniel_P._Berrange (27,920 points)   2 4 9
0 votes

On 6/15/2016 10:30 AM, Daniel P. Berrange wrote:
On Wed, Jun 15, 2016 at 03:19:28PM +0000, Jesse Keating wrote:

I'll offer a counter point.

We're not doing Juno to Mitaka, however we are doing Kilo to Mitaka, skipping
over Liberty.

The database migrations to get from Kilo to Mitaka have ran smoothly for us.

While it is great that it /appeared/ to work correctly for you, that is in
no way guaranteed. There also might be data that has silently been incorrectly
migrated due to missing the intermediate release that could cause problems
at some indeterminate point down the road. While we do test the N -> N+1
upgrade path, there is no CI testing of the N -> N + 2 upgrade path. IOW
while it may have worked for you between these 2 particular releases, there
is again no guarantee it'll work for any future pair of N, N+2 releases.
If you want to accept the risks, that's fine, but I'd certainly not suggest
it is a reasonable thing todo for deployments in general. Also what happened
to work for you may just as easily not work for other people, depending on
characteristics of their deployment configuration & data set.

Regards,
Daniel

+1 to everything Daniel said. Nova really expects release-to-release
upgrades. We do online data migrations between releases. Maybe one
reason you're getting this to work is we have a nova-manage command to
force the migration of data between releases rather than doing the
online data migration as resources are accessed. But there are some DB
migrations where we do a full stop until you've migrated the data.

--

Thanks,

Matt Riedemann


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 15, 2016 by Matt_Riedemann (48,320 points)   3 7 20
0 votes

+1 to everything Daniel said. Nova really expects release-to-release
upgrades. We do online data migrations between releases. Maybe one
reason you're getting this to work is we have a nova-manage command to
force the migration of data between releases rather than doing the
online data migration as resources are accessed. But there are some DB
migrations where we do a full stop until you've migrated the data.

Yeah, this ^

We try to put blocker migrations into the stream at critical
synchronization points to make sure that anything that has to be
online migrated will have been complete before we roll forward (usually
because we're about to drop something or introduce a constraint).
However, I'm sure there are some subtleties we don't catch with that.

To echo and expand on what Matt said, if you're going to do an
accelerated upgrade, I would at least recommend deploying the
intermediate code an running all the online migrations to completion
(after applying the intermediate schema updates) before rolling to the
next. This means running this after "db sync" for releases after it was
added:

https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L823

Just deploying the target code and running online migrations isn't
really enough, because we often remove the migration code once we know
it (should have been) run to completion. Since we can't know everyone's
upgrade cadence, and since our official support is "N-1 to N", that's
the price you pay for skipping a release.

--Dan


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 15, 2016 by Dan_Smith (9,860 points)   1 2 4
0 votes

Hello Saverio,

thank you for the link, I will have a look at this article.

In place upgrade would be the best way, becaus we do not have to setup
everything from scratch again, also at the moment it would be no problem to make
a new installation because there is not much data by now in our installation
only a few projects. But in the future we will have mor data, and the next
upgrade will come sometimes so then we have the same problems and it would be
nice not to have setup everything again.

So I think we will try to save the data and go for an in place upgrade first and
see how it works out.

Is this one the actual guid for upgrades, and is it valid for every upgrade or
ony for specific versions?:
http://docs.openstack.org/ops-guide/ops_upgrades.html

Kind regards,
Michael

Saverio Proto zioproto@gmail.com hat am 15. Juni 2016 um 11:13 geschrieben:

Hello,

first of all I suggest you read this article:
http://superuser.openstack.org/articles/openstack-upgrading-tutorial-11-pitfalls-and-solutions

What is the best way to performe an upgrade from Juno to Mitaka?

I would go for the in place upgrade, but I always upgraded without
jumping version, that AFAIK is not supported.

The main problem I see, is that database migrations are supported when
you upgrade to the next release, but if you jump form Juno to Mitaka I
have to idea how the database upgrade cloud be done.

Saverio


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 16, 2016 by Michael_Stang (1,200 points)   1 1 4
0 votes

Michael Stang writes:

Is this one the actual guid for upgrades, and is it valid for every
upgrade or ony for specific versions?:
http://docs.openstack.org/ops-guide/ops_upgrades.html

Yes, that's part of the official Operations Guide. It is not
version-specific. The examples are based on Ubuntu as the underlying OS
distribution. But the approach and recommendations are general.
--
Simon.


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 18, 2016 by Simon_Leinen (620 points)   1
0 votes

I think we give it a try then, thank you :-)

Kind regards,
Michael

Simon Leinen simon.leinen@switch.ch hat am 18. Juni 2016 um 16:21
geschrieben:

Michael Stang writes:

Is this one the actual guid for upgrades, and is it valid for every
upgrade or ony for specific versions?:
http://docs.openstack.org/ops-guide/ops_upgrades.html

Yes, that's part of the official Operations Guide. It is not
version-specific. The examples are based on Ubuntu as the underlying OS
distribution. But the approach and recommendations are general.
--
Simon.


OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
responded Jun 20, 2016 by Michael_Stang (1,200 points)   1 1 4
...