settingsLogin | Registersettings

[openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

0 votes

The keystone team is pursuing a trigger-based approach to support rolling,
zero-downtime upgrades. The proposed operator experience is documented here:

http://docs.openstack.org/developer/keystone/upgrading.html

This differs from Nova and Neutron's approaches to solve for rolling
upgrades (which use oslo.versionedobjects), however Keystone is one of the
few services that doesn't need to manage communication between multiple
releases of multiple service components talking over the message bus (which
is the original use case for oslo.versionedobjects, and for which it is
aptly suited). Keystone simply scales horizontally and every node talks
directly to the database.

Database triggers are obviously a new challenge for developers to write,
honestly challenging to debug (being side effects), and are made even more
difficult by having to hand write triggers for MySQL, PostgreSQL, and
SQLite independently (SQLAlchemy offers no assistance in this case), as
seen in this patch:

https://review.openstack.org/#/c/355618/

However, implementing an application-layer solution with
oslo.versionedobjects is not an easy task either; refer to Neutron's
implementation:

https://review.openstack.org/#/q/topic:bp/adopt-oslo-versioned-objects-for-db

Our primary concern at this point are how to effectively test the triggers
we write against our supported database systems, and their various
deployment variations. We might be able to easily drop SQLite support (as
it's only supported for our own test suite), but should we expect variation
in support and/or actual behavior of triggers across the MySQLs, MariaDBs,
Perconas, etc, of the world that would make it necessary to test each of
them independently? If you have operational experience working with
triggers at scale: are there landmines that we need to be aware of? What is
it going to take for us to say we support zero dowtime upgrades with
confidence?

Steve & Dolph


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Aug 25, 2016 in openstack-dev by s.martinelli_at_gmai (5,460 points)   1 2 2
retagged Jan 26, 2017 by admin

43 Responses

0 votes

Personally I had very bad experiences with stored procedures and
triggers in previous jobs, where the amount of side effects that
occurred and the overall lack of maintainability of triggers and stored
procedures scared me off.

We handed off changes to stored procedures and
triggers to the DBAs, who had a tendency to not apply them correctly or
forget to apply them at a site. Then it was a total nightmare to try and
figure out why things wouldn't work, until we discovered that the
changes to an SP or Trigger wasn't actually applied.

Now, I don't think OpenStack as a project suffers the same
organizational dysfunction as my previous jobs, but just overall they're
hard to debug and maintain and I don't like to use them.

/rant

--
Sean M. Collins


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 25, 2016 by Sean_M._Collins (11,480 points)   3 7 8
0 votes

On Thu, Aug 25, 2016 at 1:13 PM, Steve Martinelli s.martinelli@gmail.com
wrote:

The keystone team is pursuing a trigger-based approach to support rolling,
zero-downtime upgrades. The proposed operator experience is documented here:

http://docs.openstack.org/developer/keystone/upgrading.html

This differs from Nova and Neutron's approaches to solve for rolling
upgrades (which use oslo.versionedobjects), however Keystone is one of the
few services that doesn't need to manage communication between multiple
releases of multiple service components talking over the message bus (which
is the original use case for oslo.versionedobjects, and for which it is
aptly suited). Keystone simply scales horizontally and every node talks
directly to the database.

Database triggers are obviously a new challenge for developers to write,
honestly challenging to debug (being side effects), and are made even more
difficult by having to hand write triggers for MySQL, PostgreSQL, and
SQLite independently (SQLAlchemy offers no assistance in this case), as
seen in this patch:

https://review.openstack.org/#/c/355618/

However, implementing an application-layer solution with
oslo.versionedobjects is not an easy task either; refer to Neutron's
implementation:

https://review.openstack.org/#/q/topic:bp/adopt-oslo-
versioned-objects-for-db

Our primary concern at this point are how to effectively test the triggers
we write against our supported database systems, and their various
deployment variations. We might be able to easily drop SQLite support (as
it's only supported for our own test suite), but should we expect variation
in support and/or actual behavior of triggers across the MySQLs, MariaDBs,
Perconas, etc, of the world that would make it necessary to test each of
them independently? If you have operational experience working with
triggers at scale: are there landmines that we need to be aware of? What is
it going to take for us to say we support zero dowtime upgrades with
confidence?

Steve & Dolph

No experience to add for triggers, but I'm happy to help test this on a
MySQL Galera cluster. I'd also like to add thanks for looking into this. A
keystone outage is a cloud outage and being able to eliminate them from
upgrades will be beneficial to everyone.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 25, 2016 by Matt_Fischer (9,340 points)   1 3 7
0 votes

This differs from Nova and Neutron's approaches to solve for rolling
upgrades (which use oslo.versionedobjects), however Keystone is one of
the few services that doesn't need to manage communication between
multiple releases of multiple service components talking over the
message bus (which is the original use case for oslo.versionedobjects,
and for which it is aptly suited). Keystone simply scales horizontally
and every node talks directly to the database.

Yeah, o.vo gives you nothing really if all you want is a facade behind
which to hide the application-level migrations. That doesn't mean it
would be a bad thing to use, but maybe overkill vs. just writing a
couple wrappers.

Database triggers are obviously a new challenge for developers to write,
honestly challenging to debug (being side effects), and are made even
more difficult by having to hand write triggers for MySQL, PostgreSQL,
and SQLite independently (SQLAlchemy offers no assistance in this case),
as seen in this patch:

https://review.openstack.org/#/c/355618/

However, implementing an application-layer solution with
oslo.versionedobjects is not an easy task either; refer to Neutron's
implementation:

Yeah, it's not trivial at the application level either but at least it
is in python and write-once for any kind of compatible backend. My
(extremely limited) experience with stored procedures is that they are
very difficult to get right, even as an expert in the technology, which
almost none of us are. Application-level migrations are significantly
simpler and exist closer to the domain of the rest of the code for a
specific new feature.

I will offer one bit of anecdotal information that may be relevant:
Several of the migrations that nova has done in the past have required
things like parsing/generating JSON, and making other API calls to look
up information needed to translate from one format to another. That
would (AFAIK) be quite difficult to do in the database itself, and may
mean you end up with a combination of both approaches in the long run.

I don't think that keystone necessarily needs to adopt the same approach
as the other projects (especially in the absence of things like
cross-version RPC compatibility) and so if stored procedures are really
the best fit then that's cool. They will themselves be a landmine in
front of me should I ever have to debug such a problem, but if they are
significantly better for the most part then so be it.

--Dan


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 25, 2016 by Dan_Smith (9,860 points)   1 2 4
0 votes

On 25/08/16 01:13 PM, Steve Martinelli wrote:
The keystone team is pursuing a trigger-based approach to support rolling, zero-downtime upgrades. The proposed operator experience is documented here:

http://docs.openstack.org/developer/keystone/upgrading.html

This differs from Nova and Neutron's approaches to solve for rolling upgrades (which use oslo.versionedobjects), however Keystone is one of the few services that doesn't need to manage communication between multiple releases of multiple service components talking over the message bus (which is the original use case for oslo.versionedobjects, and for which it is aptly suited). Keystone simply scales horizontally and every node talks directly to the database.

just curious, but does Keystone have any IPC or is it still just a single service interacting with db? if the latter, you should be able to just apply migration with no downtime as long as you don't modify/delete existing columns. similar experience as others, haven't really used stored procedures in a while but it's a pain wrt to portability. considering OpenStack has a habit of supporting every driver under the sun, i'm guessing driver specific solutions will get more difficult over time.

cheers,

--
gord


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 25, 2016 by gordon_chung (19,300 points)   2 3 8
0 votes

On 08/25/2016 01:13 PM, Steve Martinelli wrote:
The keystone team is pursuing a trigger-based approach to support
rolling, zero-downtime upgrades. The proposed operator experience is
documented here:

http://docs.openstack.org/developer/keystone/upgrading.html

This differs from Nova and Neutron's approaches to solve for rolling
upgrades (which use oslo.versionedobjects), however Keystone is one of
the few services that doesn't need to manage communication between
multiple releases of multiple service components talking over the
message bus (which is the original use case for oslo.versionedobjects,
and for which it is aptly suited). Keystone simply scales horizontally
and every node talks directly to the database.

Database triggers are obviously a new challenge for developers to write,
honestly challenging to debug (being side effects), and are made even
more difficult by having to hand write triggers for MySQL, PostgreSQL,
and SQLite independently (SQLAlchemy offers no assistance in this case),
as seen in this patch:

https://review.openstack.org/#/c/355618/

However, implementing an application-layer solution with
oslo.versionedobjects is not an easy task either; refer to Neutron's
implementation:

https://review.openstack.org/#/q/topic:bp/adopt-oslo-versioned-objects-for-db

Our primary concern at this point are how to effectively test the
triggers we write against our supported database systems, and their
various deployment variations. We might be able to easily drop SQLite
support (as it's only supported for our own test suite), but should we
expect variation in support and/or actual behavior of triggers across
the MySQLs, MariaDBs, Perconas, etc, of the world that would make it
necessary to test each of them independently? If you have operational
experience working with triggers at scale: are there landmines that we
need to be aware of? What is it going to take for us to say we support
zero dowtime upgrades with confidence?

I would really hold off doing anything triggers related until there was
sufficient testing for that, especially with potentially dirty data.

Triggers also really bring in a whole new DSL that people need to learn
and understand, not just across this boundary, but in the future
debugging issues. And it means that any errors happening here are now in
a place outside of normal logging / recovery mechanisms.

There is a lot of value that in these hard problem spaces like zero down
uptime we keep to common patterns between projects because there are
limited folks with the domain knowledge, and splitting that even further
makes it hard to make this more universal among projects.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 25, 2016 by Sean_Dague (66,200 points)   4 8 14
0 votes

On 08/25/2016 01:13 PM, Steve Martinelli wrote:
The keystone team is pursuing a trigger-based approach to support
rolling, zero-downtime upgrades. The proposed operator experience is
documented here:

http://docs.openstack.org/developer/keystone/upgrading.html

This differs from Nova and Neutron's approaches to solve for rolling
upgrades (which use oslo.versionedobjects), however Keystone is one of
the few services that doesn't need to manage communication between
multiple releases of multiple service components talking over the
message bus (which is the original use case for oslo.versionedobjects,
and for which it is aptly suited). Keystone simply scales horizontally
and every node talks directly to the database.

Hi Steve -

I'm a strong proponent of looking into the use of triggers to smooth
upgrades between database versions. Even in the case of projects
using versioned objects, it still means a SQL layer has to include
functionality for both versions of a particular schema change which
itself is awkward. I'm also still a little worried that not every case
of this can be handled by orchestration at the API level, and not as a
single SQL layer method that integrates both versions of a schema change.

Using triggers would resolve the issue of SQL-specific application code
needing to refer to two versions of a schema at once, at least for those
areas where triggers and SPs can handle it. In the "ideal", it means
all the Python code can just refer to one version of a schema, and nuts
and bolts embedded into database migrations would handle all the
movement between schema versions, including the phase between expand and
contract. Not that I think the "ideal" is ever going to be realized
100%, but maybe in some / many places, this can work.

So if Keystone wants to be involved in paving the way for working with
triggers, IMO this would benefit other projects in that they could
leverage this kind of functionality in those places where it makes sense.

The problem of "zero downtime database migrations" is an incredibly
ambitious goal and I think it would be wrong to exclude any one
particular technique in pursuing this. A real-world success story would
likely integrate many different techniques as they apply to specific
scenarios, and triggers and SPs IMO are a really major one which I
believe can be supported.

Database triggers are obviously a new challenge for developers to write,
honestly challenging to debug (being side effects), and are made even
more difficult by having to hand write triggers for MySQL, PostgreSQL,
and SQLite independently (SQLAlchemy offers no assistance in this case),
as seen in this patch:

So I would also note that we've been working on the availability of
triggers and stored functions elsewhere, a very raw patch that is to be
largely rolled into oslo.db is here:

https://review.openstack.org/#/c/314054/

This patch makes use of an Alembic pattern called "replaceable object",
which is intended specifically as a means of versioning things like
triggers and stored procedures:

http://alembic.zzzcomputing.com/en/latest/cookbook.html#replaceable-objects

Within the above Neutron patch, one thing I want to move towards is that
things like triggers and SPs would only need to be specified once, in
the migration layer, and not within the model. To achieve this, tests
that work against MySQL and Postgresql would need to ensure that the
test schema is built up using migrations, and not create_all. This is
already the case in some places and not in others. There is work
ongoing in oslo.db to provide a modernized fixture system that supports
enginefacade cleanly as well as allows for migrations to be used
efficiently (read: once per many tests) for all MySQL/Postgresql test
suites, athttps://review.openstack.org/#/c/351411/ .

As far as SQLite, I have a simple opinion with SQLite which is that
migrations, triggers, and SPs should not be anywhere near a SQLite
database. SQLite should be used strictly for simple model unit tests,
the schema is created using create_all(), and that's it. The test
fixture system accommodates this as well.

Our primary concern at this point are how to effectively test the
triggers we write against our supported database systems, and their
various deployment variations. We might be able to easily drop SQLite
support (as it's only supported for our own test suite), but should we
expect variation in support and/or actual behavior of triggers across
the MySQLs, MariaDBs, Perconas, etc, of the world that would make it
necessary to test each of them independently? If you have operational
experience working with triggers at scale: are there landmines that we
need to be aware of? What is it going to take for us to say we support
zero dowtime upgrades with confidence?

zero downtime is an extremely difficult goal. I appreciate that
people are generally nervous about making more use of relational
database features in order to help with this, however as long as the
goal includes an application that can communicate with a database that
is literally in flux as far as its schema, this is already an exotic
goal. Triggers and stored procedures are in fact very boring.

SQLAlchemy does support these features, it's just the specific trigger
and SP languages themselves are written as strings, not as elaborate
Python expressions. This should be seen as a good thing. Trigger and
SP languages are not like SQL in that they are not very declarative at
all, they are imperative. SQLAlchemy's abstraction of SQL into Python
expressions only works to the degree that the SQL itself is primarily
declarative. It would not be feasible to take on the task of producing
an imperative stored procedure / trigger language that compiles into
vendor-specific dialects.

For the use case of Openstack database migrations, I would hope that a
subset of the triggers and SPs that support live migrations would fall
into a fixed "vocabulary", each meeting various requirements, that are
of general use across projects and could be maintained in oslo.db
itself. For more elaborate cases, like "version A has the data in
three tables, version B has them in a single JSON blob", that probably
needs more specific logic.

I would encourage project teams to please loop me in on discussions
about triggers, SPs and migrations.

Steve & Dolph


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 26, 2016 by Mike_Bayer (15,260 points)   1 5 6
0 votes

On 25/08/16 13:13 -0400, Steve Martinelli wrote:
The keystone team is pursuing a trigger-based approach to support rolling,
zero-downtime upgrades. The proposed operator experience is documented here:

http://docs.openstack.org/developer/keystone/upgrading.html

This differs from Nova and Neutron's approaches to solve for rolling
upgrades (which use oslo.versionedobjects), however Keystone is one of the
few services that doesn't need to manage communication between multiple
releases of multiple service components talking over the message bus (which
is the original use case for oslo.versionedobjects, and for which it is
aptly suited). Keystone simply scales horizontally and every node talks
directly to the database.

Database triggers are obviously a new challenge for developers to write,
honestly challenging to debug (being side effects), and are made even more
difficult by having to hand write triggers for MySQL, PostgreSQL, and
SQLite independently (SQLAlchemy offers no assistance in this case), as
seen in this patch:

https://review.openstack.org/#/c/355618/

However, implementing an application-layer solution with
oslo.versionedobjects is not an easy task either; refer to Neutron's
implementation:

https://review.openstack.org/#/q/topic:bp/adopt-oslo-versioned-objects-for-db

Our primary concern at this point are how to effectively test the triggers
we write against our supported database systems, and their various
deployment variations. We might be able to easily drop SQLite support (as
it's only supported for our own test suite), but should we expect variation
in support and/or actual behavior of triggers across the MySQLs, MariaDBs,
Perconas, etc, of the world that would make it necessary to test each of
them independently? If you have operational experience working with
triggers at scale: are there landmines that we need to be aware of? What is
it going to take for us to say we support zero dowtime upgrades with
confidence?

Hey Steve, Dolph,

Thanks for sending this out. There's been some discussions in the Glance
community about how we can implement rolling upgrades and it seems like Glance's
case is very similar to keystone's.

I'll make sure folks in the glance community are aware of this thread and reach
out.

Flavio

--
@flaper87
Flavio Percoco


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Aug 30, 2016 by Flavio_Percoco (36,960 points)   3 7 10
0 votes

Excerpts from Mike Bayer's message of 2016-08-26 11:50:24 -0400:

On 08/25/2016 01:13 PM, Steve Martinelli wrote:

The keystone team is pursuing a trigger-based approach to support
rolling, zero-downtime upgrades. The proposed operator experience is
documented here:

http://docs.openstack.org/developer/keystone/upgrading.html

This differs from Nova and Neutron's approaches to solve for rolling
upgrades (which use oslo.versionedobjects), however Keystone is one of
the few services that doesn't need to manage communication between
multiple releases of multiple service components talking over the
message bus (which is the original use case for oslo.versionedobjects,
and for which it is aptly suited). Keystone simply scales horizontally
and every node talks directly to the database.

Hi Steve -

I'm a strong proponent of looking into the use of triggers to smooth
upgrades between database versions. Even in the case of projects
using versioned objects, it still means a SQL layer has to include
functionality for both versions of a particular schema change which
itself is awkward. I'm also still a little worried that not every case
of this can be handled by orchestration at the API level, and not as a
single SQL layer method that integrates both versions of a schema change.

Speaking as an operator, I'd rather have awkwardness happen in safe, warm
development, rather than in the cold, dirty, broken world of operations.

Speaking as a former DBA: Triggers introduce emergent behaviors and
complicate scaling and reasonable debugging in somewhat hidden ways that
can frustrate even the most experienced DBA. We've discussed FK's before,
and how they are a 1:1 trade-off of integrity vs. performance, and thus
deserve more scrutiny than they're typically given. Well IMO, triggers are
a 1:10 trade off between development complexity, and debugging complexity.

Speaking as a developer: Every case can in fact be handled simply and
in code without the database's help if we're willing to accept a small
level of imperfection and redundancy.

Using triggers would resolve the issue of SQL-specific application code
needing to refer to two versions of a schema at once, at least for those
areas where triggers and SPs can handle it. In the "ideal", it means
all the Python code can just refer to one version of a schema, and nuts
and bolts embedded into database migrations would handle all the
movement between schema versions, including the phase between expand and
contract. Not that I think the "ideal" is ever going to be realized
100%, but maybe in some / many places, this can work.

As someone else brought up, this is an unnecessarily bleak view of how database
migrations work.

It's simple, these are the holy SQL schema commandments:

Don't delete columns, ignore them.
Don't change columns, create new ones.
When you create a column, give it a default that makes sense.
Do not add new foreign key constraints.

Following these commandments, one can run schema changes at any time. A
new schema should be completely ignorable by older code, because their
columns keep working, and no new requirements are introduced. New code
can deal with defaulted new columns gracefully.

Of course, once one can be certain that all app code is updated, one can
drop old columns and tables, and add FK constraints (if you so desire,
I personally think they're a waste of precious DB resources, but that
is a much more religious debate and I accept that it's not part of
this debate).

So if Keystone wants to be involved in paving the way for working with
triggers, IMO this would benefit other projects in that they could
leverage this kind of functionality in those places where it makes sense.

The problem of "zero downtime database migrations" is an incredibly
ambitious goal and I think it would be wrong to exclude any one
particular technique in pursuing this. A real-world success story would
likely integrate many different techniques as they apply to specific
scenarios, and triggers and SPs IMO are a really major one which I
believe can be supported.

I don't think it's all that ambitious to think we can just use tried and
tested schema evolution techniques that work for everyone else.

Database triggers are obviously a new challenge for developers to write,
honestly challenging to debug (being side effects), and are made even
more difficult by having to hand write triggers for MySQL, PostgreSQL,
and SQLite independently (SQLAlchemy offers no assistance in this case),
as seen in this patch:

So I would also note that we've been working on the availability of
triggers and stored functions elsewhere, a very raw patch that is to be
largely rolled into oslo.db is here:

https://review.openstack.org/#/c/314054/

This patch makes use of an Alembic pattern called "replaceable object",
which is intended specifically as a means of versioning things like
triggers and stored procedures:

http://alembic.zzzcomputing.com/en/latest/cookbook.html#replaceable-objects

Within the above Neutron patch, one thing I want to move towards is that
things like triggers and SPs would only need to be specified once, in
the migration layer, and not within the model. To achieve this, tests
that work against MySQL and Postgresql would need to ensure that the
test schema is built up using migrations, and not create_all. This is
already the case in some places and not in others. There is work
ongoing in oslo.db to provide a modernized fixture system that supports
enginefacade cleanly as well as allows for migrations to be used
efficiently (read: once per many tests) for all MySQL/Postgresql test
suites, athttps://review.openstack.org/#/c/351411/ .

As far as SQLite, I have a simple opinion with SQLite which is that
migrations, triggers, and SPs should not be anywhere near a SQLite
database. SQLite should be used strictly for simple model unit tests,
the schema is created using create_all(), and that's it. The test
fixture system accommodates this as well.

Agreed on this. Spin up a DB server in the functional tests if you want
to test any actual data manipulation.

Our primary concern at this point are how to effectively test the
triggers we write against our supported database systems, and their
various deployment variations. We might be able to easily drop SQLite
support (as it's only supported for our own test suite), but should we
expect variation in support and/or actual behavior of triggers across
the MySQLs, MariaDBs, Perconas, etc, of the world that would make it
necessary to test each of them independently? If you have operational
experience working with triggers at scale: are there landmines that we
need to be aware of? What is it going to take for us to say we support
zero dowtime upgrades with confidence?

zero downtime is an extremely difficult goal. I appreciate that
people are generally nervous about making more use of relational
database features in order to help with this, however as long as the
goal includes an application that can communicate with a database that
is literally in flux as far as its schema, this is already an exotic
goal. Triggers and stored procedures are in fact very boring.

Even more boring is just testing upgraded schemas with old versions of
code.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 30, 2016 by Clint_Byrum (40,940 points)   4 5 9
0 votes

Even in the case of projects using versioned objects, it still
means a SQL layer has to include functionality for both versions of
a particular schema change which itself is awkward.

That's not true. Nova doesn't have multiple models to straddle a
particular change. We just...

It's simple, these are the holy SQL schema commandments:

Don't delete columns, ignore them.
Don't change columns, create new ones.
When you create a column, give it a default that makes sense.
Do not add new foreign key constraints.

...do this ^ :)

We can drop columns once they're long-since-unused, but we still don't
need duplicate models for that.

--Dan


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 30, 2016 by Dan_Smith (9,860 points)   1 2 4
0 votes

On 08/30/2016 09:57 AM, Clint Byrum wrote:
>

As someone else brought up, this is an unnecessarily bleak view of how database
migrations work.

We aren't talking about database migrations. We are talking about
online database migrations, where we would like both the old and
new versions of the code, talking to the database at the same time.

If I write code that does this:

 SELECT foo, bar FROM table

then I do a migration that replaces "bar" with some new table, the new
SQL is:

 SELECT table.foo, othertable.bar FROM table JOIN othertable ON 

table.id == othertable.foo_id

Those two SQL statements are incompatible. The "new" version of the
code must expect and maintain the old "bar" column for the benefit of
the "old" version of the code still reading and writing to it. To me,
this seems to contradict your suggestion "don't delete columns, ignore
them". We can't ignore "bar" above.

Following these commandments, one can run schema changes at any time. A
new schema should be completely ignorable by older code, because their
columns keep working, and no new requirements are introduced. New code
can deal with defaulted new columns gracefully.

You need to specify how new code deals with the above two totally
different SQL statements "gracefully", except that it has to accommodate
for both versions of the schema at the same time. This may be
"graceful" in operator land but in developer land, there is no easy
solution for this. Unless there is, and nobody has shown it to me yet:

I don't think it's all that ambitious to think we can just use tried and
tested schema evolution techniques that work for everyone else.

People have been asking me for over a year how to do this, and I have no
easy answer, I'm glad that you do. I would like to see some examples of
these techniques.

If you can show me the SQL access code that deals with the above change,
that would help a lot.

If the answer is, "oh well just don't do a schema change like that",
then we're basically saying we aren't really changing our schemas
anymore except for totally new features that otherwise aren't accessed
by the older version of the code. That's fine. It's not what people
coming to me are saying, though.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 30, 2016 by Mike_Bayer (15,260 points)   1 5 6
...