settingsLogin | Registersettings

Re: [openstack-dev] Future Trove development

0 votes

​The email thread below is an one that I have been having with Luke and
some others at IBM who have been working on Trove.

I feel that these kind(s) of conversations are better had in the mailing
list, and in replying to this email, I let the participants know that
absent any objections, I would like to continue this conversation on the
ML. Hearing nothing in the past two days, I am forwarding this to the ML.

-amrith

On Thu, Aug 17, 2017 at 4:24 PM, Luke Browning LukeBrowning@us.ibm.com
wrote:

Hi Amrith,

I understand that the Trove project will be going into maintenance mode
soon. I have a number of Trove related enhancements and bug fixes that I
would like to submit to the community, but I don't know if they would be
considered given that the project is going into maintenance mode. Please
elaborate on what you mean by maintenance mode.

​See: https://review.openstack.org/#/c/488947/

It is not a foregone conclusion that the project will go into maintenance
mode. Thierry and I (for example) are not convinced that this is the right
course of action, but if there are no offers to contribute to the project,
it may be a decision by default.

Do you subscribe to the OpenStack-dev mailing list​. Please also see
http://openstack.markmail.org/thread/4p6gp263fei4mbru

Finally, for a description of what maintenance mode means, please refer:
https://governance.openstack.org/tc/reference/tags/status_ma
intenance-mode.html

Would Trove be updated to support new OpenStack versions?

Would support be provided for Trove gate testing?
>
Would elements be enhanced to support Xenial and newer versions of

databases?

Is there a time limit associated with maintenance mode? For example, would

it remain in effect for a year or two after the new stuff is released?

​For the four questions above, I direct your attention to the definition
of the maintenance-mode tag (https://governance.openstack.
org/tc/reference/tags/status_maintenance-mode.html) and remind you that
support for different versions from the community, for gate testing and for
elements is dependent on people participating (and their employers allowing
them to do this). At this point, I have no one who has stepped up to do
this in any significant way. There are a couple of people from China Mobile
who want to help but who are largely in the weeds, trying to fix this and
that bug but have no idea what Trove is all about. IBM has a core reviewer
on the project ​(Mariam is copied on this email). I am happy that she's
able to devote what time she can to the project but absent competent
reviewers, whether your changes ever get merged or not remains an
interesting question. Since you will be contributing elements and propose
to contribute a tool, will you be (or will IBM be) offering to support it?

Let me provide some back ground on my code changes, so you will have a
better idea of what might be submitted in a patch.

I have written a fully automated command that creates a virtual guest
image based on Ubuntu 16.04 containing a user specified version of a
database. The user's selection is validated against an internal list of
databases built into the tool. Once validated, the tool creates the image,
registers it with Glance, and then creates a Trove Datastore definition for
it. This is done in a single command invocation that typically takes about
25 minutes to complete. For some of the databases, it takes considerably
longer (2 hours for mongodb) as I have to compile source code. The guest
image that is produced is complete from a Trove perspective. It includes
the Cloud specific trove-guestagent.conf file and SSH public keys for the
DBA and OpenStack controller nodes.

​What mechanism does it use to develop the image? is it diskimage-builder
or some other?

Is there something the matter with the existing tool that exists, it
already works, it does more than just build an image, and it is already
integrated in the gate. Why not use that?​


This approach is nice (having a full guest image with the guest agent and
all that but as you observe, it takes about 2 hours per build and from a
developers perspective, this is a pain.

I assume that the tool is therefore for production use cases.

Could the elements that you have developed be used with the existing tool
or is your tool a complete replacement for the one that already exists?​

​Is there some reason that this tool was not developed in the open, using
the normal open development process?

There are ways to accelerate the compiling of source code and such through
the use of image layers (if you are using DIB). I have elements and code
that will build most of the trove elements in minutes and I'm waiting to
see whether there is any point in contributing them to the upstream or not,
at this point. I could show you how to do that, and it would be awesome to
have you build your tool into the build process somehow. That would be a
win-win for all ...​

I also have commands that setup Nova flavors on a per database basis.

These flavors can be customized by the user before they are instantiated in
the cloud.

​The existing command (trovestack) already does this.​

I wouldn't expect to drop these commands to the Trove community. Only the
changes to the Trove-guestagent and elements enabling databases.

Here's the list of databases that are currently supported for ppc64le. In
some cases, I have had to add Xenial support.

mariadb 10.1 --- package comes from the community
mongodb 3.4 community edition --- source code from the community is
compiled as there is no ppc64le package
mongodb 3.4 enterprise edition --- package comes from community
mysql 5.7 --- package comes from Ubuntu 16.04. percona-xtrabackup-24 is
compiled as there is no power support in community
postgresql 9.5 --- package comes from Ubuntu 16.04
postgresql 9.6 --- package comes from community
redis 3.0 --- package comes from Ubuntu 16.04
redis 3.2 --- source code from the community is compiled

​Why not just contribute the elements to the existing project and have the
current tool use them?​

​What of other databases currently supported? galera cluster (pxc),
vertica, db2, percona, couchdb, and couchbase?

My element related code changes add environment variables that control
the database version, the source of the database software (distro,
community, enterprise), and logging options. In this way, the logic in the
element is more generalized and can be made to support multiple versions. I
also added an environment variable that specifies the name of the source
package to be downloaded and compiled when that is applicable. I also have
environment variables to enable database and trove guest agent logging.

​Which is nice except that your guest agent code may or may not know how
to deal with that.​

My trove-guestagent code changes are mostly about not reconfiguring the
database with configuration data remotely passed to it from the task
manager. My code changes are isolated to the prepare step. This logic looks
something like: if database version x, add or remove configuration
parameter. Note there is relatively very little of this code, ~150 LOCs, as
database configuration parameters rarely change. Note there is already some
of this DB version specific logic in the trove guest agent, so I am just
adding more of it. But I believe a better way is not to alter the database
configuration data when the database is fully configured in a Trove
compliant manner via elements. This would be implemented through a new
command argument to the Trove Guest agent which says ignore configuration
changes. It would be set by the elements as well.

​Hard to say whether this is a good approach or not with out more detail.​

I haven't looked into the scale out configuration data yet, but this can
be handled differently as required.

​Not sure what you mean.

So, are these the sorts of changes that I could add in a maintenance
release?

​Maintenance release of what, trove? Are you asking whether these changes
can be made in a stable branch? if yes, which branch? See
https://docs.openstack.org/project-team-guide/stable-branches.html and
the section "Appropriate Fixes".​

1) conditionally disable the dynamic installation and configuration of
database software in the trove-guest agent via a new command argument
2) add environment variables to trove elements enabling multiple versions
of databases to be installed and configured
3) add environment variables to trove elements to enable database and
trove-guest agent logging
4) add xenial support to elements
5) compiling source code in elements for missing stack elements for
ppc64le

These changes are implemented at: https://github.com/open-power-
ref-design-toolkit/os-services/tree/master/osa/dbaas/dbimage-builder

​OK, so this is a whole new tool, sure you can propose it but how do we
make sure that it gets integrated into the current testing/gate? If you
want to drop this then I'd also have to assume that there would be some
code that would make it part of the current gate, and ideally make it
generate the images for the current gate. But for that, the fact that you
install the guest agent onto the image is a serious drag.

Also, why this wasn't proposed as a blueprint or spec in the upstream so
it could be developed in the open, I don't quite understand. After all, you
are now showing up with a complete tool and asking that it be included
which largely runs counter to the whole notion of open development, at
least as I understand it.​


Finally, once committed, do you (or IBM) commit to maintaining this code
and fixing bugs in it should they arise, and upgrade/update it from time to
time?

Thanks, Luke Browning


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Sep 3, 2017 in openstack-dev by amrith.kumar_at_gmai (3,580 points)   2 2

2 Responses

0 votes

Amrith,

See answers to your questions below.

Cheers,
Luke

I understand that the Trove project will be going into maintenance mode
soon. I have a number of Trove related enhancements and bug fixes that
I
would like to submit to the community, but I don't know if they would
be
considered given that the project is going into maintenance mode.
Please
elaborate on what you mean by maintenance mode.

​See: https://review.openstack.org/#/c/488947/

It is not a foregone conclusion that the project will go into
maintenance
mode. Thierry and I (for example) are not convinced that this is the
right
course of action, but if there are no offers to contribute to the
project,
it may be a decision by default.

Do you subscribe to the OpenStack-dev mailing list​. Please also see
http://openstack.markmail.org/thread/4p6gp263fei4mbru

Finally, for a description of what maintenance mode means, please refer:
https://governance.openstack.org/tc/reference/tags/status_ma
intenance-mode.html

Would Trove be updated to support new OpenStack versions?

Would support be provided for Trove gate testing?
>
Would elements be enhanced to support Xenial and newer versions of

databases?

Is there a time limit associated with maintenance mode? For example,
would

it remain in effect for a year or two after the new stuff is released?

​For the four questions above, I direct your attention to the
definition
of the maintenance-mode tag (https://governance.openstack.
org/tc/reference/tags/status_maintenance-mode.html) and remind you that
support for different versions from the community, for gate testing and
for
elements is dependent on people participating (and their employers
allowing
them to do this). At this point, I have no one who has stepped up to do
this in any significant way. There are a couple of people from China
Mobile
who want to help but who are largely in the weeds, trying to fix this
and
that bug but have no idea what Trove is all about. IBM has a core
reviewer
on the project ​(Mariam is copied on this email). I am happy that she's
able to devote what time she can to the project but absent competent
reviewers, whether your changes ever get merged or not remains an
interesting question. Since you will be contributing elements and
propose
to contribute a tool, will you be (or will IBM be) offering to support
it?

Let me provide some back ground on my code changes, so you will have a
better idea of what might be submitted in a patch.

I have written a fully automated command that creates a virtual guest
image based on Ubuntu 16.04 containing a user specified version of a
database. The user's selection is validated against an internal list of
databases built into the tool. Once validated, the tool creates the
image,
registers it with Glance, and then creates a Trove Datastore definition
for
it. This is done in a single command invocation that typically takes
about
25 minutes to complete. For some of the databases, it takes
considerably
longer (2 hours for mongodb) as I have to compile source code. The
guest
image that is produced is complete from a Trove perspective. It
includes
the Cloud specific trove-guestagent.conf file and SSH public keys for
the
DBA and OpenStack controller nodes.

​What mechanism does it use to develop the image? is it
diskimage-builder
or some other?

Yes, diskimage-builder and Trove elements

Is there something the matter with the existing tool that exists, it
already works, it does more than just build an image, and it is already
integrated in the gate. Why not use that?​

Is your question why don't we use trovestack? If so, we were looking
for
something that was more streamlined with fewer dependencies. The tool
would need to interface with the client's cloud. Our tool is directed at
customers and IBM lab based services, neither of whom is expected to be
an OpenStack or Trove expert.

Beyond that, there is a difference in how are images our built. First,
we need
to control which versions of databases are built as opposed to letting the
stack
determine that. ppc64le support in the database communities is at
varying
stages. In some cases, there are packages and in others there are not.
In
addition, there are specific versions that we have tested and enhanced
from
a performance perspective that we would like to include in our offering.

The second part of this is related to the Trove Guest Agent. We needed
to
make a few code changes to support different versions and to fix bugs that
we have found. Anyway, this led to us carrying patches in the tool,
which is
part of a broader strategy to stage our database deliverables. We would
like to add support for new databases over time. To deliver these to our
clients without them having to re-install Trove on the controller side.
Fundamentally, this means isolating the database specific log to the Trove
Guest Agent and Trove elements.

So, this is a rather long explanation, but that is why we couldn't use
Trovestack
or the existing tools as is. There are external considerations such as
the
state of the database communities and delivery strategies that warranted
a slightly different approach.


This approach is nice (having a full guest image with the guest agent
and
all that but as you observe, it takes about 2 hours per build and from a
developers perspective, this is a pain.

It takes 2 hours to build mongodb, because there is no package provided
by the community for ppc64le. In general, it takes 25 minutes to
generate
images and create the datastore. I expect that the times are roughly
the
same as with the existing tools.

I assume that the tool is therefore for production use cases.

Our offering is intended for development use only, but it is a big step
in the direction of production use cases. The images are built for
specific
users in a more prescribed manner.

Could the elements that you have developed be used with the existing
tool
or is your tool a complete replacement for the one that already
exists?​

With a little bit of work they may be used with the existing tool. I am
not
planning on replacing existing tools. In fact, I wasn't planning on
dropping
the new tool but I would if there is interest. The new tool is at a
public
which I noted in my original note.

​Is there some reason that this tool was not developed in the open,
using
the normal open development process?

We were working towards a short release schedule for our new OpenDBaaS
offering.
It took three months to develop the tool and provide support for 5
databases,
including multiple versions of some of them. This is our second step
to engage with the community.

There are ways to accelerate the compiling of source code and such
through
the use of image layers (if you are using DIB). I have elements and code
that will build most of the trove elements in minutes and I'm waiting to
see whether there is any point in contributing them to the upstream or
not,
at this point. I could show you how to do that, and it would be awesome
to
have you build your tool into the build process somehow. That would be a
win-win for all ...​

I also have commands that setup Nova flavors on a per database basis.

These flavors can be customized by the user before they are
instantiated in
the cloud.That's good to hear.

​The existing command (trovestack) already does this.​

I didn't see that in the Trovestack. Does it allow the user to customize
the values. Can you provide a provide a link?

I wouldn't expect to drop these commands to the Trove community. Only
the
changes to the Trove-guestagent and elements enabling databases.

Here's the list of databases that are currently supported for ppc64le.
In
some cases, I have had to add Xenial support.

mariadb 10.1 --- package comes from the community
mongodb 3.4 community edition --- source code from the community is
compiled as there is no ppc64le package
mongodb 3.4 enterprise edition --- package comes from community
mysql 5.7 --- package comes from Ubuntu 16.04. percona-xtrabackup-24 is
compiled as there is no power support in community
postgresql 9.5 --- package comes from Ubuntu 16.04
postgresql 9.6 --- package comes from community
redis 3.0 --- package comes from Ubuntu 16.04
redis 3.2 --- source code from the community is compiled

​Why not just contribute the elements to the existing project and have
the
current tool use them?​

That is my plan.

​What of other databases currently supported? galera cluster (pxc),
vertica, db2, percona, couchdb, and couchbase?

I haven't messed with them. My changes are hidden under environment
variables so the absence of environment variables should yield the old
code.

My element related code changes add environment variables that control
the database version, the source of the database software (distro,
community, enterprise), and logging options. In this way, the logic in
the
element is more generalized and can be made to support multiple
versions. I
also added an environment variable that specifies the name of the
source
package to be downloaded and compiled when that is applicable. I also
have
environment variables to enable database and trove guest agent logging.

​Which is nice except that your guest agent code may or may not know
how
to deal with that.​

The guest agent and image logic are coordinated.

My trove-guestagent code changes are mostly about not reconfiguring the
database with configuration data remotely passed to it from the task
manager. My code changes are isolated to the prepare step. This logic
looks
something like: if database version x, add or remove configuration
parameter. Note there is relatively very little of this code, ~150
LOCs, as
database configuration parameters rarely change. Note there is already
some
of this DB version specific logic in the trove guest agent, so I am
just
adding more of it. But I believe a better way is not to alter the
database
configuration data when the database is fully configured in a Trove
compliant manner via elements. This would be implemented through a new
command argument to the Trove Guest agent which says ignore
configuration
changes. It would be set by the elements as well.

​Hard to say whether this is a good approach or not with out more
detail.​

I haven't looked into the scale out configuration data yet, but this
can
be handled differently as required.

​Not sure what you mean.

I was talking about the database cluster configuration code which is a
future
deliverable for our offering. We are starting simple and will
incrementally expand the
offering as time goes by.

So, are these the sorts of changes that I could add in a maintenance
release?

​Maintenance release of what, trove? Are you asking whether these
changes
can be made in a stable branch? if yes, which branch? See
https://docs.openstack.org/project-team-guide/stable-branches.html and
the section "Appropriate Fixes".​

1) conditionally disable the dynamic installation and configuration of
database software in the trove-guest agent via a new command argument
2) add environment variables to trove elements enabling multiple
versions
of databases to be installed and configured
3) add environment variables to trove elements to enable database and
trove-guest agent logging
4) add xenial support to elements
5) compiling source code in elements for missing stack elements for
ppc64le

These changes are implemented at: https://github.com/open-power-
ref-design-toolkit/os-services/tree/master/osa/dbaas/dbimage-builder

​OK, so this is a whole new tool, sure you can propose it but how do we
make sure that it gets integrated into the current testing/gate? If you
want to drop this then I'd also have to assume that there would be some
code that would make it part of the current gate, and ideally make it
generate the images for the current gate. But for that, the fact that
you
install the guest agent onto the image is a serious drag.

Also, why this wasn't proposed as a blueprint or spec in the upstream so
it could be developed in the open, I don't quite understand. After all,
you
are now showing up with a complete tool and asking that it be included
which largely runs counter to the whole notion of open development, at
least as I understand it.​

Above, in my original node, I said that I was only planning on dropping
code
changes to the element and that I had a small design change to the guest
agent
that I would pursue with the community. I am not proposing to drop the
tool, unless there was interest in that from the community.


Finally, once committed, do you (or IBM) commit to maintaining this code
and fixing bugs in it should they arise, and upgrade/update it from time
to
time?

Yes

Thanks, Luke Browning


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 31, 2017 by Luke_Browning (140 points)  
0 votes

Luke, please see inline below.

--
Amrith Kumar
mailto:amrith.kumar@gmail.com

From: Luke Browning [mailto:LukeBrowning@us.ibm.com]
Sent: Thursday, August 31, 2017 1:00 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] Future Trove development

Amrith,

See answers to your questions below.

Cheers,
Luke

I understand that the Trove project will be going into maintenance mode
soon. I have a number of Trove related enhancements and bug fixes that I
would like to submit to the community, but I don't know if they would be
considered given that the project is going into maintenance mode. Please
elaborate on what you mean by maintenance mode.

See: https://review.openstack.org/#/c/488947/

It is not a foregone conclusion that the project will go into maintenance
mode. Thierry and I (for example) are not convinced that this is the right
course of action, but if there are no offers to contribute to the project,
it may be a decision by default.

Do you subscribe to the OpenStack-dev mailing list. Please also see
http://openstack.markmail.org/thread/4p6gp263fei4mbru

Finally, for a description of what maintenance mode means, please refer:
https://governance.openstack.org/tc/reference/tags/status_ma
intenance-mode.html

Would Trove be updated to support new OpenStack versions?

Would support be provided for Trove gate testing?
>
Would elements be enhanced to support Xenial and newer versions of

databases?

Is there a time limit associated with maintenance mode? For example, would

it remain in effect for a year or two after the new stuff is released?

For the four questions above, I direct your attention to the definition
of the maintenance-mode tag (https://governance.openstack/.
org/tc/reference/tags/status_maintenance-mode.html) and remind you that
support for different versions from the community, for gate testing and for
elements is dependent on people participating (and their employers allowing
them to do this). At this point, I have no one who has stepped up to do
this in any significant way. There are a couple of people from China Mobile
who want to help but who are largely in the weeds, trying to fix this and
that bug but have no idea what Trove is all about. IBM has a core reviewer
on the project (Mariam is copied on this email). I am happy that she's
able to devote what time she can to the project but absent competent
reviewers, whether your changes ever get merged or not remains an
interesting question. Since you will be contributing elements and propose
to contribute a tool, will you be (or will IBM be) offering to support it?

Let me provide some back ground on my code changes, so you will have a
better idea of what might be submitted in a patch.

I have written a fully automated command that creates a virtual guest
image based on Ubuntu 16.04 containing a user specified version of a
database. The user's selection is validated against an internal list of
databases built into the tool. Once validated, the tool creates the image,
registers it with Glance, and then creates a Trove Datastore definition for
it. This is done in a single command invocation that typically takes about
25 minutes to complete. For some of the databases, it takes considerably
longer (2 hours for mongodb) as I have to compile source code. The guest
image that is produced is complete from a Trove perspective. It includes
the Cloud specific trove-guestagent.conf file and SSH public keys for the
DBA and OpenStack controller nodes.

What mechanism does it use to develop the image? is it diskimage-builder
or some other?

Yes, diskimage-builder and Trove elements

Is there something the matter with the existing tool that exists, it
already works, it does more than just build an image, and it is already
integrated in the gate. Why not use that?

I should have said this before (like in the last reply) and I may not have, but I am broadly in favor of the approach you proposed (I did look briefly at the repository that you provided). The place(s) where I had reservations relate primarily to how this new tool would interface/interact with the existing Trove. More details about that below.

Is your question why don't we use trovestack? If so, we were looking for
something that was more streamlined with fewer dependencies. The tool
would need to interface with the client's cloud. Our tool is directed at
customers and IBM lab based services, neither of whom is expected to be
an OpenStack or Trove expert.

The guts of image building in trovestack are easy to abstract out. There is a trovestack command (image-build) which is a code path that can be easily leveraged for this. The reason I bring this up is that I would much rather this new tool, and the elements it comes with be the thing that is actively tested by the CI each time a Trove build is tested. I would therefore, much rather that the existing image building stuff be somehow integrated with the new tool that you have created, and the old stuff can be junked.

Beyond that, there is a difference in how are images our built. First, we need
to control which versions of databases are built as opposed to letting the stack
determine that. ppc64le support in the database communities is at varying
stages. In some cases, there are packages and in others there are not. In
addition, there are specific versions that we have tested and enhanced from
a performance perspective that we would like to include in our offering.

Makes perfect sense.

The second part of this is related to the Trove Guest Agent. We needed to
make a few code changes to support different versions and to fix bugs that
we have found. Anyway, this led to us carrying patches in the tool, which is
part of a broader strategy to stage our database deliverables. We would
like to add support for new databases over time. To deliver these to our
clients without them having to re-install Trove on the controller side.
Fundamentally, this means isolating the database specific log to the Trove
Guest Agent and Trove elements.

I don't completely understand the issue with isolating log files but I would much rather see the fixes you have incorporated into the upstream, and not carried in your elements.

So, this is a rather long explanation, but that is why we couldn't use Trovestack
or the existing tools as is. There are external considerations such as the
state of the database communities and delivery strategies that warranted
a slightly different approach.

Makes sense, why not make that the thing that Trove ships instead of creating a support nightmare with a Trove way and the IBM tool way?

This approach is nice (having a full guest image with the guest agent and
all that but as you observe, it takes about 2 hours per build and from a
developers perspective, this is a pain.

It takes 2 hours to build mongodb, because there is no package provided
by the community for ppc64le. In general, it takes 25 minutes to generate
images and create the datastore. I expect that the times are roughly the
same as with the existing tools.

So your tool will likely not run in the CI as it will timeout. What about building and caching images? Also from the perspective of your users, they would much rather have a tool that finishes in 15 minutes than 2 hours so you may benefit from refactoring your elements to create an image which is all-but-guest-agent, and then have a second set of elements which takes that image as an input (not a cloud image) and just slaps the guest agent into it and does the required configurations. Happy to help with that, I've been working on this for trovestack anyway.

I assume that the tool is therefore for production use cases.

Our offering is intended for development use only, but it is a big step
in the direction of production use cases. The images are built for specific
users in a more prescribed manner.

So, why not make that the default tool chain for trove?

Could the elements that you have developed be used with the existing tool
or is your tool a complete replacement for the one that already exists?

With a little bit of work they may be used with the existing tool. I am not
planning on replacing existing tools. In fact, I wasn't planning on dropping
the new tool but I would if there is interest. The new tool is at a public
which I noted in my original note.

I don't understand this comment.

Is there some reason that this tool was not developed in the open, using
the normal open development process?

We were working towards a short release schedule for our new OpenDBaaS offering.
It took three months to develop the tool and provide support for 5 databases,
including multiple versions of some of them. This is our second step
to engage with the community.

There are ways to accelerate the compiling of source code and such through
the use of image layers (if you are using DIB). I have elements and code
that will build most of the trove elements in minutes and I'm waiting to
see whether there is any point in contributing them to the upstream or not,
at this point. I could show you how to do that, and it would be awesome to
have you build your tool into the build process somehow. That would be a
win-win for all ...

I also have commands that setup Nova flavors on a per database basis.

These flavors can be customized by the user before they are instantiated in
the cloud.That's good to hear.

The existing command (trovestack) already does this.

I didn't see that in the Trovestack. Does it allow the user to customize
the values. Can you provide a provide a link?

trovestack image-build (or build-image) is the command, if you follow that code path it will just setup environment variables (used by the elements) and run dib.

I wouldn't expect to drop these commands to the Trove community. Only the
changes to the Trove-guestagent and elements enabling databases.

Here's the list of databases that are currently supported for ppc64le. In
some cases, I have had to add Xenial support.

mariadb 10.1 --- package comes from the community
mongodb 3.4 community edition --- source code from the community is
compiled as there is no ppc64le package
mongodb 3.4 enterprise edition --- package comes from community
mysql 5.7 --- package comes from Ubuntu 16.04. percona-xtrabackup-24 is
compiled as there is no power support in community
postgresql 9.5 --- package comes from Ubuntu 16.04
postgresql 9.6 --- package comes from community
redis 3.0 --- package comes from Ubuntu 16.04
redis 3.2 --- source code from the community is compiled

Why not just contribute the elements to the existing project and have the
current tool use them?

That is my plan.

Wait, you will have the existing tool use your elements? That's awesome.

What of other databases currently supported? galera cluster (pxc),
vertica, db2, percona, couchdb, and couchbase?

I haven't messed with them. My changes are hidden under environment
variables so the absence of environment variables should yield the old code.

Sounds like a support challenge in the making.

My element related code changes add environment variables that control
the database version, the source of the database software (distro,
community, enterprise), and logging options. In this way, the logic in the
element is more generalized and can be made to support multiple versions. I
also added an environment variable that specifies the name of the source
package to be downloaded and compiled when that is applicable. I also have
environment variables to enable database and trove guest agent logging.

Which is nice except that your guest agent code may or may not know how
to deal with that.

The guest agent and image logic are coordinated.

My trove-guestagent code changes are mostly about not reconfiguring the
database with configuration data remotely passed to it from the task
manager. My code changes are isolated to the prepare step. This logic looks
something like: if database version x, add or remove configuration
parameter. Note there is relatively very little of this code, ~150 LOCs, as
database configuration parameters rarely change. Note there is already some
of this DB version specific logic in the trove guest agent, so I am just
adding more of it. But I believe a better way is not to alter the database
configuration data when the database is fully configured in a Trove
compliant manner via elements. This would be implemented through a new
command argument to the Trove Guest agent which says ignore configuration
changes. It would be set by the elements as well.

Hard to say whether this is a good approach or not with out more detail.

I haven't looked into the scale out configuration data yet, but this can
be handled differently as required.

Not sure what you mean.

I was talking about the database cluster configuration code which is a future
deliverable for our offering. We are starting simple and will incrementally expand the
offering as time goes by.

So, are these the sorts of changes that I could add in a maintenance
release?

Maintenance release of what, trove? Are you asking whether these changes
can be made in a stable branch? if yes, which branch? See
https://docs.openstack.org/project-team-guide/stable-branches.htmland
the section "Appropriate Fixes".

1) conditionally disable the dynamic installation and configuration of
database software in the trove-guest agent via a new command argument
2) add environment variables to trove elements enabling multiple versions
of databases to be installed and configured
3) add environment variables to trove elements to enable database and
trove-guest agent logging
4) add xenial support to elements
5) compiling source code in elements for missing stack elements for
ppc64le

These changes are implemented at: https://github.com/open-power-
ref-design-toolkit/os-services/tree/master/osa/dbaas/dbimage-builder

OK, so this is a whole new tool, sure you can propose it but how do we
make sure that it gets integrated into the current testing/gate? If you
want to drop this then I'd also have to assume that there would be some
code that would make it part of the current gate, and ideally make it
generate the images for the current gate. But for that, the fact that you
install the guest agent onto the image is a serious drag.

Also, why this wasn't proposed as a blueprint or spec in the upstream so
it could be developed in the open, I don't quite understand. After all, you
are now showing up with a complete tool and asking that it be included
which largely runs counter to the whole notion of open development, at
least as I understand it.

Above, in my original node, I said that I was only planning on dropping code
changes to the element and that I had a small design change to the guest agent
that I would pursue with the community. I am not proposing to drop the
tool, unless there was interest in that from the community.

Finally, once committed, do you (or IBM) commit to maintaining this code
and fixing bugs in it should they arise, and upgrade/update it from time to
time?

Yes

Thanks, Luke Browning


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 3, 2017 by amrith.kumar_at_gmai (3,580 points)   2 2
...