settingsLogin | Registersettings

[openstack-dev] conflicting names in python-openstackclient: could we have some exception handling please?

0 votes

Hi,

tl;dr: let's add a exception handling so that python-*client having
conflicting command names isn't a problem anymore, and "openstack help"
always work as much as it can.

Longer version:

This is just a suggestion for contributors to python-openstackclient.

I saw a few packages that had conflicts with the namespace of others
within openstackclient. To the point that typing "openstack help" just
fails. Here's an example:

openstack help

[ ...]
project create Create new project
project delete Delete project(s)
project list List projects
project set Set project properties
project show Display project details
Could not load EntryPoint.parse('ptrrecordlist =
designateclient.v2.cli.reverse:ListFloatingIPCommand')
'ArgumentParser' object has no attribute 'debug'

This first happened to me with saharaclient. Lucky, upgrading to latest
version fixed it. Then I had the problem with zaqarclient, which I fixed
with a few patches to its setup.cfg. Then now designate, but this time,
patching setup.cfg doesn't seem to cut it (ie: after changing the name
of the command, "openstack help" just fails).

Note: I don't care which project is at fault, this isn't the point here.
The point is that command name conflicts aren't handled (see below)
which is the problem.

With Horizon being a large consumer of nearly all python-*client
packages, removing one of them also removes Horizon in my CI which is
not what I want to (or can) do to debug a tempest problem. End of the
story: since Liberty b3, I never could have "openstack help" to work
correctly in my CI... :(

Which leads me to write this:

Since we have a very large amount of projects, with each and everyone of
them adding new commands to openstackclient, I would really nice if we
could have some kind of checks to make sure that conflicts are either 1/
not possible or 2/ handled gracefully.

Your thoughts?
Cheers,

Thomas Goirand (zigo)

P.S: It wasn't the point of this message, but do we have a fix for
designateclient? It'd be nice to have this fixed before Liberty is out.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 6, 2015 in openstack-dev by Thomas_Goirand (18,640 points)   2 10 16

7 Responses

0 votes

On 10/06/2015 05:15 PM, Thomas Goirand wrote:
Hi,

tl;dr: let's add a exception handling so that python-*client having
conflicting command names isn't a problem anymore, and "openstack help"
always work as much as it can.

Standardizing on "openstack <noun[s]> verb" would likely be
the best solution for both the immediate problem and for the broader
"naming stuff" issue.

Sharing a flat namespace is a recipe for pain with a growing number of
projects. Devs and users are unlikely to use every project, they
probably won't notice conflicts naturally except in cases like horizon.

If we look over the fence at AWS, you'll note that their nice unified
CLI that stops the non-uniform awk bloodshed is namespaced.

  • aws s3 ...
  • aws cloudformation ...
  • aws ec2 ...

A flat namespace was a mostly-fine idea when all integrated projects
were expected to put their CLI in-tree in openstackclient. There were
reviews, and discussions about what noun belonged to whom.

Noun conflict will only get worse: lots of projects will share words
like stack, domain, user, container, address, and so on.

Namespaces are one honking great idea -- let's do more of those!

Longer version:

This is just a suggestion for contributors to python-openstackclient.

I saw a few packages that had conflicts with the namespace of others
within openstackclient. To the point that typing "openstack help" just
fails. Here's an example:

openstack help

[ ...]
project create Create new project
project delete Delete project(s)
project list List projects
project set Set project properties
project show Display project details
Could not load EntryPoint.parse('ptrrecordlist =
designateclient.v2.cli.reverse:ListFloatingIPCommand')
'ArgumentParser' object has no attribute 'debug'

This first happened to me with saharaclient. Lucky, upgrading to latest
version fixed it. Then I had the problem with zaqarclient, which I fixed
with a few patches to its setup.cfg. Then now designate, but this time,
patching setup.cfg doesn't seem to cut it (ie: after changing the name
of the command, "openstack help" just fails).

Note: I don't care which project is at fault, this isn't the point here.
The point is that command name conflicts aren't handled (see below)
which is the problem.

+1, this isn't a problem specific to any project, it's systemic with
flat namespacing.

With Horizon being a large consumer of nearly all python-*client
packages, removing one of them also removes Horizon in my CI which is
not what I want to (or can) do to debug a tempest problem. End of the
story: since Liberty b3, I never could have "openstack help" to work
correctly in my CI... :(

O.O That's unfortunate.

Which leads me to write this:

Since we have a very large amount of projects, with each and everyone of
them adding new commands to openstackclient, I would really nice if we
could have some kind of checks to make sure that conflicts are either 1/
not possible or 2/ handled gracefully.

To your (1) we could have a gate job that installs all the clients and
fails on conflicts.

The downside of doing that without addressing the namespace problem is
that there will be inconsistent conventions everywhere. Zaqar will have
"openstack queue ...." but "openstack message flavor ..." which creates
the sort of confusion a unified client is supposed to avoid.

A central reservation process for nouns won't really scale, but
namespacing will because we already namespace projects.

Your thoughts?
Cheers,

Thomas Goirand (zigo)

P.S: It wasn't the point of this message, but do we have a fix for
designateclient? It'd be nice to have this fixed before Liberty is out.

--
Ryan Brown / Senior Software Engineer, Openstack / Red Hat, Inc.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 7, 2015 by Ryan_Brown (3,520 points)   2 2
0 votes

On 07/10/15 14:42, Ryan Brown wrote:
On 10/06/2015 05:15 PM, Thomas Goirand wrote:

Hi,

tl;dr: let's add a exception handling so that python-*client having
conflicting command names isn't a problem anymore, and "openstack help"
always work as much as it can.

Standardizing on "openstack <noun[s]> verb" would likely be
the best solution for both the immediate problem and for the broader
"naming stuff" issue.

Sharing a flat namespace is a recipe for pain with a growing number of
projects. Devs and users are unlikely to use every project, they
probably won't notice conflicts naturally except in cases like horizon.

If we look over the fence at AWS, you'll note that their nice unified
CLI that stops the non-uniform awk bloodshed is namespaced.

  • aws s3 ...
  • aws cloudformation ...
  • aws ec2 ...

A flat namespace was a mostly-fine idea when all integrated projects
were expected to put their CLI in-tree in openstackclient. There were
reviews, and discussions about what noun belonged to whom.

Noun conflict will only get worse: lots of projects will share words
like stack, domain, user, container, address, and so on.

Namespaces are one honking great idea -- let's do more of those!

Longer version:

This is just a suggestion for contributors to python-openstackclient.

I saw a few packages that had conflicts with the namespace of others
within openstackclient. To the point that typing "openstack help" just
fails. Here's an example:

openstack help

[ ...]
project create Create new project
project delete Delete project(s)
project list List projects
project set Set project properties
project show Display project details
Could not load EntryPoint.parse('ptrrecordlist =
designateclient.v2.cli.reverse:ListFloatingIPCommand')
'ArgumentParser' object has no attribute 'debug'

This first happened to me with saharaclient. Lucky, upgrading to latest
version fixed it. Then I had the problem with zaqarclient, which I fixed
with a few patches to its setup.cfg. Then now designate, but this time,
patching setup.cfg doesn't seem to cut it (ie: after changing the name
of the command, "openstack help" just fails).

Note: I don't care which project is at fault, this isn't the point here.
The point is that command name conflicts aren't handled (see below)
which is the problem.

+1, this isn't a problem specific to any project, it's systemic with
flat namespacing.

With Horizon being a large consumer of nearly all python-*client
packages, removing one of them also removes Horizon in my CI which is
not what I want to (or can) do to debug a tempest problem. End of the
story: since Liberty b3, I never could have "openstack help" to work
correctly in my CI... :(

O.O That's unfortunate.

Which leads me to write this:

Since we have a very large amount of projects, with each and everyone of
them adding new commands to openstackclient, I would really nice if we
could have some kind of checks to make sure that conflicts are either 1/
not possible or 2/ handled gracefully.

To your (1) we could have a gate job that installs all the clients and
fails on conflicts.

The downside of doing that without addressing the namespace problem is
that there will be inconsistent conventions everywhere. Zaqar will have
"openstack queue ...." but "openstack message flavor ..." which creates
the sort of confusion a unified client is supposed to avoid.

A central reservation process for nouns won't really scale, but
namespacing will because we already namespace projects.

Your thoughts?
Cheers,

Thomas Goirand (zigo)

P.S: It wasn't the point of this message, but do we have a fix for
designateclient? It'd be nice to have this fixed before Liberty is out.

Is there a bug filed for it? I haven't seen this before and it seems to
be working for me locally :/

(I have openstackclient == 1.7.1 & designateclient == 1.5.0)

If we can find the issue we will try and get a fix out.

--
Graham Hayes
Software Engineer
DNS as a Service
Advanced Network Services
HP Helion Cloud - Platform Services

GPG Key: 7D28E972

graham.hayes@hpe.com
M +353 87 377 8315

P +353 1 525 1589
Dublin,
Ireland

HP


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 7, 2015 by graham.hayes_at_hpe. (6,780 points)   1 2 2
0 votes

On Tue, Oct 6, 2015 at 4:15 PM, Thomas Goirand zigo@debian.org wrote:

tl;dr: let's add a exception handling so that python-*client having
conflicting command names isn't a problem anymore, and "openstack help"
always work as much as it can.

This creates a first-one-wins scenario that requires *-client install order
to be deterministic. Or we need a registry of which API owns which object
namens (nouns), which would actually solve our problem today without any
changes.

Longer version:

This is just a suggestion for contributors to python-openstackclient.

I saw a few packages that had conflicts with the namespace of others
within openstackclient. To the point that typing "openstack help" just
fails. Here's an example:

openstack help

[ ...]
project create Create new project
project delete Delete project(s)
project list List projects
project set Set project properties
project show Display project details
Could not load EntryPoint.parse('ptrrecordlist =
designateclient.v2.cli.reverse:ListFloatingIPCommand')
'ArgumentParser' object has no attribute 'debug'

This first happened to me with saharaclient. Lucky, upgrading to latest
version fixed it. Then I had the problem with zaqarclient, which I fixed
with a few patches to its setup.cfg. Then now designate, but this time,
patching setup.cfg doesn't seem to cut it (ie: after changing the name
of the command, "openstack help" just fails).

This is a problem with the implementation of the plugin, not with the
selection of command names. Your experience with saharaclient and the fact
that editing setup.cfg did not help should confirm that.

Which leads me to write this:

Since we have a very large amount of projects, with each and everyone of
them adding new commands to openstackclient, I would really nice if we
could have some kind of checks to make sure that conflicts are either 1/
not possible or 2/ handled gracefully.

The conflicting command names is due to the way setuptools handles entry
points by default (I'm not sure if this can even be changed easily). It
turns out that having multiple classes registered to a common command name
is useful, I have a plugin that does this intentionally in order to modify
the behaviour of an existing command (changes server create to add --ram,
--disk and --cpu to auto-select a flavor).

dt

--

Dean Troyer
dtroyer@gmail.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 7, 2015 by Dean_Troyer (13,100 points)   1 3 3
0 votes

On Wed, Oct 7, 2015 at 8:39 AM, Ryan Brown rybrown@redhat.com wrote:

Standardizing on "openstack <noun[s]> verb" would likely be the
best solution for both the immediate problem and for the broader "naming
stuff" issue.

This is the approach that a number of plugins are taking. I have STRONGLY
recommended that the project name not be used, OSC never uses project names
in a user-visible way. Some plugins do it anyway.

A flat namespace was a mostly-fine idea when all integrated projects were
expected to put their CLI in-tree in openstackclient. There were reviews,
and discussions about what noun belonged to whom.

I wish there had been more of that. You'd be surprised how little there
actually has been...

Noun conflict will only get worse: lots of projects will share words like
stack, domain, user, container, address, and so on.

This is one reason multiple-word object names are possible. For example,
we've just merged commands to support the object store 'account' functions
and used 'object store account' for those commands because we felt that
'account' is too generic.

A central reservation process for nouns won't really scale, but
namespacing will because we already namespace projects.

I believe it will scale far enough to encomapss the realistic possibilites
of putting everything under the top-level 'openstack' command. I think
that is the assumption that we need to be addressing. As you said, it
worked when we started 4 years ago, but even then there were conflicts. We
are discussing alternatives that include multiple top-level commands. But
we need to get input from actual users who are not OpenStack devs on this
and do not have the pre-concieved notions of this-api and that-api to the
extent that those of us in the project do. This is just getting underway,
and will be a summit topic.

Users should not have to know or care which API implements something.

I am beginning to regret giving up the command name control as it is
re-creating the situation we had with the original CLIs and the lack of
consistency.

Also I want to add that the philosophy of OpenStackClient is not to simply
accept everything that the API declares as the CLI and stop there, but to
make the operations simple and meaningful for the users. We have
abstracted some API bits and flat out changed others to do this.

dt

"The 'C' in OSC == 'consistent'."

--

Dean Troyer
dtroyer@gmail.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 7, 2015 by Dean_Troyer (13,100 points)   1 3 3
0 votes

On 10/07/2015 03:39 PM, Ryan Brown wrote:
Sharing a flat namespace is a recipe for pain with a growing number of
projects. Devs and users are unlikely to use every project, they
probably won't notice conflicts naturally except in cases like horizon.

Well, users would typically install Horizon on a "controler" machine,
with everything installed there, and then specialized compute, network
or storage node. This is a very common practice. In such a case, if you
want to debug anything, you'd use python-openstackclient cli on the
controler machine, and probably at some point will want to type
"openstack help". So very likely, this will hit our users.

To your (1) we could have a gate job that installs all the clients and
fails on conflicts.

That's a very good idea.

Thomas


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 7, 2015 by Thomas_Goirand (18,640 points)   2 10 16
0 votes

On 10/07/2015 03:57 PM, Hayes, Graham wrote:
On 07/10/15 14:42, Ryan Brown wrote:

On 10/06/2015 05:15 PM, Thomas Goirand wrote:

P.S: It wasn't the point of this message, but do we have a fix for
designateclient? It'd be nice to have this fixed before Liberty is out.

Is there a bug filed for it? I haven't seen this before and it seems to
be working for me locally :/

(I have openstackclient == 1.7.1 & designateclient == 1.5.0)

If we can find the issue we will try and get a fix out.

Here you are...
https://bugs.launchpad.net/python-designateclient/+bug/1503736

Cheers,

Thomas

P.S: That's a "normal" Debian Jessie machine with the latest Liberty
packages backported-built and available at (deb repo details there...):
http://liberty-jessie.pkgs.mirantis.com/


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 7, 2015 by Thomas_Goirand (18,640 points)   2 10 16
0 votes

On 10/07/2015 10:50 AM, Dean Troyer wrote:
On Wed, Oct 7, 2015 at 8:39 AM, Ryan Brown <rybrown@redhat.com
rybrown@redhat.com> wrote:

Standardizing on "openstack <project> <noun[s]> verb" would likely
be the best solution for both the immediate problem and for the
broader "naming stuff" issue.

This is the approach that a number of plugins are taking. I have
STRONGLY recommended that the project name not be used, OSC never uses
project names in a user-visible way. Some plugins do it anyway.

A flat namespace was a mostly-fine idea when all integrated projects
were expected to put their CLI in-tree in openstackclient. There
were reviews, and discussions about what noun belonged to whom.

I wish there had been more of that. You'd be surprised how little there
actually has been...

Noun conflict will only get worse: lots of projects will share words
like stack, domain, user, container, address, and so on.

This is one reason multiple-word object names are possible. For
example, we've just merged commands to support the object store
'account' functions and used 'object store account' for those commands
because we felt that 'account' is too generic.

A central reservation process for nouns won't really scale, but
namespacing will because we *already* namespace projects.

I believe it will scale far enough to encomapss the realistic
possibilites of putting everything under the top-level 'openstack'
command. I think that is the assumption that we need to be addressing.
As you said, it worked when we started 4 years ago, but even then there
were conflicts. We are discussing alternatives that include multiple
top-level commands. But we need to get input from actual users who are
not OpenStack devs on this and do not have the pre-concieved notions of
this-api and that-api to the extent that those of us in the project do.
This is just getting underway, and will be a summit topic.

Users should not have to know or care which API implements something.

For the CLI, definitely agree. Balkanizing back to a namespace per
project is the wrong direction.

I am beginning to regret giving up the command name control as it is
re-creating the situation we had with the original CLIs and the lack of
consistency.

Also I want to add that the philosophy of OpenStackClient is not to
simply accept everything that the API declares as the CLI and stop
there, but to make the operations simple and meaningful for the users.
We have abstracted some API bits and flat out changed others to do this.

dt

"The 'C' in OSC == 'consistent'."

--

Dean Troyer
dtroyer@gmail.com dtroyer@gmail.com


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 7, 2015 by Sean_Dague (66,200 points)   4 8 14
...