settingsLogin | Registersettings

Re: [openstack-dev] [magnum] About clean none use container image

0 votes

@Jay Lau I agree with you that image pull be time-consuming.
Did is much better to supply a magnum api let end-user can clean image that not used?
Clean or not clean is decide by the end-user. Or we can supply a method let
end-user can decide whether or not clean image.
------------------ Original ------------------
From: "openstack-dev-request";openstack-dev-request@lists.openstack.org;
Date: Mon, Apr 13, 2015 04:59 PM
To: "openstack-dev"openstack-dev@lists.openstack.org;

Subject: OpenStack-dev Digest, Vol 36, Issue 32

Send OpenStack-dev mailing list submissions to
openstack-dev@lists.openstack.org

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
or, via email, send a message with subject or body 'help' to
openstack-dev-request@lists.openstack.org

You can reach the person managing the list at
openstack-dev-owner@lists.openstack.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of OpenStack-dev digest..."

Today's Topics:

  1. Re: [Nova] VMware CI (Davanum Srinivas)
  2. Re: [Nova] VMware CI (Gary Kotton)
  3. Re: [TripleO] Consistent variable documentation for
    diskimage-builder elements (Gregory Haynes)
  4. Re: [neutron] Neutron scaling datapoints? (Joshua Harlow)
  5. Re: [OpenStack-docs] What's Up Doc? Apr 10 2015 (Monty Taylor)
  6. Re: [neutron] Neutron scaling datapoints? (Kevin Benton)
  7. Re: [neutron] Neutron scaling datapoints? (Kevin Benton)
  8. [all][pbr] splitting our deployment vs install dependencies
    (Robert Collins)
  9. Re: [all][pbr] splitting our deployment vs install
    dependencies (Monty Taylor)

    1. Re: [all][pbr] splitting our deployment vs install
      dependencies (James Polley)
    2. Re: [all][pbr] splitting our deployment vs install
      dependencies (Monty Taylor)
    3. Re: [all][pbr] splitting our deployment vs install
      dependencies (Robert Collins)
    4. Re: [all][pbr] splitting our deployment vs install
      dependencies (Robert Collins)
    5. [all] Problems with keystoneclient stable branch (and maybe
      yours too) (Brant Knudson)
    6. Re: In loving memory of Chris Yeoh (Michael Still)
    7. Re: [all][pbr] splitting our deployment vs install
      dependencies (Robert Collins)
    8. Re: [neutron] Neutron scaling datapoints? (joehuang)
    9. Re: [neutron] Neutron scaling datapoints? (Joshua Harlow)
    10. Re: [neutron] Neutron scaling datapoints? (Joshua Harlow)
    11. Re: [neutron] Neutron scaling datapoints? (Joshua Harlow)
    12. Re: [puppet] Puppet PTL (Emilien Macchi)
    13. Re: In loving memory of Chris Yeoh (Gary Kotton)
    14. Re: [neutron] Neutron scaling datapoints? (Attila Fazekas)
    15. Re: [Neutron] Regarding neutron bug # 1432582 (Kevin Benton)
    16. ?magnum?About clean none use container imag
      (=?gb18030?B?NDQ5MTcxMzQy?=)
    17. Re: ?magnum?About clean none use container imag (Jay Lau)
    18. Re: [neutron] Neutron scaling datapoints? (Attila Fazekas)
    19. Re: In loving memory of Chris Yeoh (wu jiang)
    20. Re: [oslo] eventlet 0.17.3 is now fully Python 3 compatible
      (Victor Stinner)
    21. [Fuel] Nominate Andrey Skedzinskiy for fuel-qa(devops) core
      (Anastasia Urlapova)
    22. Re: [Fuel] Nominate Andrey Skedzinskiy for fuel-qa(devops)
      core (Alexander Kislitsky)
    23. Re: [all][pbr] splitting our deployment vs install
      dependencies (Miguel Angel Ajo Pelayo)
    24. ??: [neutron] Neutron scaling datapoints? (Wangbibo)

Message: 1
Date: Sun, 12 Apr 2015 08:04:43 -0400
From: Davanum Srinivas davanum@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Nova] VMware CI
Message-ID:
CANw6fcHxxWT215_AtJeX5LjvwnMrTvGpqXXuGi=oFj9prw_SyA@mail.gmail.com
Content-Type: text/plain; charset=UTF-8

Gary, John,

Just to speed things up, i filed a backport:
https://review.openstack.org/#/c/172710/

thanks,
dims

On Sun, Apr 12, 2015 at 4:23 AM, John Garbutt john@johngarbutt.com wrote:

I have updated the bug so it's high priority and tagged with
kilo-rc-potential, and added your note from below as a comment on the bug.

It looks like it might be worth a backport so it gets into RC2? Can anyone
take that bit on please?

Thanks,
John

On Sunday, April 12, 2015, Gary Kotton gkotton@vmware.com wrote:

Hi,
Can a core please take a look at https://review.openstack.org/#/c/171037.
The CI is broken due to commit e7ae5bb7fbdd5b79bde8937958dd0a645554a5f0.
Thanks
Gary


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Davanum Srinivas :: https://twitter.com/dims


Message: 2
Date: Sun, 12 Apr 2015 12:36:51 +0000
From: Gary Kotton gkotton@vmware.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Nova] VMware CI
Message-ID: D150418B.7456D%gkotton@vmware.com
Content-Type: text/plain; charset="us-ascii"

Thanks!

On 4/12/15, 3:04 PM, "Davanum Srinivas" davanum@gmail.com wrote:

Gary, John,

Just to speed things up, i filed a backport:
https://review.openstack.org/#/c/172710/

thanks,
dims

On Sun, Apr 12, 2015 at 4:23 AM, John Garbutt john@johngarbutt.com
wrote:

I have updated the bug so it's high priority and tagged with
kilo-rc-potential, and added your note from below as a comment on the
bug.

It looks like it might be worth a backport so it gets into RC2? Can
anyone
take that bit on please?

Thanks,
John

On Sunday, April 12, 2015, Gary Kotton gkotton@vmware.com wrote:

Hi,
Can a core please take a look at
https://review.openstack.org/#/c/171037.
The CI is broken due to commit
e7ae5bb7fbdd5b79bde8937958dd0a645554a5f0.
Thanks
Gary


_
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Davanum Srinivas ::
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_dims&d=Aw
ICAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JY
Bk8YTeq9N3-diTlNj4GyNc&m=O_vdKYuE0xFSaX6xbIHw3qdu0asR94NVcbUKhC9t2vs&s=zv9
uaaxIRcEZR4SDIuS8EJW7YjE4-2QEZKZtxKcl4ow&e=


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Message: 3
Date: Sun, 12 Apr 2015 16:36:32 +0000
From: Gregory Haynes greg@greghaynes.net
To: openstack-dev openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [TripleO] Consistent variable
documentation for diskimage-builder elements
Message-ID: 1428855849-sup-8142@greghaynes0.opus.gah
Content-Type: text/plain; charset=UTF-8

Excerpts from Clint Byrum's message of 2015-04-08 23:11:29 +0000:

I discussed a format for something similar here:

https://review.openstack.org/#/c/162267/

Perhaps we could merge the effort.

The design and implementation in that might take some time, but if we
can document the variables at the same time we prepare the inputs for
isolation, that seems like a winning path forward.

The solution presented there would be awesome for not having to document
the variables manually at all - we can do some sphinx plugin magic to
autogen the doc sections and even get some annoying to write out
features like static links for each var (Im sure you knew this, just
spelling it out).

I agree that itd be better to not put a lot of effort into switching all
the README's over right now and instead work on the argument isolation.
My hope is that in the meanwhile new elements we create and possibly
README's we end up editing get moved over to this new format. Then, we
can try and autogen something that is pretty similar when the time
comes.

Now, lets get that arg isolation donw already. ;)

Cheers,
Greg


Message: 4
Date: Sun, 12 Apr 2015 09:38:20 -0700
From: Joshua Harlow harlowja@outlook.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID: BLU437-SMTP29438D4D9324695F8F13EFD8F80@phx.gbl
Content-Type: text/plain; charset="UTF-8"; format=flowed

Kevin Benton wrote:

So IIUC tooz would be handling the liveness detection for the agents.
That would be nice to get ride of that logic in Neutron and just
register callbacks for rescheduling the dead.

Where does it store that state, does it persist timestamps to the DB
like Neutron does? If so, how would that scale better? If not, who does
a given node ask to know if an agent is online or offline when making a
scheduling decision?

Timestamps are just one way (and likely the most primitive), using redis
(or memcache) key/value and expiry are another (and letting memcache or
redis expire using its own internal algorithms), using zookeeper
ephemeral nodes[1] are another... The point being that its backend
specific and tooz supports varying backends.

However, before (what I assume is) the large code change to implement
tooz, I would like to quantify that the heartbeats are actually a
bottleneck. When I was doing some profiling of them on the master branch
a few months ago, processing a heartbeat took an order of magnitude less
time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A
few query optimizations might buy us a lot more headroom before we have
to fall back to large refactors.

Sure, always good to avoid prematurely optimizing things...

Although this is relevant for u I think anyway:

https://review.openstack.org/#/c/138607/ (same thing/nearly same in nova)...

https://review.openstack.org/#/c/172502/ (a WIP implementation of the
latter).

[1]
https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes

Kevin Benton wrote:

One of the most common is the heartbeat from each agent. However, I
don't think we can't eliminate them because they are used to determine
if the agents are still alive for scheduling purposes. Did you have
something else in mind to determine if an agent is alive?

Put each agent in a tooz[1] group; have each agent periodically
heartbeat[2], have whoever needs to schedule read the active members of
that group (or use [3] to get notified via a callback), profit...

Pick from your favorite (supporting) driver at:

http://docs.openstack.org/__developer/tooz/compatibility.__html
http://docs.openstack.org/developer/tooz/compatibility.html

[1]
http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
http://docs.openstack.org/developer/tooz/compatibility.html#grouping
[2]
https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315
[3]
http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Message: 5
Date: Sun, 12 Apr 2015 13:43:04 -0400
From: Monty Taylor mordred@inaugust.com
To: Bernd Bausch berndbausch@gmail.com, "'OpenStack Development
Mailing List (not for usage questions)'"
openstack-dev@lists.openstack.org,
openstack-docs@lists.openstack.org, openstack-i18n@lists.openstack.org
Cc: 'Jesse Noller' jesse.noller@rackspace.com
Subject: Re: [openstack-dev] [OpenStack-docs] What's Up Doc? Apr 10
2015
Message-ID: 552AAEA8.3020409@inaugust.com
Content-Type: text/plain; charset=windows-1252

On 04/12/2015 04:16 AM, Bernd Bausch wrote:

There is nothing like a good rage on a Sunday (yes Sunday) afternoon. Many
thanks, Monty. You helped me make glance work for my particular case; I will
limit any further messages to the docs mailing list.

Rage on a Sunday followed up by rage coding:

https://review.openstack.org/172728

I figured I should stop flapping my mouth and write some code.

For now I will use API v1 (export OSIMAGEAPI_VERSION=1), pending further
discussions in the install guide team. To me, install guides are more a way
to enter the OpenStack world than an official installation guide; no need to
expose newbies including myself to the complexity of v2.

Bernd

-----Original Message-----
From: Monty Taylor [mailto:mordred@inaugust.com]
Sent: Sunday, April 12, 2015 6:22 AM
To: OpenStack Development Mailing List (not for usage questions);
openstack-docs@lists.openstack.org; openstack-i18n@lists.openstack.org
Cc: Jesse Noller
Subject: Re: [OpenStack-docs] [openstack-dev] What's Up Doc? Apr 10 2015

Sorry for top posting - I wasn't subscribed to the doc list before clarkb
told me about this thread. Warning ... rage coming ... if you don't want to
read rage on a Saturday, I recommend skipping this email.

a) There may be a doc bug here, but I'm not 100% convinced it's a doc bug -
I'll try to characterize it in this way:

"As a user, I do not know what version of glance I am or should be
interacting with"

That part of this is about the default version that python-glanceclient may
or may not use and what version you may or may not need to provide on the
command line is a badness I'll get to in a second - but a clear "so you want
to upload an image, here's what you need to know" is, I think, what Bernd
was looking for

b) Glance is categorically broken in all regards related to this topic.
This thing is the most painful and most broken of everything that exists in
OpenStack. It is the source of MONTHS of development to deal with it in
Infra, and even the workarounds are terrible.

Let me expand:

glance image-upload MAY OR MAY NOT work on your cloud, and there is
absolutely no way you as a user can tell. You just have to try and find out.

IF glance image-upload does not work for you, it may be because of two
things, neither of which are possible for you as a user to find out:

Either:

  • Your cloud has decided to not enable image upload permissions in their
    policy.json file, which is a completely opaque choice that you as a user
    have no way of finding out. If this is the case you have no recourse, sorry.
  • Your cloud has deployed a recent glance and has configured it for glance
    v2 and has configured it in the policy.json file to ONLY allow v2 and to
    disallow image-upload

If the second is true, which you have no way to discover except for trying,
what you need to do is:

  • upload the image to swift
  • glance task-create --type=import --input='{"importfrom":
    "$PATH
    TOIMAGEINSWIFT", "imageproperties" : {"name": "Human Readable
    Image Name"}}'

Yes, you do have to pass JSON on the command line, because BONGHITS (/me
glares at the now absent Brian Waldon with withering disdain for having
inflicted such an absolutely craptastic API on the world.)

Then, you need to poll glance task-status for the status of the import_from
task until your image has imported.

c) The python-glanceclient command line client should encapsulate that
ridiculous logic for you, but it does not

d) It should be possible to discover from the cloud which of the approaches
you should take, but it isn't

Now - I'm honestly not sure how far the docs team should take working around
this - because fully describing how to successfully upload an image without
resorting to calling people names is impossible - but is it really the Docs
team job to make an impossible API seem user friendly? Or, should we not
treat this as a docs bug and instead treat it as a Glance bug and demand a
v3 API that rolls back the task interface?

I vote for the latter.

BTW - the shade library encodes as much of the logic above as it can.
That it exists makes me sad.

Monty

On Sat, Apr 11, 2015 at 10:50 AM, Matt Kassawara
wrote:

Sounds like a problem with one or more packages (perhaps
python-glanceclient?) because that command using the source version
(not
packages) returns the normal list of help items. Maybe try the source
version using "pip install python-glanceclient"?

On Sat, Apr 11, 2015 at 5:55 AM, Bernd Bausch
wrote:

glance help image-create. Sorry for being vague.

When running glance with the parameters from the install guide (the
trunk version), I am told that I am not doing it correctly; I don't
have the precise message handy.

My fear is that I will hit similar problems later. You solving the
problem would be nice but not enough :)

From: Matt Kassawara [mailto:mkassawara at gmail.com]
Sent: Saturday, April 11, 2015 1:59 PM
To: Bernd Bausch
Cc: openstack-docs at lists.openstack.org

Subject: Re: [OpenStack-docs] [install-guide] RE: What's Up Doc?
Apr
10 2015

When you run "glance help image-create" or just "glance image-create"
with no arguments?

On Fri, Apr 10, 2015 at 11:45 PM, Bernd Bausch
wrote:

This is what I get when running glance image-create:

            usage: glance image-create [--property <key=value>] 

[--file ]

[--progress]

            Create a new image.



            Positional arguments:

              <unavailable>         Please run with connection

parameters set to retrieve

                                                   the schema for 

generating help for this command

So I wonder how I can get to the bottom of this.

From: Matt Kassawara [mailto:mkassawara at gmail.com]
Sent: Saturday, April 11, 2015 1:39 PM
To: Bernd Bausch; openstack-docs at lists.openstack.org
Subject: Re: [OpenStack-docs] [install-guide] RE: What's Up Doc?
Apr
10 2015

I'd use the conventional python-*client for all services except
keystone because the Openstack client doesn't seem very complete for
them. If
you're
using the glance client, it defaults to the v1 API and the commands
from the Juno installation guide should work. If you use the v2 API,
one thing changes with how to set public/private visibility.

On Fri, Apr 10, 2015 at 8:11 PM, Bernd Bausch
wrote:

Regarding the installation guide, I need some advice. Perhaps the
docs community can help?

I am trying to install Kilo on yum-based systems using a repo from
the RDO project. I have hit a few roadblocks that I have been able to
deal with, but I am unsure what to do with the current one.

My questions are: Is it appropriate to ask developers about the
intended way of doing things, if the old ways don't work anymore? If
yes, what are the best channels - chat, dev mailing list, personal
email, .? If no,
what
else can I do? Do developers make such changes public somewhere?

Below is the problem I am currently trying to solve. Note that I
am including it as an illustration what I am struggling with (more
problems will show up as I continue working on this); I am not asking
you to solve this particular problem for me.

So far, to upload an image to Glance, the "glance image-create"
command is used. This command doesn't work anymore as in the past,
and I don't understand what the "glance help image-create" is trying
to say. On the other hand, I haven't found an equivalent command in the
new "openstack"
CLI client. So my question is - what is the correct way to upload an
image
these days.

Have a great weekend,

Bernd

From: Anne Gentle [mailto:annegentle at justwriteclick.com]
Sent: Saturday, April 11, 2015 12:24 AM
To: openstack-docs at lists.openstack.org; OpenStack Development
Mailing List; openstack-i18n at lists.openstack.org
Cc: Jesse Noller
Subject: [OpenStack-docs] What's Up Doc? Apr 10 2015

Hi all,

As you probably saw from PTL nominations last week, I'm happy to hand
the docs PTL baton to Lana Brindley! I loved leading this group and
thank you all for supporting me. Thank you Lana for your willingness
to lead. I'm still here to bring us to the Kilo release, so this
week's What's Up Doc brings sharp focus requests to everyone to work
on docs. These are
the top
priorities that we all need to work on - devs, writers, testers,
gaters, everyone.

  1. Bug triaging and fixing, especially for openstack-manuals. There
    are nearly 300 DocImpact bugs logged that we need developers to
    circle
    back to.
    With nearly 600 bugs overall, we need lots of focus here. To that
    end, I propose we hold a bug triage day. I'll send details in a separate
    email.

  2. Install Guide testing and reviewing. The Install Guide team has a
    published spec that will help reviewers see what's changing with the
    Kilo Install guide:

http://specs.openstack.org/openstack/docs-specs/specs/kilo/installguide-kilo
.html

Join them for weekly meetings Tuesdays at at 13:00 UTC (8:00 AM US
CDT) in
Google Hangout:

https://plus.google.com/hangouts/_/calendar/a2FyaW4ua2F0aG9kZUBnbWFpbC5jb20.
jj2lu2nbj71a0dan11vatdav3k

If you do nothing else but these two focus areas we'll be in good shape.
There are other activities going on leading up to Vancouver but those
two are top priorities.

RST Migration

We are working to resolve translation tool expectations with the i18N
team. I want to publish the RST-based English End User Guide and
Admin User
Guide once we're all comfortable with the way forward. Daisy will
discuss the implications at the next i18N team meeting Thursday at
0800 UTC, and we'll implement and communicate the plan.

Networking Guide

Next on the list is Networking Guide testing and reviewing. The
Networking Guide team has a talk in Vancouver and needs to get their
guide
in shape for publishing. The neutron team is holding a doc day April 23.
Please join in -- they'll post details in their notes.

First App Tutorial

There's also the First Application Tutorial that needs to finish the
spec and needs an editing cleanup prior to publishing. Ideally that
will
happen
before Vancouver, we need to get it to the finish line.

HA Guide

With everything else going on we need an updated spec for the HA
Guide - the wiki page isn't enough. Based on this week's doc team
meeting it
sounds
like that can go to RST as well but we need it in the spec so we can
plan and cross-coordinate with the other priorities leading up to
Vancouver.

_Vancouver Summit _
The docs team has four Fishbowl time slots, 2 Workroom slots, and 1
Meetup allocated now. If you'd like to discuss a cross-project idea,
please
use the form to suggest new ideas:
http://goo.gl/forms/S69HM6XEeb. You can see the current suggestions
already posted here:

https://docs.google.com/spreadsheets/d/1vCTZBJKCMZ2xBhglnuK3ciKo3E8UMFo5S5lm
IAYMCSE/edit?usp=sharing

Lana or I will send out an etherpad in the next week or so with topic
suggestions for our allocation.

Thanks,
Anne

--
Anne Gentle
annegentle at justwriteclick.com


OpenStack-docs mailing list
OpenStack-docs at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs


OpenStack-docs mailing list
OpenStack-docs at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs


OpenStack-docs mailing list
OpenStack-docs@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs

Message: 6
Date: Sun, 12 Apr 2015 12:40:07 -0700
From: Kevin Benton blak111@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
CAO_F6JMFRGSGAQu_CyXGZZmkuHmy8vuO2gCUWHcL+DMeuyxd1Q@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

I assumed that all agents are connected to same IP address of RabbitMQ,
then the connection will exceed the port ranges limitation.

Only if the clients are all using the same IP address. If connections
weren't scoped by source IP, busy servers would be completely unreliable
because clients would keep having source port collisions.

For example, the following is a netstat output from a server with two
connections to a service running on port 4000 with both clients using
source port 50000: http://paste.openstack.org/show/203211/

the client should be aware of the cluster member failure, and reconnect to
other survive member. No such mechnism has been implemented yet.

If I understand what you are suggesting, it already has been implemented
that way. The neutron agents and servers can be configured with multiple
rabbitmq servers and they will cycle through the list whenever there is a
failure.

The only downside to that approach is that every neutron agent and server
has to be configured with every rabbitmq server address. This gets tedious
to manage if you want to add cluster members dynamically so using a load
balancer can help relieve that.

Hi, Kevin,

I assumed that all agents are connected to same IP address of RabbitMQ,
then the connection will exceed the port ranges limitation.

For a RabbitMQ cluster, for sure the client can connect to any one of
member in the cluster, but in this case, the client has to be designed in
fail-safe manner: the client should be aware of the cluster member failure,
and reconnect to other survive member. No such mechnism has
been implemented yet.

Other way is to use LVS or DNS based like load balancer, or something else.
If you put one load balancer ahead of a cluster, then we have to take care
of the port number limitation, there are so many agents will require
connection concurrently, 100k level, and the requests can not be rejected.

Best Regards

Chaoyi Huang ( joehuang )


From: Kevin Benton [blak111@gmail.com]
Sent: 12 April 2015 9:59
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

The TCP/IP stack keeps track of connections as a combination of IP + TCP
port. The two byte port limit doesn't matter unless all of the agents are
connecting from the same IP address, which shouldn't be the case unless
compute nodes connect to the rabbitmq server via one IP address running
port address translation.

Either way, the agents don't connect directly to the Neutron server, they
connect to the rabbit MQ cluster. Since as many Neutron server processes
can be launched as necessary, the bottlenecks will likely show up at the
messaging or DB layer.

On Sat, Apr 11, 2015 at 6:46 PM, joehuang joehuang@huawei.com wrote:

As Kevin talking about agents, I want to remind that in TCP/IP stack,
port ( not Neutron Port ) is a two bytes field, i.e. port ranges from 0 ~
65535, supports maximum 64k port number.

" above 100k managed node " means more than 100k L2 agents/L3 agents...
will be alive under Neutron.

Want to know the detail design how to support 99.9% possibility for
scaling Neutron in this way, and PoC and test would be a good support for
this idea.

"I'm 99.9% sure, for scaling above 100k managed node,
we do not really need to split the openstack to multiple smaller openstack,
or use significant number of extra controller machine."

Best Regards

Chaoyi Huang ( joehuang )


From: Kevin Benton [blak111@gmail.com]
Sent: 11 April 2015 12:34
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Which periodic updates did you have in mind to eliminate? One of the
few remaining ones I can think of is sync_routers but it would be great if
you can enumerate the ones you observed because eliminating overhead in
agents is something I've been working on as well.

One of the most common is the heartbeat from each agent. However, I
don't think we can't eliminate them because they are used to determine if
the agents are still alive for scheduling purposes. Did you have something
else in mind to determine if an agent is alive?

On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas afazekas@redhat.com
wrote:

I'm 99.9% sure, for scaling above 100k managed node,
we do not really need to split the openstack to multiple smaller
openstack,
or use significant number of extra controller machine.

The problem is openstack using the right tools SQL/AMQP/(zk),
but in a wrong way.

For example.:
Periodic updates can be avoided almost in all cases

The new data can be pushed to the agent just when it needed.
The agent can know when the AMQP connection become unreliable (queue or
connection loose),
and needs to do full sync.
https://bugs.launchpad.net/neutron/+bug/1438159

Also the agents when gets some notification, they start asking for
details via the
AMQP -> SQL. Why they do not know it already or get it with the
notification ?

----- Original Message -----

From: "Neil Jerram" Neil.Jerram@metaswitch.com
To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev@lists.openstack.org>
Sent: Thursday, April 9, 2015 5:01:45 PM
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Hi Joe,

Many thanks for your reply!

On 09/04/15 03:34, joehuang wrote:

Hi, Neil,

From theoretic, Neutron is like a "broadcast" domain, for example,
enforcement of DVR and security group has to touch each regarding
host
where there is VM of this project resides. Even using SDN
controller, the
"touch" to regarding host is inevitable. If there are plenty of
physical
hosts, for example, 10k, inside one Neutron, it's very hard to
overcome
the "broadcast storm" issue under concurrent operation, that's the
bottleneck for scalability of Neutron.

I think I understand that in general terms - but can you be more
specific about the broadcast storm? Is there one particular message
exchange that involves broadcasting? Is it only from the server to
agents, or are there 'broadcasts' in other directions as well?

(I presume you are talking about control plane messages here, i.e.
between Neutron components. Is that right? Obviously there can also be
broadcast storm problems in the data plane - but I don't think that's
what you are talking about here.)

We need layered architecture in Neutron to solve the "broadcast
domain"
bottleneck of scalability. The test report from OpenStack cascading
shows
that through layered architecture "Neutron cascading", Neutron can
supports up to million level ports and 100k level physical hosts. You
can
find the report here:

http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

Many thanks, I will take a look at this.

"Neutron cascading" also brings extra benefit: One cascading Neutron
can
have many cascaded Neutrons, and different cascaded Neutron can
leverage
different SDN controller, maybe one is ODL, the other one is
OpenContrail.

----------------Cascading Neutron-------------------
/ \
--cascaded Neutron-- --cascaded Neutron-----
| |
---------ODL------ ----OpenContrail--------

And furthermore, if using Neutron cascading in multiple data centers,
the
DCI controller (Data center inter-connection controller) can also be
used
under cascading Neutron, to provide NaaS ( network as a service )
across
data centers.

---------------------------Cascading Neutron--------------------------
/ | \
--cascaded Neutron-- -DCI controller- --cascaded Neutron-----
| | |
---------ODL------ | ----OpenContrail--------
|
--(Data center 1)-- --(DCI networking)-- --(Data center 2)--

Is it possible for us to discuss this in OpenStack Vancouver summit?

Most certainly, yes. I will be there from mid Monday afternoon through
to end Friday. But it will be my first summit, so I have no idea yet as
to how I might run into you - please can you suggest!

Best Regards
Chaoyi Huang ( Joe Huang )

Regards,
Neil


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/ea01d543/attachment-0001.html

Message: 7
Date: Sun, 12 Apr 2015 12:51:30 -0700
From: Kevin Benton blak111@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
CAO_F6JM61uqXNY_ebNNXKxFNV5-C7wr7A3LSZC_0RY+NMQKcdw@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

Timestamps are just one way (and likely the most primitive), using redis
(or memcache) key/value and expiry are another (and letting memcache or
redis expire using its own internal algorithms), using zookeeper ephemeral
nodes[1] are another... The point being that its backend specific and tooz
supports varying backends.

Very cool. Is the backend completely transparent so a deployer could choose
a service they are comfortable maintaining, or will that change the
properties WRT to resiliency of state on node restarts, partitions, etc?

The Nova implementation of Tooz seemed pretty straight-forward, although it
looked like it had pluggable drivers for service management already. Before
I dig into it much further I'll file a spec on the Neutron side to see if I
can get some other cores onboard to do the review work if I push a change
to tooz.

On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow harlowja@outlook.com wrote:

Kevin Benton wrote:

So IIUC tooz would be handling the liveness detection for the agents.
That would be nice to get ride of that logic in Neutron and just
register callbacks for rescheduling the dead.

Where does it store that state, does it persist timestamps to the DB
like Neutron does? If so, how would that scale better? If not, who does
a given node ask to know if an agent is online or offline when making a
scheduling decision?

Timestamps are just one way (and likely the most primitive), using redis
(or memcache) key/value and expiry are another (and letting memcache or
redis expire using its own internal algorithms), using zookeeper ephemeral
nodes[1] are another... The point being that its backend specific and tooz
supports varying backends.

However, before (what I assume is) the large code change to implement
tooz, I would like to quantify that the heartbeats are actually a
bottleneck. When I was doing some profiling of them on the master branch
a few months ago, processing a heartbeat took an order of magnitude less
time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A
few query optimizations might buy us a lot more headroom before we have
to fall back to large refactors.

Sure, always good to avoid prematurely optimizing things...

Although this is relevant for u I think anyway:

https://review.openstack.org/#/c/138607/ (same thing/nearly same in
nova)...

https://review.openstack.org/#/c/172502/ (a WIP implementation of the
latter).

[1] https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#
Ephemeral+Nodes

Kevin Benton wrote:

One of the most common is the heartbeat from each agent. However, I
don't think we can't eliminate them because they are used to determine
if the agents are still alive for scheduling purposes. Did you have
something else in mind to determine if an agent is alive?

Put each agent in a tooz[1] group; have each agent periodically
heartbeat[2], have whoever needs to schedule read the active members of
that group (or use [3] to get notified via a callback), profit...

Pick from your favorite (supporting) driver at:

http://docs.openstack.org/__developer/tooz/compatibility.__html
http://docs.openstack.org/developer/tooz/compatibility.html

[1]
http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
http://docs.openstack.org/developer/tooz/compatibility.html#grouping
[2]
https://github.com/openstack/__tooz/blob/0.13.1/tooz/__
coordination.py#L315
https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315
[3]
http://docs.openstack.org/__developer/tooz/tutorial/group_
__membership.html#watching-__group-changes
http://docs.openstack.org/developer/tooz/tutorial/group_
membership.html#watching-group-changes>



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/c5c3d2dc/attachment-0001.html


Message: 8
Date: Mon, 13 Apr 2015 10:43:57 +1200
From: Robert Collins robertc@robertcollins.net
To: OpenStack Development Mailing List
openstack-dev@lists.openstack.org
Subject: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
CAJ3HoZ2JLwPJUBRgTLyBZ-z00sVO4TyOupmR4LprjkNMHg9o0g@mail.gmail.com
Content-Type: text/plain; charset=UTF-8

Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.

Upstream there are two separate concepts.

installrequirements, which are meant to document what must be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install
requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.

deploy requirements - requirements.txt - which are meant to be local
to a deployment
, and are commonly expected to specify very narrow (or
even exact fit) versions.

What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -
http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63

I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.

Concretely, I think we need to:
- teach pbr to read in installrequires from setup.cfg, not requirements.txt
- when there are requirements in setup.cfg, stop reading requirements.txt
- separate out the global intall
requirements from the global CI
requirements, and update our syncing code to be aware of this

Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.

Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


Message: 9
Date: Sun, 12 Apr 2015 19:12:38 -0400
From: Monty Taylor mordred@inaugust.com
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID: 552AFBE6.4010606@inaugust.com
Content-Type: text/plain; charset=windows-1252

On 04/12/2015 06:43 PM, Robert Collins wrote:

Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.

Upstream there are two separate concepts.

installrequirements, which are meant to document what must be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install
requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.

deploy requirements - requirements.txt - which are meant to be local
to a deployment
, and are commonly expected to specify very narrow (or
even exact fit) versions.

tl;dr - I'm mostly in support of what you're saying - but I'm going to
bludgeon it some.

To be either fair or unfair, depending on how you look at it - some
people upstream consider those two to be a pattern, but it is not
encoded anywhere except in hidden lore that is shared between secret
people. Upstream's tools have bumpkiss for support for this, and if we
hadn't drawn a line in the sand encoding SOME behavior there would still
be nothing.

Nor, btw, is it the right split. It optimizes for the wrong thing.

rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
application should ship with a Cargo.lock and a library should not.

Without the library/application distinction, the effort in
differentiating is misplaced, I believe - because it's libraries that
need flexible depends - and applications where the specific set of
depends that were tested in CI become important to consumers.

What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -
http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63

That is, btw, because that's what the overwhelming majority of consumers
assume those files mean. I take "overwhelming majority" from the days
when we had files but did not process them automatically and everyone
was confused.

I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.

Ok. I mean, they don't have a better answer, but if it makes "python"
hate us less, sweet.

Concretely, I think we need to:
- teach pbr to read in installrequires from setup.cfg, not requirements.txt
- when there are requirements in setup.cfg, stop reading requirements.txt
- separate out the global intall
requirements from the global CI
requirements, and update our syncing code to be aware of this

Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.

Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.

I'm concerned that it adds a layer of difference that is confusing to
people for the sole benefit of pleasing someone else's pedantic worldview.

I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that still
isn't going to be "how people do it"

I'm concerned that "how people do it" is a myth not worth chasing.

I'm not opposed to making this richer and more useful for people. I
just don't know what's broken currently for us.

Monty


Message: 10
Date: Mon, 13 Apr 2015 10:01:44 +1000
From: James Polley jp@jamezpolley.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
CAPtRfUEoLc0KQoqPaHffxXevGwCiXaExm3Kx2oJWzNGFjL3NWA@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

On Mon, Apr 13, 2015 at 9:12 AM, Monty Taylor mordred@inaugust.com wrote:

On 04/12/2015 06:43 PM, Robert Collins wrote:

Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.

Upstream there are two separate concepts.

installrequirements, which are meant to document what must be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install
requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.

deploy requirements - requirements.txt - which are meant to be local
to a deployment
, and are commonly expected to specify very narrow (or
even exact fit) versions.

That sounds, to me, very similar to a discussion we had a few weeks ago in
the context of our stable branches.

In that context, we have two competing requirements. One requirement is
that our CI system wants a very tightly pinned requirements, as do
downstream CI systems and deployers that want to test and deploy exact
known-tested versions of things. On the other hand, downstream distributors
(including OS packagers) need to balance OpenStack's version requirements
with version requirements from all the other packages in their
distribution; the tighter the requirements we list are, the harder it is
for the requirements to work with the requirements of other packages in the
distribution.

tl;dr - I'm mostly in support of what you're saying - but I'm going to
bludgeon it some.

To be either fair or unfair, depending on how you look at it - some
people upstream consider those two to be a pattern, but it is not
encoded anywhere except in hidden lore that is shared between secret
people. Upstream's tools have bumpkiss for support for this, and if we
hadn't drawn a line in the sand encoding SOME behavior there would still
be nothing.

Nor, btw, is it the right split. It optimizes for the wrong thing.

rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
application should ship with a Cargo.lock and a library should not.

This sounds similar to a solution that was proposed for the stable
branches: a requirements.in with mandatory version constraints while being
as loose as otherwise possible, which is used to generate a
requirements.txt which has the "local to deployment" exact versions that
are used in our CI. The details of the proposal are in
https://review.openstack.org/#/c/161047/

Without the library/application distinction, the effort in
differentiating is misplaced, I believe - because it's libraries that
need flexible depends - and applications where the specific set of
depends that were tested in CI become important to consumers.

What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -

http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63

That is, btw, because that's what the overwhelming majority of consumers
assume those files mean. I take "overwhelming majority" from the days
when we had files but did not process them automatically and everyone
was confused.

I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.

Ok. I mean, they don't have a better answer, but if it makes "python"
hate us less, sweet.

Concretely, I think we need to:
- teach pbr to read in installrequires from setup.cfg, not
requirements.txt
- when there are requirements in setup.cfg, stop reading
requirements.txt
- separate out the global intall
requirements from the global CI
requirements, and update our syncing code to be aware of this

Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.

Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.

I'm concerned that it adds a layer of difference that is confusing to
people for the sole benefit of pleasing someone else's pedantic worldview.

I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that still
isn't going to be "how people do it"

I'm concerned that "how people do it" is a myth not worth chasing.

I'm not opposed to making this richer and more useful for people. I
just don't know what's broken currently for us.

To be clear, I don't mean to suggest that the solution proposed in
https://review.openstack.org/#/c/161047/ is necessarily the correct
solution to this problem - but I do think that it is illustrative of at
last one thing that's currently broken for us.

Monty


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/4e267c09/attachment-0001.html


Message: 11
Date: Sun, 12 Apr 2015 20:53:50 -0400
From: Monty Taylor mordred@inaugust.com
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID: 552B139E.5010007@inaugust.com
Content-Type: text/plain; charset=windows-1252

On 04/12/2015 08:01 PM, James Polley wrote:

On Mon, Apr 13, 2015 at 9:12 AM, Monty Taylor mordred@inaugust.com wrote:

On 04/12/2015 06:43 PM, Robert Collins wrote:

Right now we do something that upstream pip considers wrong: we make
our requirements.txt be our install_requires.

Upstream there are two separate concepts.

installrequirements, which are meant to document what must be
installed to import the package, and should encode any mandatory
version constraints while being as loose as otherwise possible. E.g.
if package A depends on package B version 1.5 or above, it should say
B>=1.5 in A's install
requires. They should not specify maximum
versions except when that is known to be a problem: they shouldn't
borrow trouble.

deploy requirements - requirements.txt - which are meant to be local
to a deployment
, and are commonly expected to specify very narrow (or
even exact fit) versions.

That sounds, to me, very similar to a discussion we had a few weeks ago in
the context of our stable branches.

In that context, we have two competing requirements. One requirement is
that our CI system wants a very tightly pinned requirements, as do
downstream CI systems and deployers that want to test and deploy exact
known-tested versions of things. On the other hand, downstream distributors
(including OS packagers) need to balance OpenStack's version requirements
with version requirements from all the other packages in their
distribution; the tighter the requirements we list are, the harder it is
for the requirements to work with the requirements of other packages in the
distribution.

This is not accurate. During distro packaging activities, pbr does not
process dependencies at all. So no matter how we pin things in
OpenStack, it does not make it harder for the distros.

tl;dr - I'm mostly in support of what you're saying - but I'm going to
bludgeon it some.

To be either fair or unfair, depending on how you look at it - some
people upstream consider those two to be a pattern, but it is not
encoded anywhere except in hidden lore that is shared between secret
people. Upstream's tools have bumpkiss for support for this, and if we
hadn't drawn a line in the sand encoding SOME behavior there would still
be nothing.

Nor, btw, is it the right split. It optimizes for the wrong thing.

rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
application should ship with a Cargo.lock and a library should not.

This sounds similar to a solution that was proposed for the stable
branches: a requirements.in with mandatory version constraints while being
as loose as otherwise possible, which is used to generate a
requirements.txt which has the "local to deployment" exact versions that
are used in our CI. The details of the proposal are in
https://review.openstack.org/#/c/161047/

I disagree with this proposal. It's also not helping any users. Because
of what I said above, there is no flexibility that we lose downstream by
being strict and pedantic with our versions. So, having the "lose" and
the "strict" file just gets us two files and doubles the confusion.
Having a list of what we know the state to be is great. We should give
that to users. If they want to use something other than pip to install,
awesome - the person in charge of curating that content can test the
version interactions in their environment.

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Without the library/application distinction, the effort in
differentiating is misplaced, I believe - because it's libraries that
need flexible depends - and applications where the specific set of
depends that were tested in CI become important to consumers.

What pbr, which nearly if not all OpenStack projects use, does, is to
map the contents of requirements.txt into install_requires. And then
we use the same requirements.txt in our CI to control whats deployed
in our test environment[*]. and there we often have tight constraints
like seen here -

http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n63

That is, btw, because that's what the overwhelming majority of consumers
assume those files mean. I take "overwhelming majority" from the days
when we had files but did not process them automatically and everyone
was confused.

I'd like to align our patterns with those of upstream, so that we're
not fighting our tooling so much.

Ok. I mean, they don't have a better answer, but if it makes "python"
hate us less, sweet.

Concretely, I think we need to:
- teach pbr to read in installrequires from setup.cfg, not
requirements.txt
- when there are requirements in setup.cfg, stop reading
requirements.txt
- separate out the global intall
requirements from the global CI
requirements, and update our syncing code to be aware of this

Then, setup.cfg contains more open requirements suitable for being on
PyPI, requirements.txt is the local CI set we know works - and can be
much more restrictive as needed.

Thoughts? If there's broad apathy-or-agreement I can turn this into a
spec for fine coverage of ramifications and corner cases.

I'm concerned that it adds a layer of difference that is confusing to
people for the sole benefit of pleasing someone else's pedantic worldview.

I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that still
isn't going to be "how people do it"

I'm concerned that "how people do it" is a myth not worth chasing.

I'm not opposed to making this richer and more useful for people. I
just don't know what's broken currently for us.

To be clear, I don't mean to suggest that the solution proposed in
https://review.openstack.org/#/c/161047/ is necessarily the correct
solution to this problem - but I do think that it is illustrative of at
last one thing that's currently broken for us.

I disagree that anything is broken for us that is not caused by our
inability to remember that distro packaging concerns are not the same as
our concerns, and that the mechanism already exists for distro pacakgers
to do what they want. Furthermore, it is caused by an insistence that we
need to keep versions "open" for some ephemeral reason such as "upstream
might release a bug fix" Since we all know that "if it's not tested,
it's broken" - any changes to upstream software should be considered
broken until proven otherwise. History over the last 5 years has shown
this to be accurate more than the other thing.

If we pin the stable branches with hard pins of direct and indirect
dependencies, we can have our stable branch artifacts be installable.
Thats awesome. IF there is a bugfix release or a security update to a
dependent library - someone can propose it. Otherwise, the stable
release should not be moving.

Monty


Message: 12
Date: Mon, 13 Apr 2015 13:02:13 +1200
From: Robert Collins robertc@robertcollins.net
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
CAJ3HoZ2N9+EvsN0WkO2tiQLPKDi9q2tX_Qk8JRskdz+EX5+vBw@mail.gmail.com
Content-Type: text/plain; charset=UTF-8

On 13 April 2015 at 12:01, James Polley jp@jamezpolley.com wrote:
>
>

That sounds, to me, very similar to a discussion we had a few weeks ago in
the context of our stable branches.

In that context, we have two competing requirements. One requirement is that
our CI system wants a very tightly pinned requirements, as do downstream CI
systems and deployers that want to test and deploy exact known-tested
versions of things. On the other hand, downstream distributors (including OS
packagers) need to balance OpenStack's version requirements with version
requirements from all the other packages in their distribution; the tighter
the requirements we list are, the harder it is for the requirements to work
with the requirements of other packages in the distribution.

They are analogous yes.
..

rust gets it wright. There is a Cargo.toml and a Cargo.lock, which are
understood by the tooling in a manner similar to what you have
described, and it is not just understood but DOCUMENTED that an
application should ship with a Cargo.lock and a library should not.

This sounds similar to a solution that was proposed for the stable branches:
a requirements.in with mandatory version constraints while being as loose as
otherwise possible, which is used to generate a requirements.txt which has
the "local to deployment" exact versions that are used in our CI. The
details of the proposal are in https://review.openstack.org/#/c/161047/

That proposal is still under discussion, and seems stuck between the
distro and -infra folk. if it ends up doing the transitive thing, I
think that that would make a sensible requirements.txt, yes. However
see the spec for that thread of discussion.
.

I'm also concerned that dstufft is actively wanting to move towards a
world where the build tooling is not needed or used as part of the
install pipeline (metadata 2.0) -- so I'm concerned that we're aligning
with a pattern that isn't very robust and isn't supported by tooling
directly and that we're going to need to change understood usage
patterns across a large developer based to chase something that still
isn't going to be "how people do it"

Monty:
So wheels are already in that space. metadata-2.0 is about bringing
that declarative stuff forward in the pipeline, from binary packages
to source packages. I'm currently using frustration based development
to bring it in at the start - for developers, in the lead-in to source
packages.

So you're concerned - but about what specifically? What goes wrong
with wheels (not wheels with C code). Whats not robust about the
pattern? The cargo package manager you referred to is entirely
declarative....

James: I don't think the binary distribution stuff is relevant to my
discussion, since I'm talking entirely about 'using-pip' use cases,
whereas dpkg and rpm packages don't use that at all.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


Message: 13
Date: Mon, 13 Apr 2015 13:09:44 +1200
From: Robert Collins robertc@robertcollins.net
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
CAJ3HoZ0_O6y3tmc15xaO-P_04R8zFGUceQEUmd_6w9Q7Bk_vOg@mail.gmail.com
Content-Type: text/plain; charset=UTF-8

On 13 April 2015 at 12:53, Monty Taylor mordred@inaugust.com wrote:

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Actually its not.

What we test is point in time. At 2:45 UTC on Monday installing this
git ref of nova worked.

Noone can reconstruct that today.

I entirely agree with the sentiment you're expressing, but we're not
delivering that sentiment today.

We need to balance the inability to atomically update things - which
forces a degree of freedom on install_requires - with being able to
give someone the same install that we tested.

That is the fundamental tension that we're not handling well, nor have
I seen a proposal to tackle it so far.

I'll have to spend some time noodling on this, but one of the clear
constraints is that install_requires cannot both be:
- flexible enough to permit us to upgrade requirements across many
git based packages [because we could do coordinated releases of sdists
to approximate atomic bulk changes]
- tight enough enough to give the next person trying to run that ref
of the package the same things we installed in CI.

-> I think we need something other than install_requires

..

I disagree that anything is broken for us that is not caused by our
inability to remember that distro packaging concerns are not the same as
our concerns, and that the mechanism already exists for distro pacakgers
to do what they want. Furthermore, it is caused by an insistence that we
need to keep versions "open" for some ephemeral reason such as "upstream
might release a bug fix" Since we all know that "if it's not tested,
it's broken" - any changes to upstream software should be considered
broken until proven otherwise. History over the last 5 years has shown
this to be accurate more than the other thing.

This seems like a strong argument for really being able to reconstruct
what was in CI.

If we pin the stable branches with hard pins of direct and indirect
dependencies, we can have our stable branch artifacts be installable.
Thats awesome. IF there is a bugfix release or a security update to a
dependent library - someone can propose it. Otherwise, the stable
release should not be moving.

Can we do that in stable branches? We've still got the problem of
bumping dependencies across multiple packages.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


Message: 14
Date: Sun, 12 Apr 2015 20:14:00 -0500
From: Brant Knudson blk@acm.org
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: [openstack-dev] [all] Problems with keystoneclient stable
branch (and maybe yours too)
Message-ID:
CAHjeE=Qw1E=bKfKCuEvqckgZpG0VOjAsjYpdVJjOF0fBtVTEJA@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

There were several problems with the keystoneclient stable/juno branch that
have been or are in the process of being fixed since its creation.
Hopefully this note will be useful to other projects that create stable
branches for their libraries.

1) Unit tests didn't pass with earlier packages

The supported versions of several of the packages in requirements.txt in
the stable branch are in the process of being capped[0], so that the tests
are now running with older versions of the packages. Since we don't
normally test with the older packages we didn't know that the
keystoneclient unit tests don't actually pass with the old version of the
package. This is fixed by correcting the tests to work with the older
versions of the packages.[1][2]

[0] https://review.openstack.org/#/c/172220/
[1] https://review.openstack.org/#/c/172655/
[2] https://review.openstack.org/#/c/172256/

It would be great if we were testing with the minimum versions of the
packages that we say we support somehow since that would have caught this.

2) Incorrect cap in requirements.txt

python-keystoneclient in stable/juno was capped at <=1.1.0, and 1.1.0 is
the version tagged for the stable branch. When you create a review in
stable/juno it installs python-keystoneclient and now the system has got a
version like 1.1.0.post1, which is >1.1.0, so now python-keystoneclient
doesn't match the requirements and swift-proxy fails to start (swift-proxy
is very good at catching this problem for whatever reason). The cap should
have been <1.2.0 so that we can propose patches and also make fix releases
(1.1.1, 1.1.2, etc.).[3]

[3] https://review.openstack.org/#/c/172718/

I tried to recap all of the clients but that didn't pass Jenkins, probably
because one or more clients didn't use semver correctly and have
requirements updates in a micro release.[4]

[4] https://review.openstack.org/#/c/172719/

3) Unsupported functional tests

We added support for functional tests (tox -e functional) in K, but Jenkins
was configured to run the functional job on all branches and it fails when
the tox target doesn't exist. The fix was to exclude the current stable/
branches for keystoneclient.[5]

[5] https://review.openstack.org/#/c/172658/

4) Tempest -juno job?

For some reason keystoneclient has 2 tempest-neutron jobs:
gate-tempest-dsvm-neutron-src-python-keystoneclient and
..-keystoneclient-juno , and the -juno job is failing in stable/juno. It
didn't make sense to me that we needed to run both in python-keystoneclient
stable/juno. I was told that we didn't need the -juno one anymore on any
branch since we have a stable/juno branch, so that job is removed.[6]

[6] https://review.openstack.org/#/c/172662/

Hopefully with these changes the python-keystoneclient stable/juno branch
will be working.


Message: 15
Date: Mon, 13 Apr 2015 11:33:25 +1000
From: Michael Still mikal@stillhq.com
To: OpenStack Development Mailing List
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] In loving memory of Chris Yeoh
Message-ID:
CAEd1pt7j2JutUjihkx1-S1ikZ_mZ_QHrMEhNMsUTkx7G7gY-GQ@mail.gmail.com
Content-Type: text/plain; charset=UTF-8

Hi, as promised I now have details of a charity for people to donate
to in Chris' memory:

http://participate.freetobreathe.org/site/TR?px=1582460&fr_id=2710&pg=personal#.VSscH5SUd90

In the words of the family:

"We would prefer that people donate to lung cancer research in lieu of
flowers. Lung cancer has the highest mortality rate out of all the
cancers, and the lowest funding out of all the cancers. There is a
stigma attached that lung cancer is a smoker's disease, and that
sufferers deserve their fate. They bring it on through lifestyle
choice. Except that Chris has never smoked in his life, like a
surprisingly large percentage of lung cancer sufferers. These people
suffer for the incorrect beliefs of the masses, and those that are
left behind are equally innocent. We shouldn't be doing this now. He
shouldn't be gone. We need to do more to fix this. There will be
charity envelopes available at the funeral, or you can choose your
preferred research to fund, should you wish to do so. You have our
thanks."

Michael

On Wed, Apr 8, 2015 at 2:49 PM, Michael Still mikal@stillhq.com wrote:

It is my sad duty to inform the community that Chris Yeoh passed away this
morning. Chris leaves behind a daughter Alyssa, aged 6, who I hope will
remember Chris as the clever and caring person that I will remember him as.
I haven?t had a chance to confirm with the family if they want flowers or a
donation to a charity. As soon as I know those details I will reply to this
email.

Chris worked on open source for a very long time, with OpenStack being just
the most recent in a long chain of contributions. He worked tirelessly on
his contributions to Nova, including mentoring other developers. He was
dedicated to the cause, with a strong vision of what OpenStack could become.
He even named his cat after the project.

Chris might be the only person to have ever sent an email to his coworkers
explaining what his code review strategy would be after brain surgery. It
takes phenomenal strength to carry on in the face of that kind of adversity,
but somehow he did. Frankly, I think I would have just sat on the beach.

Chris was also a contributor to the Linux Standards Base (LSB), where he
helped improve the consistency and interoperability between Linux
distributions. He ran the ?Hackfest? programming contests for a number of
years at Australia?s open source conference -- linux.conf.au. He supported
local Linux user groups in South Australia and Canberra, including
involvement at installfests and speaking at local meetups. He competed in a
programming challenge called Loki Hack, and beat out the world to win the
event[1].

Alyssa?s memories of her dad need to last her a long time, so we?ve decided
to try and collect some fond memories of Chris to help her along the way. If
you feel comfortable doing so, please contribute a memory or two at
https://docs.google.com/forms/d/1kX-ePqAO7Cuudppwqz1cqgBXAsJx27GkdM-eCZ0c1V8/viewform

Chris was humble, helpful and honest. The OpenStack and broader Open Source
communities are poorer for his passing.

Michael

[1] http://www.lokigames.com/hack/

--
Rackspace Australia


Message: 16
Date: Mon, 13 Apr 2015 13:53:36 +1200
From: Robert Collins robertc@robertcollins.net
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID:
CAJ3HoZ3RKSi8TQLnbS=eWFekOxztVtkE2EE+xFoXWvVnKUoBWw@mail.gmail.com
Content-Type: text/plain; charset=UTF-8

On 13 April 2015 at 13:09, Robert Collins robertc@robertcollins.net wrote:

On 13 April 2015 at 12:53, Monty Taylor mordred@inaugust.com wrote:

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Actually its not.

What we test is point in time. At 2:45 UTC on Monday installing this
git ref of nova worked.

Noone can reconstruct that today.

I entirely agree with the sentiment you're expressing, but we're not
delivering that sentiment today.

This observation led to yet more IRC discussion and eventually
https://etherpad.openstack.org/p/stable-omg-deps

In short, the proposal is that we:
- stop trying to use install_requires to reproduce exactly what
works, and instead use it to communicate known constraints (> X, Y is
broken etc).
- use a requirements.txt file we create *during* CI to capture
exactly what worked, and also capture the dpkg and rpm versions of
packages that were present when it worked, and so on. So we'll build a
git tree where its history is an audit trail of exactly what worked
for everything that passed CI, formatted to make it really really easy
for other people to consume.

-Rob

--
Robert Collins rbtcollins@hp.com
Distinguished Technologist
HP Converged Cloud


Message: 17
Date: Mon, 13 Apr 2015 02:17:22 +0000
From: joehuang joehuang@huawei.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
5E7A3D1BF5FD014E86E5F971CF446EFF54256EDB@szxema505-mbs.china.huawei.com

Content-Type: text/plain; charset="utf-8"

Hi, Kevin and Joshua,

As my understanding, Tooz only addresses the issue of agent status management, but how to solve the concurrent dynamic load impact on large scale ( for example 100k managed nodes with the dynamic load like security goup rule update, routers_updated, etc )

And one more question is, if we have 100k managed nodes, how to do the partition? Or all nodes will be managed by one Tooz service, like Zookeeper? Can Zookeeper manage 100k nodes status?

Best Regards
Chaoyi Huang ( Joe Huang )

From: Kevin Benton [mailto:blak111@gmail.com]
Sent: Monday, April 13, 2015 3:52 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Timestamps are just one way (and likely the most primitive), using redis (or memcache) key/value and expiry are another (and letting memcache or redis expire using its own internal algorithms), using zookeeper ephemeral nodes[1] are another... The point being that its backend specific and tooz supports varying backends.

Very cool. Is the backend completely transparent so a deployer could choose a service they are comfortable maintaining, or will that change the properties WRT to resiliency of state on node restarts, partitions, etc?

The Nova implementation of Tooz seemed pretty straight-forward, although it looked like it had pluggable drivers for service management already. Before I dig into it much further I'll file a spec on the Neutron side to see if I can get some other cores onboard to do the review work if I push a change to tooz.

On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja@outlook.comharlowja@outlook.com> wrote:
Kevin Benton wrote:
So IIUC tooz would be handling the liveness detection for the agents.
That would be nice to get ride of that logic in Neutron and just
register callbacks for rescheduling the dead.

Where does it store that state, does it persist timestamps to the DB
like Neutron does? If so, how would that scale better? If not, who does
a given node ask to know if an agent is online or offline when making a
scheduling decision?

Timestamps are just one way (and likely the most primitive), using redis (or memcache) key/value and expiry are another (and letting memcache or redis expire using its own internal algorithms), using zookeeper ephemeral nodes[1] are another... The point being that its backend specific and tooz supports varying backends.

However, before (what I assume is) the large code change to implement
tooz, I would like to quantify that the heartbeats are actually a
bottleneck. When I was doing some profiling of them on the master branch
a few months ago, processing a heartbeat took an order of magnitude less
time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A
few query optimizations might buy us a lot more headroom before we have
to fall back to large refactors.

Sure, always good to avoid prematurely optimizing things...

Although this is relevant for u I think anyway:

https://review.openstack.org/#/c/138607/ (same thing/nearly same in nova)...

https://review.openstack.org/#/c/172502/ (a WIP implementation of the latter).

[1] https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes

Kevin Benton wrote:

One of the most common is the heartbeat from each agent. However, I
don't think we can't eliminate them because they are used to determine
if the agents are still alive for scheduling purposes. Did you have
something else in mind to determine if an agent is alive?

Put each agent in a tooz[1] group; have each agent periodically
heartbeat[2], have whoever needs to schedule read the active members of
that group (or use [3] to get notified via a callback), profit...

Pick from your favorite (supporting) driver at:

http://docs.openstack.org/__developer/tooz/compatibility.__html
http://docs.openstack.org/developer/tooz/compatibility.html

[1]
http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
http://docs.openstack.org/developer/tooz/compatibility.html#grouping
[2]
https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315
[3]
http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribehttp://openstack.org?subject:__unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribehttp://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribehttp://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/710773cf/attachment-0001.html


Message: 18
Date: Sun, 12 Apr 2015 20:06:02 -0700
From: Joshua Harlow harlowja@outlook.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID: BLU436-SMTP198723039606360764ABB95D8E70@phx.gbl
Content-Type: text/plain; charset="UTF-8"; format=flowed

Kevin Benton wrote:

Timestamps are just one way (and likely the most primitive), using
redis (or memcache) key/value and expiry are another (and letting
memcache or redis expire using its own internal algorithms), using
zookeeper ephemeral nodes[1] are another... The point being that its
backend specific and tooz supports varying backends.

Very cool. Is the backend completely transparent so a deployer could
choose a service they are comfortable maintaining, or will that change
the properties WRT to resiliency of state on node restarts, partitions, etc?

Of course... we tried to make it 'completely' transparent, but in
reality certain backends (zookeeper which uses a paxos-like algorithm
and redis with sentinel support...) are better (more resilient, more
consistent, handle partitions/restarts better...) than others (memcached
is after all just a distributed cache). This is just the nature of the
game...

The Nova implementation of Tooz seemed pretty straight-forward, although
it looked like it had pluggable drivers for service management already.
Before I dig into it much further I'll file a spec on the Neutron side
to see if I can get some other cores onboard to do the review work if I
push a change to tooz.

Sounds good to me.

On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja@outlook.com
harlowja@outlook.com> wrote:

Kevin Benton wrote:

    So IIUC tooz would be handling the liveness detection for the
    agents.
    That would be nice to get ride of that logic in Neutron and just
    register callbacks for rescheduling the dead.

    Where does it store that state, does it persist timestamps to the DB
    like Neutron does? If so, how would that scale better? If not,
    who does
    a given node ask to know if an agent is online or offline when
    making a
    scheduling decision?


Timestamps are just one way (and likely the most primitive), using
redis (or memcache) key/value and expiry are another (and letting
memcache or redis expire using its own internal algorithms), using
zookeeper ephemeral nodes[1] are another... The point being that its
backend specific and tooz supports varying backends.


    However, before (what I assume is) the large code change to
    implement
    tooz, I would like to quantify that the heartbeats are actually a
    bottleneck. When I was doing some profiling of them on the
    master branch
    a few months ago, processing a heartbeat took an order of
    magnitude less
    time (<50ms) than the 'sync routers' task of the l3 agent
    (~300ms). A
    few query optimizations might buy us a lot more headroom before
    we have
    to fall back to large refactors.


Sure, always good to avoid prematurely optimizing things...

Although this is relevant for u I think anyway:

https://review.openstack.org/#__/c/138607/
<https://review.openstack.org/#/c/138607/> (same thing/nearly same
in nova)...

https://review.openstack.org/#__/c/172502/
<https://review.openstack.org/#/c/172502/> (a WIP implementation of
the latter).

[1]
https://zookeeper.apache.org/__doc/trunk/__zookeeperProgrammers.html#__Ephemeral+Nodes
<https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes>


    Kevin Benton wrote:


         One of the most common is the heartbeat from each agent.
    However, I
         don't think we can't eliminate them because they are used
    to determine
         if the agents are still alive for scheduling purposes. Did
    you have
         something else in mind to determine if an agent is alive?


    Put each agent in a tooz[1] group; have each agent periodically
    heartbeat[2], have whoever needs to schedule read the active
    members of
    that group (or use [3] to get notified via a callback), profit...

    Pick from your favorite (supporting) driver at:

    http://docs.openstack.org/____developer/tooz/compatibility.____html
    <http://docs.openstack.org/__developer/tooz/compatibility.__html>
    <http://docs.openstack.org/__developer/tooz/compatibility.__html
    <http://docs.openstack.org/developer/tooz/compatibility.html>>

    [1]
    http://docs.openstack.org/____developer/tooz/compatibility.____html#grouping
    <http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping>
    <http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
    <http://docs.openstack.org/developer/tooz/compatibility.html#grouping>>
    [2]
    https://github.com/openstack/____tooz/blob/0.13.1/tooz/____coordination.py#L315
    <https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315>
    <https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
    <https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>>
    [3]
    http://docs.openstack.org/____developer/tooz/tutorial/group_____membership.html#watching-____group-changes
    <http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes>
    <http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
    <http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes>>


    __________________________________________________________________________________
    OpenStack Development Mailing List (not for usage questions)
    Unsubscribe:
    OpenStack-dev-request@lists.____openstack.org?subject:____unsubscribe
    <http://openstack.org?subject:__unsubscribe>
    <http://OpenStack-dev-request@__lists.openstack.org?subject:__unsubscribe
    <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>>
    http://lists.openstack.org/____cgi-bin/mailman/listinfo/____openstack-dev
    <http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev>
    <http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
    <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>>

    ______________________________________________________________________________
    OpenStack Development Mailing List (not for usage questions)
    Unsubscribe:
    OpenStack-dev-request@lists.__openstack.org?subject:__unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
    http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
    <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>


______________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.__openstack.org?subject:__unsubscribe
<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>

--
Kevin Benton


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Message: 19
Date: Sun, 12 Apr 2015 20:10:41 -0700
From: Joshua Harlow harlowja@outlook.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID: BLU437-SMTP392487FC50F749804A13DDD8E70@phx.gbl
Content-Type: text/plain; charset="UTF-8"; format=flowed

joehuang wrote:

Hi, Kevin and Joshua,

As my understanding, Tooz only addresses the issue of agent status
management, but how to solve the concurrent dynamic load impact on large
scale ( for example 100k managed nodes with the dynamic load like
security goup rule update, routers_updated, etc )

Yes, that is correct, let's not confuse status/liveness management with
updates... since IMHO they are to very different things (the latter can
be eventually consistent IMHO will the liveness 'question' probably
should not be...).

And one more question is, if we have 100k managed nodes, how to do the
partition? Or all nodes will be managed by one Tooz service, like
Zookeeper? Can Zookeeper manage 100k nodes status?

I can get u some data/numbers from some studies I've seen, but what u
are talking about is highly specific as to what u are doing with
zookeeper... There is no one solution for all the things IMHO; choose
what's best from your tool-belt for each problem...

Best Regards

Chaoyi Huang ( Joe Huang )

From:Kevin Benton [mailto:blak111@gmail.com]
Sent: Monday, April 13, 2015 3:52 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Timestamps are just one way (and likely the most primitive), using redis
(or memcache) key/value and expiry are another (and letting memcache or
redis expire using its own internal algorithms), using zookeeper
ephemeral nodes[1] are another... The point being that its backend
specific and tooz supports varying backends.

Very cool. Is the backend completely transparent so a deployer could
choose a service they are comfortable maintaining, or will that change
the properties WRT to resiliency of state on node restarts, partitions, etc?

The Nova implementation of Tooz seemed pretty straight-forward, although
it looked like it had pluggable drivers for service management already.
Before I dig into it much further I'll file a spec on the Neutron side
to see if I can get some other cores onboard to do the review work if I
push a change to tooz.

On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja@outlook.com
harlowja@outlook.com> wrote:

Kevin Benton wrote:

So IIUC tooz would be handling the liveness detection for the agents.
That would be nice to get ride of that logic in Neutron and just
register callbacks for rescheduling the dead.

Where does it store that state, does it persist timestamps to the DB
like Neutron does? If so, how would that scale better? If not, who does
a given node ask to know if an agent is online or offline when making a
scheduling decision?

Timestamps are just one way (and likely the most primitive), using redis
(or memcache) key/value and expiry are another (and letting memcache or
redis expire using its own internal algorithms), using zookeeper
ephemeral nodes[1] are another... The point being that its backend
specific and tooz supports varying backends.

However, before (what I assume is) the large code change to implement
tooz, I would like to quantify that the heartbeats are actually a
bottleneck. When I was doing some profiling of them on the master branch
a few months ago, processing a heartbeat took an order of magnitude less
time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A
few query optimizations might buy us a lot more headroom before we have
to fall back to large refactors.

Sure, always good to avoid prematurely optimizing things...

Although this is relevant for u I think anyway:

https://review.openstack.org/#/c/138607/ (same thing/nearly same in nova)...

https://review.openstack.org/#/c/172502/ (a WIP implementation of the
latter).

[1]
https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes
https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes

Kevin Benton wrote:

One of the most common is the heartbeat from each agent. However, I
don't think we can't eliminate them because they are used to determine
if the agents are still alive for scheduling purposes. Did you have
something else in mind to determine if an agent is alive?

Put each agent in a tooz[1] group; have each agent periodically
heartbeat[2], have whoever needs to schedule read the active members of
that group (or use [3] to get notified via a callback), profit...

Pick from your favorite (supporting) driver at:

http://docs.openstack.org/__developer/tooz/compatibility.__html
http://docs.openstack.org/developer/tooz/compatibility.html

[1]
http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
http://docs.openstack.org/developer/tooz/compatibility.html#grouping
[2]
https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315
https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315
[3]
http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes
http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://openstack.org?subject:__unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--

Kevin Benton


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Message: 20
Date: Sun, 12 Apr 2015 20:14:51 -0700
From: Joshua Harlow harlowja@outlook.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID: BLU436-SMTP1166EE6F77C4A1152DAEC4AD8E70@phx.gbl
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed

Joshua Harlow wrote:

Kevin Benton wrote:

Timestamps are just one way (and likely the most primitive), using
redis (or memcache) key/value and expiry are another (and letting
memcache or redis expire using its own internal algorithms), using
zookeeper ephemeral nodes[1] are another... The point being that its
backend specific and tooz supports varying backends.

Very cool. Is the backend completely transparent so a deployer could
choose a service they are comfortable maintaining, or will that change
the properties WRT to resiliency of state on node restarts,
partitions, etc?

Of course... we tried to make it 'completely' transparent, but in
reality certain backends (zookeeper which uses a paxos-like algorithm
and redis with sentinel support...) are better (more resilient, more
consistent, handle partitions/restarts better...) than others (memcached
is after all just a distributed cache). This is just the nature of the
game...

And for some more reading fun:

https://aphyr.com/posts/315-call-me-maybe-rabbitmq

https://aphyr.com/posts/291-call-me-maybe-zookeeper

https://aphyr.com/posts/283-call-me-maybe-redis

https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul

.. (aphyr.com has alot of these neat posts)...

The Nova implementation of Tooz seemed pretty straight-forward, although
it looked like it had pluggable drivers for service management already.
Before I dig into it much further I'll file a spec on the Neutron side
to see if I can get some other cores onboard to do the review work if I
push a change to tooz.

Sounds good to me.

On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <harlowja@outlook.com
harlowja@outlook.com> wrote:

Kevin Benton wrote:

So IIUC tooz would be handling the liveness detection for the
agents.
That would be nice to get ride of that logic in Neutron and just
register callbacks for rescheduling the dead.

Where does it store that state, does it persist timestamps to the DB
like Neutron does? If so, how would that scale better? If not,
who does
a given node ask to know if an agent is online or offline when
making a
scheduling decision?

Timestamps are just one way (and likely the most primitive), using
redis (or memcache) key/value and expiry are another (and letting
memcache or redis expire using its own internal algorithms), using
zookeeper ephemeral nodes[1] are another... The point being that its
backend specific and tooz supports varying backends.

However, before (what I assume is) the large code change to
implement
tooz, I would like to quantify that the heartbeats are actually a
bottleneck. When I was doing some profiling of them on the
master branch
a few months ago, processing a heartbeat took an order of
magnitude less
time (<50ms) than the 'sync routers' task of the l3 agent
(~300ms). A
few query optimizations might buy us a lot more headroom before
we have
to fall back to large refactors.

Sure, always good to avoid prematurely optimizing things...

Although this is relevant for u I think anyway:

https://review.openstack.org/#__/c/138607/
https://review.openstack.org/#/c/138607/ (same thing/nearly same
in nova)...

https://review.openstack.org/#__/c/172502/
https://review.openstack.org/#/c/172502/ (a WIP implementation of
the latter).

[1]
https://zookeeper.apache.org/__doc/trunk/__zookeeperProgrammers.html#__Ephemeral+Nodes

https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes

Kevin Benton wrote:

One of the most common is the heartbeat from each agent.
However, I
don't think we can't eliminate them because they are used
to determine
if the agents are still alive for scheduling purposes. Did
you have
something else in mind to determine if an agent is alive?

Put each agent in a tooz[1] group; have each agent periodically
heartbeat[2], have whoever needs to schedule read the active
members of
that group (or use [3] to get notified via a callback), profit...

Pick from your favorite (supporting) driver at:

http://docs.openstack.org/____developer/tooz/compatibility.____html
http://docs.openstack.org/__developer/tooz/compatibility.__html
http://docs.openstack.org/__developer/tooz/compatibility.__html
http://docs.openstack.org/developer/tooz/compatibility.html>

[1]
http://docs.openstack.org/____developer/tooz/compatibility.____html#grouping

http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping

http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping
http://docs.openstack.org/developer/tooz/compatibility.html#grouping>
[2]
https://github.com/openstack/____tooz/blob/0.13.1/tooz/____coordination.py#L315

https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315

https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315

https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315>

[3]
http://docs.openstack.org/____developer/tooz/tutorial/group_____membership.html#watching-____group-changes

http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes

http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes

http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.____openstack.org?subject:____unsubscribe
http://openstack.org?subject:__unsubscribe
http://OpenStack-dev-request@__lists.openstack.org?subject:__unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/____cgi-bin/mailman/listinfo/____openstack-dev
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Message: 21
Date: Sun, 12 Apr 2015 23:23:13 -0400
From: Emilien Macchi emilien@redhat.com
To: puppet-openstack@puppetlabs.com, openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [puppet] Puppet PTL
Message-ID: 552B36A1.6030009@redhat.com
Content-Type: text/plain; charset="utf-8"

On 04/10/2015 06:58 PM, Colleen Murphy wrote:

Just to make it official: since we only had one nominee for PTL, we will
go ahead and name Emilien Macchi as our new PTL without proceeding with
an election process. Thanks, Emilien, for all your hard work and for
taking on this responsibility!

Well, I think this is a great opportunity to me to say something here.
First of all, thank you for your trust and I'll do my best to succeed in
this new position. You know me enough I'm always open to any feedback,
so please do.

Also, I would like to thank our whole community (core & non-core) for
the hard work we all did these years.
I truly think we will succeed under the big tent only if we continue to
work together as a team. But I'm confident, this is part of our DNA
and this is why we are at this stage today.

I'm really looking forward to seeing you at the next Summit.
Thanks,

Colleen (crinkle)

--

To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-openstack+unsubscribe@puppetlabs.com
puppet-openstack+unsubscribe@puppetlabs.com.

--
Emilien Macchi

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/4e811715/attachment-0001.pgp


Message: 22
Date: Mon, 13 Apr 2015 05:03:52 +0000
From: Gary Kotton gkotton@vmware.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] In loving memory of Chris Yeoh
Message-ID: D15127AD.74647%gkotton@vmware.com
Content-Type: text/plain; charset="Windows-1252"

Hi,
I am very saddened to read this. Not only will Chris be missed on a
professional level but on a personal level. He was a real mensh
(http://www.thefreedictionary.com/mensh). He was always helpful and
supportive. Wishing his family a long life.
Thanks
Gary

On 4/13/15, 4:33 AM, "Michael Still" mikal@stillhq.com wrote:

Hi, as promised I now have details of a charity for people to donate
to in Chris' memory:


https://urldefense.proofpoint.com/v2/url?u=http-3A__participate.freetobrea
the.orgsiteTR-3Fpx-3D1582460-26fr-5Fid-3D2710-26pg-3Dpersonal-23.VSscH5S
Ud90&d=AwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzk
WT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRm
sD8&s=B3EgunFqBdY8twmv-iJ7G7xvKZ4Th48oB4HKSv2uGKg&e=

In the words of the family:

"We would prefer that people donate to lung cancer research in lieu of
flowers. Lung cancer has the highest mortality rate out of all the
cancers, and the lowest funding out of all the cancers. There is a
stigma attached that lung cancer is a smoker's disease, and that
sufferers deserve their fate. They bring it on through lifestyle
choice. Except that Chris has never smoked in his life, like a
surprisingly large percentage of lung cancer sufferers. These people
suffer for the incorrect beliefs of the masses, and those that are
left behind are equally innocent. We shouldn't be doing this now. He
shouldn't be gone. We need to do more to fix this. There will be
charity envelopes available at the funeral, or you can choose your
preferred research to fund, should you wish to do so. You have our
thanks."

Michael

On Wed, Apr 8, 2015 at 2:49 PM, Michael Still mikal@stillhq.com wrote:

It is my sad duty to inform the community that Chris Yeoh passed away
this
morning. Chris leaves behind a daughter Alyssa, aged 6, who I hope will
remember Chris as the clever and caring person that I will remember him
as.
I haven?t had a chance to confirm with the family if they want flowers
or a
donation to a charity. As soon as I know those details I will reply to
this
email.

Chris worked on open source for a very long time, with OpenStack being
just
the most recent in a long chain of contributions. He worked tirelessly
on
his contributions to Nova, including mentoring other developers. He was
dedicated to the cause, with a strong vision of what OpenStack could
become.
He even named his cat after the project.

Chris might be the only person to have ever sent an email to his
coworkers
explaining what his code review strategy would be after brain surgery.
It
takes phenomenal strength to carry on in the face of that kind of
adversity,
but somehow he did. Frankly, I think I would have just sat on the beach.

Chris was also a contributor to the Linux Standards Base (LSB), where he
helped improve the consistency and interoperability between Linux
distributions. He ran the ?Hackfest? programming contests for a number
of
years at Australia?s open source conference -- linux.conf.au. He
supported
local Linux user groups in South Australia and Canberra, including
involvement at installfests and speaking at local meetups. He competed
in a
programming challenge called Loki Hack, and beat out the world to win
the
event[1].

Alyssa?s memories of her dad need to last her a long time, so we?ve
decided
to try and collect some fond memories of Chris to help her along the
way. If
you feel comfortable doing so, please contribute a memory or two at

https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_form
sd1kX-2DePqAO7Cuudppwqz1cqgBXAsJx27GkdM-2DeCZ0c1V8_viewform&d=AwIGaQ&c=
Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTe
q9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRmsD8&s=iihsaOMe
lNeIR3VZapWKjr5KLgMQArZ3nifKDo1yy8o&e=

Chris was humble, helpful and honest. The OpenStack and broader Open
Source
communities are poorer for his passing.

Michael

[1]
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lokigames.com_hac
k_&d=AwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkW
T5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRm
sD8&s=9SJI7QK-jzCsVUN2hTXSthqiXNEbq2Fvl9JqQiX9tfo&e=

--
Rackspace Australia


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Message: 23
Date: Mon, 13 Apr 2015 02:21:59 -0400 (EDT)
From: Attila Fazekas afazekas@redhat.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
147828721.17010859.1428906119294.JavaMail.zimbra@redhat.com
Content-Type: text/plain; charset=utf-8

----- Original Message -----

From: "Kevin Benton" blak111@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Sent: Sunday, April 12, 2015 4:17:29 AM
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

So IIUC tooz would be handling the liveness detection for the agents. That
would be nice to get ride of that logic in Neutron and just register
callbacks for rescheduling the dead.

Where does it store that state, does it persist timestamps to the DB like
Neutron does? If so, how would that scale better? If not, who does a given
node ask to know if an agent is online or offline when making a scheduling
decision?

You might find interesting the proposed solution in this bug:
https://bugs.launchpad.net/nova/+bug/1437199

However, before (what I assume is) the large code change to implement tooz, I
would like to quantify that the heartbeats are actually a bottleneck. When I
was doing some profiling of them on the master branch a few months ago,
processing a heartbeat took an order of magnitude less time (<50ms) than the
'sync routers' task of the l3 agent (~300ms). A few query optimizations
might buy us a lot more headroom before we have to fall back to large
refactors.
Kevin Benton wrote:

One of the most common is the heartbeat from each agent. However, I
don't think we can't eliminate them because they are used to determine
if the agents are still alive for scheduling purposes. Did you have
something else in mind to determine if an agent is alive?

Put each agent in a tooz[1] group; have each agent periodically heartbeat[2],
have whoever needs to schedule read the active members of that group (or use
[3] to get notified via a callback), profit...

Pick from your favorite (supporting) driver at:

http://docs.openstack.org/ developer/tooz/compatibility. html

[1] http://docs.openstack.org/ developer/tooz/compatibility. html#grouping
[2] https://github.com/openstack/ tooz/blob/0.13.1/tooz/ coordination.py#L315
[3] http://docs.openstack.org/ developer/tooz/tutorial/group_
membership.html#watching- group-changes


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists. openstack.org?subject: unsubscribe
http://lists.openstack.org/ cgi-bin/mailman/listinfo/ openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Message: 24
Date: Sun, 12 Apr 2015 23:26:10 -0700
From: Kevin Benton blak111@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Neutron] Regarding neutron bug # 1432582
Message-ID:
CAO_F6JP673t3trD4QudmHZKgUekPGVtqihg3zDHyhGf=7HpQrQ@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

I would like to see some form of this merged at least as an error message.
If a server has a bad CMOS battery and suffers a power outage, it's clock
could easily be several years behind. In that scenario, the NTP daemon
could refuse to sync due to a sanity check.

On Wed, Apr 8, 2015 at 10:46 AM, Sudipto Biswas <sbiswas7@linux.vnet.ibm.com

wrote:

Hi Guys, I'd really appreciate your feedback on this.

Thanks,
Sudipto

On Monday 30 March 2015 12:11 PM, Sudipto Biswas wrote:

Someone from my team had installed the OS on baremetal with a wrong 'date'
When this node was added to the Openstack controller, the logs from the
neutron-agent on the compute node showed - "AMQP connected". But the
neutron
agent-list command would not list this agent at all.

I could figure out the problem when the neutron-server debug logs were
enabled
and it vaguely pointed at the rejection of AMQP connections due to a
timestamp
miss match. The neutron-server was treating these requests as stale due
to the
timestamp of the node being behind the neutron-server. However, there's no
good way to detect this if the agent runs on a node which is ahead of
time.

I recently raised a bug here: https://bugs.launchpad.net/
neutron/+bug/1432582

And tried to resolve this with the review:
https://review.openstack.org/#/c/165539/

It went through quite a few +2s after 15 odd patch sets but we still are
not
in common ground w.r.t addressing this situation.

My fix tries to log better and throw up an exception to the neutron agent
on
FIRST time boot of the agent for better detection of the problem.

I would like to get your thoughts on this fix. Whether this seems legit
to have
the fix per the patch OR could you suggest a approach to tackle this OR
suggest
just abandoning the change.


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150412/b8eef87a/attachment-0001.html


Message: 25
Date: Mon, 13 Apr 2015 14:51:05 +0800
From: "=?gb18030?B?NDQ5MTcxMzQy?=" 449171342@qq.com
To: "=?gb18030?B?b3BlbnN0YWNrLWRldg==?="
openstack-dev@lists.openstack.org
Subject: [openstack-dev] ?magnum?About clean none use container imag
Message-ID: tencent_1B594E50513691B07D1B8F9E@qq.com
Content-Type: text/plain; charset="gb18030"

From now on magnum had container create and delete api .The container create api will pull docker image from docker-registry.But the container delete api didn't delete image.It will let the image remain even though didn't had container use it.Is it much better we can clear the image in someway?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/7072441b/attachment-0001.html


Message: 26
Date: Mon, 13 Apr 2015 15:09:18 +0800
From: Jay Lau jay.lau.513@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] ?magnum?About clean none use container
imag
Message-ID:
CAFyztAG6EOd97Ho5zir7U0Gmywq=J=AErH9WJAi-2yswx2E5Ng@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

Interesting topic, pulling image is time consuming, so someone might not
want to delete the container; But for some cases, if the image was not
used, then it is better to remove them from disk to release space. You may
want to send out an email to [openstack][magnum] ML to get more feedback ;-)

2015-04-13 14:51 GMT+08:00 449171342 449171342@qq.com:

From now on magnum had container create and delete api .The container create api will pull docker image from docker-registry.But the container delete api didn't delete image.It will let the image remain even though didn't had container use it.Is it much better we can clear the image in someway?


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks,

Jay Lau (Guangya Liu)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/38091051/attachment-0001.html


Message: 27
Date: Mon, 13 Apr 2015 03:18:34 -0400 (EDT)
From: Attila Fazekas afazekas@redhat.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Message-ID:
1115924511.17061105.1428909514804.JavaMail.zimbra@redhat.com
Content-Type: text/plain; charset=utf-8

----- Original Message -----

From: "joehuang" joehuang@huawei.com
To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Sent: Sunday, April 12, 2015 1:20:48 PM
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Hi, Kevin,

I assumed that all agents are connected to same IP address of RabbitMQ, then
the connection will exceed the port ranges limitation.

https://news.ycombinator.com/item?id=1571300

"TCP connections are identified by the (src ip, src port, dest ip, dest port) tuple."

"The server doesn't need multiple IPs to handle > 65535 connections. All the server connections to a given IP are to the same port. For a given client, the unique key for an http connection is (client-ip, PORT, server-ip, 80). The only number that can vary is PORT, and that's a value on the client. So, the client is limited to 65535 connections to the server. But, a second client could also have another 65K connections to the same server-ip:port."

For a RabbitMQ cluster, for sure the client can connect to any one of member
in the cluster, but in this case, the client has to be designed in fail-safe
manner: the client should be aware of the cluster member failure, and
reconnect to other survive member. No such mechnism has been implemented
yet.

Other way is to use LVS or DNS based like load balancer, or something else.
If you put one load balancer ahead of a cluster, then we have to take care
of the port number limitation, there are so many agents will require
connection concurrently, 100k level, and the requests can not be rejected.

Best Regards

Chaoyi Huang ( joehuang )

From: Kevin Benton [blak111@gmail.com]
Sent: 12 April 2015 9:59
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

The TCP/IP stack keeps track of connections as a combination of IP + TCP
port. The two byte port limit doesn't matter unless all of the agents are
connecting from the same IP address, which shouldn't be the case unless
compute nodes connect to the rabbitmq server via one IP address running port
address translation.

Either way, the agents don't connect directly to the Neutron server, they
connect to the rabbit MQ cluster. Since as many Neutron server processes can
be launched as necessary, the bottlenecks will likely show up at the
messaging or DB layer.

On Sat, Apr 11, 2015 at 6:46 PM, joehuang < joehuang@huawei.com > wrote:

As Kevin talking about agents, I want to remind that in TCP/IP stack, port (
not Neutron Port ) is a two bytes field, i.e. port ranges from 0 ~ 65535,
supports maximum 64k port number.

" above 100k managed node " means more than 100k L2 agents/L3 agents... will
be alive under Neutron.

Want to know the detail design how to support 99.9% possibility for scaling
Neutron in this way, and PoC and test would be a good support for this idea.

"I'm 99.9% sure, for scaling above 100k managed node,
we do not really need to split the openstack to multiple smaller openstack,
or use significant number of extra controller machine."

Best Regards

Chaoyi Huang ( joehuang )

From: Kevin Benton [ blak111@gmail.com ]
Sent: 11 April 2015 12:34
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Which periodic updates did you have in mind to eliminate? One of the few
remaining ones I can think of is sync_routers but it would be great if you
can enumerate the ones you observed because eliminating overhead in agents
is something I've been working on as well.

One of the most common is the heartbeat from each agent. However, I don't
think we can't eliminate them because they are used to determine if the
agents are still alive for scheduling purposes. Did you have something else
in mind to determine if an agent is alive?

On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas < afazekas@redhat.com >
wrote:

I'm 99.9% sure, for scaling above 100k managed node,
we do not really need to split the openstack to multiple smaller openstack,
or use significant number of extra controller machine.

The problem is openstack using the right tools SQL/AMQP/(zk),
but in a wrong way.

For example.:
Periodic updates can be avoided almost in all cases

The new data can be pushed to the agent just when it needed.
The agent can know when the AMQP connection become unreliable (queue or
connection loose),
and needs to do full sync.
https://bugs.launchpad.net/neutron/+bug/1438159

Also the agents when gets some notification, they start asking for details
via the
AMQP -> SQL. Why they do not know it already or get it with the notification
?

----- Original Message -----

From: "Neil Jerram" < Neil.Jerram@metaswitch.com >
To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev@lists.openstack.org >
Sent: Thursday, April 9, 2015 5:01:45 PM
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Hi Joe,

Many thanks for your reply!

On 09/04/15 03:34, joehuang wrote:

Hi, Neil,

From theoretic, Neutron is like a "broadcast" domain, for example,
enforcement of DVR and security group has to touch each regarding host
where there is VM of this project resides. Even using SDN controller, the
"touch" to regarding host is inevitable. If there are plenty of physical
hosts, for example, 10k, inside one Neutron, it's very hard to overcome
the "broadcast storm" issue under concurrent operation, that's the
bottleneck for scalability of Neutron.

I think I understand that in general terms - but can you be more
specific about the broadcast storm? Is there one particular message
exchange that involves broadcasting? Is it only from the server to
agents, or are there 'broadcasts' in other directions as well?

(I presume you are talking about control plane messages here, i.e.
between Neutron components. Is that right? Obviously there can also be
broadcast storm problems in the data plane - but I don't think that's
what you are talking about here.)

We need layered architecture in Neutron to solve the "broadcast domain"
bottleneck of scalability. The test report from OpenStack cascading shows
that through layered architecture "Neutron cascading", Neutron can
supports up to million level ports and 100k level physical hosts. You can
find the report here:
http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

Many thanks, I will take a look at this.

"Neutron cascading" also brings extra benefit: One cascading Neutron can
have many cascaded Neutrons, and different cascaded Neutron can leverage
different SDN controller, maybe one is ODL, the other one is
OpenContrail.

----------------Cascading Neutron-------------------
/ \
--cascaded Neutron-- --cascaded Neutron-----
| |
---------ODL------ ----OpenContrail--------

And furthermore, if using Neutron cascading in multiple data centers, the
DCI controller (Data center inter-connection controller) can also be used
under cascading Neutron, to provide NaaS ( network as a service ) across
data centers.

---------------------------Cascading Neutron--------------------------
/ | \
--cascaded Neutron-- -DCI controller- --cascaded Neutron-----
| | |
---------ODL------ | ----OpenContrail--------
|
--(Data center 1)-- --(DCI networking)-- --(Data center 2)--

Is it possible for us to discuss this in OpenStack Vancouver summit?

Most certainly, yes. I will be there from mid Monday afternoon through
to end Friday. But it will be my first summit, so I have no idea yet as
to how I might run into you - please can you suggest!

Best Regards
Chaoyi Huang ( Joe Huang )

Regards,
Neil


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Message: 28
Date: Mon, 13 Apr 2015 15:23:21 +0800
From: wu jiang wingwj@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] In loving memory of Chris Yeoh
Message-ID:
CAEfJYtTacCxQ2MQXzqQQ2b_VoX6o_Pxr=WmoaXB=HDp5EmZ92w@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

What bad news.. Chris helped me a lot, we lost a mentor and friend.
May God bless his/her soul.

WingWJ

On Mon, Apr 13, 2015 at 1:03 PM, Gary Kotton gkotton@vmware.com wrote:

Hi,
I am very saddened to read this. Not only will Chris be missed on a
professional level but on a personal level. He was a real mensh
(http://www.thefreedictionary.com/mensh). He was always helpful and
supportive. Wishing his family a long life.
Thanks
Gary

On 4/13/15, 4:33 AM, "Michael Still" mikal@stillhq.com wrote:

Hi, as promised I now have details of a charity for people to donate
to in Chris' memory:

https://urldefense.proofpoint.com/v2/url?u=http-3A__participate.freetobrea

the.orgsiteTR-3Fpx-3D1582460-26fr-5Fid-3D2710-26pg-3Dpersonal-23.VSscH5S
Ud90&d=AwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzk
WT5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRm
sD8&s=B3EgunFqBdY8twmv-iJ7G7xvKZ4Th48oB4HKSv2uGKg&e=

In the words of the family:

"We would prefer that people donate to lung cancer research in lieu of
flowers. Lung cancer has the highest mortality rate out of all the
cancers, and the lowest funding out of all the cancers. There is a
stigma attached that lung cancer is a smoker's disease, and that
sufferers deserve their fate. They bring it on through lifestyle
choice. Except that Chris has never smoked in his life, like a
surprisingly large percentage of lung cancer sufferers. These people
suffer for the incorrect beliefs of the masses, and those that are
left behind are equally innocent. We shouldn't be doing this now. He
shouldn't be gone. We need to do more to fix this. There will be
charity envelopes available at the funeral, or you can choose your
preferred research to fund, should you wish to do so. You have our
thanks."

Michael

On Wed, Apr 8, 2015 at 2:49 PM, Michael Still mikal@stillhq.com wrote:

It is my sad duty to inform the community that Chris Yeoh passed away
this
morning. Chris leaves behind a daughter Alyssa, aged 6, who I hope will
remember Chris as the clever and caring person that I will remember him
as.
I haven?t had a chance to confirm with the family if they want flowers
or a
donation to a charity. As soon as I know those details I will reply to
this
email.

Chris worked on open source for a very long time, with OpenStack being
just
the most recent in a long chain of contributions. He worked tirelessly
on
his contributions to Nova, including mentoring other developers. He was
dedicated to the cause, with a strong vision of what OpenStack could
become.
He even named his cat after the project.

Chris might be the only person to have ever sent an email to his
coworkers
explaining what his code review strategy would be after brain surgery.
It
takes phenomenal strength to carry on in the face of that kind of
adversity,
but somehow he did. Frankly, I think I would have just sat on the beach.

Chris was also a contributor to the Linux Standards Base (LSB), where he
helped improve the consistency and interoperability between Linux
distributions. He ran the ?Hackfest? programming contests for a number
of
years at Australia?s open source conference -- linux.conf.au. He
supported
local Linux user groups in South Australia and Canberra, including
involvement at installfests and speaking at local meetups. He competed
in a
programming challenge called Loki Hack, and beat out the world to win
the
event[1].

Alyssa?s memories of her dad need to last her a long time, so we?ve
decided
to try and collect some fond memories of Chris to help her along the
way. If
you feel comfortable doing so, please contribute a memory or two at

https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_form

sd1kX-2DePqAO7Cuudppwqz1cqgBXAsJx27GkdM-2DeCZ0c1V8_viewform&d=AwIGaQ&c=
Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkWT5jqz9JYBk8YTe
q9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRmsD8&s=iihsaOMe
lNeIR3VZapWKjr5KLgMQArZ3nifKDo1yy8o&e=

Chris was humble, helpful and honest. The OpenStack and broader Open
Source
communities are poorer for his passing.

Michael

[1]

https://urldefense.proofpoint.com/v2/url?u=http-3A__www.lokigames.com_hac

k_&d=AwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=VlZxHpZBmzzkW
T5jqz9JYBk8YTeq9N3-diTlNj4GyNc&m=IFwED7YYaddl7JbqZ5OLChF6gtEGxYkxfFHwjWRm
sD8&s=9SJI7QK-jzCsVUN2hTXSthqiXNEbq2Fvl9JqQiX9tfo&e=

--
Rackspace Australia


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/9e436bfd/attachment-0001.html


Message: 29
Date: Mon, 13 Apr 2015 04:03:49 -0400 (EDT)
From: Victor Stinner vstinner@redhat.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully
Python 3 compatible
Message-ID:
868807730.15816798.1428912229323.JavaMail.zimbra@redhat.com
Content-Type: text/plain; charset=utf-8

Worth noting we've already switched to using PyMySQL in nodepool,
storyboard and some of the subunit2sql tooling. It's been working
out great so far.

Great. Did you notice a performance regression? Mike wrote that PyMySQL is much slower than MySQL-Python.

Victor


Message: 30
Date: Mon, 13 Apr 2015 11:22:38 +0300
From: Anastasia Urlapova aurlapova@mirantis.com
To: Andrey Sledzinskiy asledzinskiy@mirantis.com,
"mos-qa@mirantis.com" mos-qa@mirantis.com, "OpenStack Development
Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Fuel] Nominate Andrey Skedzinskiy for
fuel-qa(devops) core
Message-ID:
CAC+Xjbbpm9jGap+j=qYayiJY8teqbRkJPjvZ0qieYVDDUFphOg@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

Guys,
I would like to nominate Andrey Skedzinskiy[1] for
fuel-qa[2]/fuel-devops[3] core team.

Andrey is one of the strongest reviewers, under his watchful eye are such
features as:
- updrade/rollback master node
- collect usage information
- OS patching
- UI tests
and others

Please vote for Andrey!

Nastya.

[1]http://stackalytics.com/?project_type=stackforge&user_id=asledzinskiy
[2]https://github.com/stackforge/fuel-qa
[3]https://github.com/stackforge/fuel-devops
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/92ee8866/attachment-0001.html


Message: 31
Date: Mon, 13 Apr 2015 11:37:44 +0300
From: Alexander Kislitsky akislitsky@mirantis.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Cc: "mos-qa@mirantis.com" mos-qa@mirantis.com
Subject: Re: [openstack-dev] [Fuel] Nominate Andrey Skedzinskiy for
fuel-qa(devops) core
Message-ID:
CAHWr6fkksK1-i3vSKvC01X5F7sGBQAgBt9W6q4CJjtQqNFLv=g@mail.gmail.com
Content-Type: text/plain; charset="utf-8"

Andrew shows great attention to the details. +1 for him.

On Mon, Apr 13, 2015 at 11:22 AM, Anastasia Urlapova <aurlapova@mirantis.com

wrote:

Guys,
I would like to nominate Andrey Skedzinskiy[1] for
fuel-qa[2]/fuel-devops[3] core team.

Andrey is one of the strongest reviewers, under his watchful eye are such
features as:
- updrade/rollback master node
- collect usage information
- OS patching
- UI tests
and others

Please vote for Andrey!

Nastya.

[1]http://stackalytics.com/?project_type=stackforge&user_id=asledzinskiy
[2]https://github.com/stackforge/fuel-qa
[3]https://github.com/stackforge/fuel-devops


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/8239f0df/attachment-0001.html


Message: 32
Date: Mon, 13 Apr 2015 10:52:11 +0200
From: Miguel Angel Ajo Pelayo majopela@redhat.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all][pbr] splitting our deployment vs
install dependencies
Message-ID: 818AF012-6625-4766-B883-712D36A5851C@redhat.com
Content-Type: text/plain; charset=us-ascii

On 13/4/2015, at 3:53, Robert Collins robertc@robertcollins.net wrote:

On 13 April 2015 at 13:09, Robert Collins robertc@robertcollins.net wrote:

On 13 April 2015 at 12:53, Monty Taylor mordred@inaugust.com wrote:

What we have in the gate is the thing that produces the artifacts that
someone installing using the pip tool would get. Shipping anything with
those artifacts other that a direct communication of what we tested is
just mean to our end users.

Actually its not.

What we test is point in time. At 2:45 UTC on Monday installing this
git ref of nova worked.

Noone can reconstruct that today.

I entirely agree with the sentiment you're expressing, but we're not
delivering that sentiment today.

This observation led to yet more IRC discussion and eventually
https://etherpad.openstack.org/p/stable-omg-deps

In short, the proposal is that we:
- stop trying to use install_requires to reproduce exactly what
works, and instead use it to communicate known constraints (> X, Y is
broken etc).
- use a requirements.txt file we create *during* CI to capture
exactly what worked, and also capture the dpkg and rpm versions of
packages that were present when it worked, and so on. So we'll build a
git tree where its history is an audit trail of exactly what worked
for everything that passed CI, formatted to make it really really easy
for other people to consume.

That sounds like a very neat idea, this way we could look back, and backtrack
to discover which package version change breaks the system.

Miguel Angel Ajo


Message: 33
Date: Mon, 13 Apr 2015 08:51:39 +0000
From: Wangbibo wangbibo@huawei.com
To: "OpenStack Development Mailing List (not for usage questions)"
openstack-dev@lists.openstack.org
Subject: [openstack-dev] ??: [neutron] Neutron scaling datapoints?
Message-ID:
EA5BE4191F3B3F4AB624BDC5B27A31A74E45D0A8@nkgeml504-mbs.china.huawei.com

Content-Type: text/plain; charset="utf-8"

Hi Kevin,

Totally agree with you that heartbeat from each agent is something that we cannot eliminate currently. Agent status depends on it, and further scheduler and HA depends on agent status.

I proposed a Liberty spec for introducing open framework/pluggable agent status drivers.[1][2] It allows us to use some other 3rd party backend to monitor agent status, such as zookeeper, memcached. Meanwhile, it guarantees backward compatibility so that users could still use db-based status monitoring mechanism as their default choice.

Base on that, we may do further optimization on issues Attila and you mentioned. Thanks.

[1] BP - https://blueprints.launchpad.net/neutron/+spec/agent-group-and-status-drivers
[2] Liberty Spec proposed - https://review.openstack.org/#/c/168921/

Best,
Robin

???: Kevin Benton [mailto:blak111@gmail.com]
????: 2015?4?11? 12:35
???: OpenStack Development Mailing List (not for usage questions)
??: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Which periodic updates did you have in mind to eliminate? One of the few remaining ones I can think of is sync_routers but it would be great if you can enumerate the ones you observed because eliminating overhead in agents is something I've been working on as well.

One of the most common is the heartbeat from each agent. However, I don't think we can't eliminate them because they are used to determine if the agents are still alive for scheduling purposes. Did you have something else in mind to determine if an agent is alive?

On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas <afazekas@redhat.comafazekas@redhat.com> wrote:
I'm 99.9% sure, for scaling above 100k managed node,
we do not really need to split the openstack to multiple smaller openstack,
or use significant number of extra controller machine.

The problem is openstack using the right tools SQL/AMQP/(zk),
but in a wrong way.

For example.:
Periodic updates can be avoided almost in all cases

The new data can be pushed to the agent just when it needed.
The agent can know when the AMQP connection become unreliable (queue or connection loose),
and needs to do full sync.
https://bugs.launchpad.net/neutron/+bug/1438159

Also the agents when gets some notification, they start asking for details via the
AMQP -> SQL. Why they do not know it already or get it with the notification ?

----- Original Message -----

From: "Neil Jerram" <Neil.Jerram@metaswitch.comNeil.Jerram@metaswitch.com>
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.orgopenstack-dev@lists.openstack.org>
Sent: Thursday, April 9, 2015 5:01:45 PM
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?

Hi Joe,

Many thanks for your reply!

On 09/04/15 03:34, joehuang wrote:

Hi, Neil,

From theoretic, Neutron is like a "broadcast" domain, for example,
enforcement of DVR and security group has to touch each regarding host
where there is VM of this project resides. Even using SDN controller, the
"touch" to regarding host is inevitable. If there are plenty of physical
hosts, for example, 10k, inside one Neutron, it's very hard to overcome
the "broadcast storm" issue under concurrent operation, that's the
bottleneck for scalability of Neutron.

I think I understand that in general terms - but can you be more
specific about the broadcast storm? Is there one particular message
exchange that involves broadcasting? Is it only from the server to
agents, or are there 'broadcasts' in other directions as well?

(I presume you are talking about control plane messages here, i.e.
between Neutron components. Is that right? Obviously there can also be
broadcast storm problems in the data plane - but I don't think that's
what you are talking about here.)

We need layered architecture in Neutron to solve the "broadcast domain"
bottleneck of scalability. The test report from OpenStack cascading shows
that through layered architecture "Neutron cascading", Neutron can
supports up to million level ports and 100k level physical hosts. You can
find the report here:
http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

Many thanks, I will take a look at this.

"Neutron cascading" also brings extra benefit: One cascading Neutron can
have many cascaded Neutrons, and different cascaded Neutron can leverage
different SDN controller, maybe one is ODL, the other one is OpenContrail.

----------------Cascading Neutron-------------------
/ \
--cascaded Neutron-- --cascaded Neutron-----
| |
---------ODL------ ----OpenContrail--------

And furthermore, if using Neutron cascading in multiple data centers, the
DCI controller (Data center inter-connection controller) can also be used
under cascading Neutron, to provide NaaS ( network as a service ) across
data centers.

---------------------------Cascading Neutron--------------------------
/ | \
--cascaded Neutron-- -DCI controller- --cascaded Neutron-----
| | |
---------ODL------ | ----OpenContrail--------
|
--(Data center 1)-- --(DCI networking)-- --(Data center 2)--

Is it possible for us to discuss this in OpenStack Vancouver summit?

Most certainly, yes. I will be there from mid Monday afternoon through
to end Friday. But it will be my first summit, so I have no idea yet as
to how I might run into you - please can you suggest!

Best Regards
Chaoyi Huang ( Joe Huang )

Regards,
Neil


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribehttp://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribehttp://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstack.org/pipermail/openstack-dev/attachments/20150413/2569d73e/attachment.html



OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

End of OpenStack-dev Digest, Vol 36, Issue 32
*********************************************__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

asked Apr 14, 2015 in openstack-dev by 449171342 (180 points)   1
retagged Apr 14, 2015 by admin

2 Responses

0 votes

Indeed image pull be time-consuming.
Is it much better to supply a magnum api let end-user can clean image that not used?
Clean or not clean is decide by the end-user. Or we can supply a method let
end-user can decide whether or not clean image.

Date: Mon, 13 Apr 2015 14:51:05 +0800
From: "=?gb18030?B?NDQ5MTcxMzQy?=" 449171342@qq.com
To: "=?gb18030?B?b3BlbnN0YWNrLWRldg==?="

Subject: [openstack-dev] ?magnum?About clean none use container imag
Message-ID: tencent_1B594E50513691B07D1B8F9E@qq.com
Content-Type: text/plain; charset="gb18030"

From now on magnum had container create and delete api .The container create api will pull docker image from docker-registry.But the container delete api didn't delete image.It will let the image remain even though didn't had container use it.Is it much better we can clear the image in someway?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: ;


Message: 26
Date: Mon, 13 Apr 2015 15:09:18 +0800
From: Jay Lau jay.lau.513@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)"

Subject: Re: [openstack-dev] ?magnum?About clean none use container
imag
Message-ID:

Content-Type: text/plain; charset="utf-8"

Interesting topic, pulling image is time consuming, so someone might not
want to delete the container; But for some cases, if the image was not
used, then it is better to remove them from disk to release space. You may
want to send out an email to [openstack][magnum] ML to get more feedback ;-)

2015-04-13 14:51 GMT+08:00 449171342 449171342@qq.com:

From now on magnum had container create and delete api .The container create api will pull docker image from docker-registry.But the container delete api didn't delete image.It will let the image remain even though didn't had container use it.Is it much better we can clear the image in someway?


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks,

Jay Lau (Guangya Liu)__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Apr 14, 2015 by 449171342 (180 points)   1
0 votes

I agree with Fang that controlling image purge functionality with optional parameters offers the flexibility needed to balance performance and security concerns.

Perhaps there are two API calls where we could implement such an option:

/v1/container/post (create)
/vi/container/delete

That way you could default to either “secure” or “performance”, and optionally select the alternate depending on the value of the parameter you supply when the API is called.

Note that Docker (unless forced) will not allow the deletion of a container image until all containers created from that image are also deleted. We will want to be sure we have adequate tests to be sure that related uses are tested, and have exceptions that raise actionable error messages when raised. Let’s not have an “unknown exception” when I try to delete a container, and I get an error when it tries to delete the container image, but can’t because there is another container running that used the same image.

Finally, we might also want to allow a user to specify whether untagged parents should be pruned or not (ie: docker rmi —no-prune=true)

Adrian

On Apr 14, 2015, at 2:31 AM, Fang Fenghua 449171342@qq.com wrote:

Indeed image pull be time-consuming.
Is it much better to supply a magnum api let end-user can clean image that not used?
Clean or not clean is decide by the end-user. Or we can supply a method let
end-user can decide whether or not clean image.

Date: Mon, 13 Apr 2015 14:51:05 +0800
From: "=?gb18030?B?NDQ5MTcxMzQy?=" 449171342@qq.com
To: "=?gb18030?B?b3BlbnN0YWNrLWRldg==?="

Subject: [openstack-dev] ?magnum?About clean none use container imag
Message-ID: tencent_1B594E50513691B07D1B8F9E@qq.com
Content-Type: text/plain; charset="gb18030"

From now on magnum had container create and delete api .The container create api will pull docker image from docker-registry.But the container delete api didn't delete image.It will let the image remain even though didn't had container use it.Is it much better we can clear the image in someway?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: ;


Message: 26
Date: Mon, 13 Apr 2015 15:09:18 +0800
From: Jay Lau jay.lau.513@gmail.com
To: "OpenStack Development Mailing List (not for usage questions)"

Subject: Re: [openstack-dev] ?magnum?About clean none use container
imag
Message-ID:

Content-Type: text/plain; charset="utf-8"

Interesting topic, pulling image is time consuming, so someone might not
want to delete the container; But for some cases, if the image was not
used, then it is better to remove them from disk to release space. You may
want to send out an email to [openstack][magnum] ML to get more feedback ;-)

2015-04-13 14:51 GMT+08:00 449171342 449171342@qq.com:

From now on magnum had container create and delete api .The container create api will pull docker image from docker-registry.But the container delete api didn't delete image.It will let the image remain even though didn't had container use it.Is it much better we can clear the image in someway?


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Thanks,

Jay Lau (Guangya Liu)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 14, 2015 by Adrian_Otto (11,060 points)   2 4 8
...