settingsLogin | Registersettings

[openstack-dev] [devstack] experimenting with systemd for services

0 votes

At the PTG clayg brought up an excellent question about what the
expected flow was to restart a bunch of services in devstack after a
code changes that impacts many of them (be it common code, or a
library). People had created a bunch of various screen hacks over the
years, but screen is flakey, and this is definitely not an ideal workflow.

Over lunch clayg, clarkb, and I chatted about some options. Clark
brought up the idea of doing systemd units for all of the services. A
couple of weeks ago I decided it was time for me to understand systemd
better anyway, and started playing around with what this might look like.

The results landed here https://review.openstack.org/#/c/448323/.
Documentation is here
http://git.openstack.org/cgit/openstack-dev/devstack/tree/SYSTEMD.rst

This is currently an opt in. All the services in base devstack however
do work in this new model, and I and a few others have been using this
mode the last week or so. It's honestly really great. Working on
oslo.log changes it's now:

pip install -U .
sudo systemctl restart devstack@*

And the change is now in all your services.

There is also an oslo.log change for native systemd journal support
(https://review.openstack.org/#/c/451525/), which once that has landed
and been released, will let us do some neat query of the journal during
development to see slices across services like this
https://dague.net/2017/03/30/in-praise-of-systemd/.

ACTION REQUIRED:

If you maintain a devstack plugin that starts any services, now would be
a great time to test to see if this works for them. The biggest issue is
that the commands sent to run_process need to be absolute pathed.

My hope is that by end of cycle this is going to be the default mode in
devstack for both the gate and development, which eliminates one major
difference between the two. I'm also hoping that we'll be able to keep
and archive the journals from the runs, so you can download and query
those directly. Especially once the oslo.log enhancements are there to
add the additional structured data.

-Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Apr 6, 2017 in openstack-dev by Sean_Dague (66,200 points)   4 11 18

3 Responses

0 votes

On Wed, Apr 5, 2017 at 9:14 PM Sean Dague sean@dague.net wrote:

At the PTG clayg brought up an excellent question about what the
expected flow was to restart a bunch of services in devstack after a
code changes that impacts many of them (be it common code, or a
library). People had created a bunch of various screen hacks over the
years, but screen is flakey, and this is definitely not an ideal workflow.

Over lunch clayg, clarkb, and I chatted about some options. Clark
brought up the idea of doing systemd units for all of the services. A
couple of weeks ago I decided it was time for me to understand systemd
better anyway, and started playing around with what this might look like.

The results landed here https://review.openstack.org/#/c/448323/.
Documentation is here
http://git.openstack.org/cgit/openstack-dev/devstack/tree/SYSTEMD.rst

This is currently an opt in. All the services in base devstack however
do work in this new model, and I and a few others have been using this
mode the last week or so. It's honestly really great. Working on
oslo.log changes it's now:

pip install -U .
sudo systemctl restart devstack@*

And the change is now in all your services.

There is also an oslo.log change for native systemd journal support
(https://review.openstack.org/#/c/451525/), which once that has landed
and been released, will let us do some neat query of the journal during
development to see slices across services like this
https://dague.net/2017/03/30/in-praise-of-systemd/.

I just want to say thank you! to you clarkb clayg and everyone involved :)
This is so much better!

andreaf

ACTION REQUIRED:

If you maintain a devstack plugin that starts any services, now would be
a great time to test to see if this works for them. The biggest issue is
that the commands sent to run_process need to be absolute pathed.

My hope is that by end of cycle this is going to be the default mode in
devstack for both the gate and development, which eliminates one major
difference between the two. I'm also hoping that we'll be able to keep
and archive the journals from the runs, so you can download and query
those directly. Especially once the oslo.log enhancements are there to
add the additional structured data.

    -Sean

--
Sean Dague
http://dague.net


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 5, 2017 by Andrea_Frittoli (5,920 points)   1 2 3
0 votes

On Wed, Apr 5, 2017 at 1:30 PM, Andrea Frittoli andrea.frittoli@gmail.com
wrote:

I just want to say thank you! to you clarkb clayg and everyone involved :)
This is so much better!

andreaf

Sean is throwing credit at me where none is due. IIRC I was both in the
room and in a very-normal-for-me state of confusion while he and clark
talked about this - but I did not know they were working on it.
Nevertheless, I am dusting off my devstack vm with USE_SYSTEMD=True and
found his blog post interesting ;)

Kudos!

-Clay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 5, 2017 by Clay_Gerrard (5,800 points)   1 2 3
0 votes

On 4/5/2017 3:09 PM, Sean Dague wrote:
At the PTG clayg brought up an excellent question about what the
expected flow was to restart a bunch of services in devstack after a
code changes that impacts many of them (be it common code, or a
library). People had created a bunch of various screen hacks over the
years, but screen is flakey, and this is definitely not an ideal workflow.

Over lunch clayg, clarkb, and I chatted about some options. Clark
brought up the idea of doing systemd units for all of the services. A
couple of weeks ago I decided it was time for me to understand systemd
better anyway, and started playing around with what this might look like.

The results landed here https://review.openstack.org/#/c/448323/.
Documentation is here
http://git.openstack.org/cgit/openstack-dev/devstack/tree/SYSTEMD.rst

This is currently an opt in. All the services in base devstack however
do work in this new model, and I and a few others have been using this
mode the last week or so. It's honestly really great. Working on
oslo.log changes it's now:

pip install -U .
sudo systemctl restart devstack@*

And the change is now in all your services.

There is also an oslo.log change for native systemd journal support
(https://review.openstack.org/#/c/451525/), which once that has landed
and been released, will let us do some neat query of the journal during
development to see slices across services like this
https://dague.net/2017/03/30/in-praise-of-systemd/.

ACTION REQUIRED:

If you maintain a devstack plugin that starts any services, now would be
a great time to test to see if this works for them. The biggest issue is
that the commands sent to run_process need to be absolute pathed.

My hope is that by end of cycle this is going to be the default mode in
devstack for both the gate and development, which eliminates one major
difference between the two. I'm also hoping that we'll be able to keep
and archive the journals from the runs, so you can download and query
those directly. Especially once the oslo.log enhancements are there to
add the additional structured data.

-Sean

Cool. I've always wanted this. When I started on OpenStack I was used to
using sysv-init files and service commands with RHEL and wasn't used to
screen, so pretty much hated that in devstack but felt like I couldn't
ever express that because all of the cool kids loved screen so much.

Well well well, how the tables have turned.

--

Thanks,

Matt


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Apr 6, 2017 by mriedemos_at_gmail.c (15,720 points)   2 5 11
...