settingsLogin | Registersettings

[Openstack] [Fuel] node name issue

0 votes

hi all,

I am using Fuel 10.

i have 2 nodes I am trying to deploy as compute nodes. at one time in the
past I was attempting to deploy them too. I assume back then their node
names were node-11 and node-20.

they were never successfully deploy and now I've worked out their hardware
issues and are attempting to deploy them again. now Fuel has given them the
names node-80 and node-81.
(i may be at 80 in my node names but I only have 17 nodes so far)

the deploy of these 2 nodes does not get past installing Ubuntu. The nodes
reboot after Ubuntu is installed and come up incorrectly as node-11 and
node-20. After that Fuel sits for a long while and then gives an error
(pasted at the end of email). I assume the nodes come up with the wrong
name/ip/ssh-key and Fuel can't contact them.

I'm a novice at using the fuel and fuel2 cli's but I've tried deleting
these nodes and removing from database. Then re-PXE boot the nodes and
start a fresh deploy just to have them named node11 and 20 again. Fuel cli
does show the correct host name for these nodes, but I've tried anyway to
(re)set the host name for these node with no affect.

If I try to delete node-11 and node-20 I get this error
404 Client Error: Not Found for url:
http://10.20.243.1:8000/api/v1/nodes/?ids=11 (NodeCollection not found)

what can I do to get past this please?

Errors from the Fuel Astute log:
2017-09-25 21:06:28 ERROR [1565] Error running provisioning:

,

trace: ["/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:178:in
rescue in initialize_mclient'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:161:ininitializemclient'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:51:in
initialize'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:innew'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:421:in
run_shell_without_check'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:449:inupdatenodestatus'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgunhooks.rb:313:in
reboot_hook'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:38:inblock in process'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:26:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:inprocess'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/imageprovision.rb:117:in
reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:273:insoft
reboot'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:240:in
provision_piece'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:126:inblock
(3 levels) in provisionandwatchprogress'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:in call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:insleep
notgreaterthan'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:120:in block (2 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:inloop'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in block in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:incatch'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in
provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:52:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:109:in
provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:46:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:37:in
image_provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:172:indispatchmessage'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:131:in
block in dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:incall'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task
queue.rb:64:in
block in each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:56:ineach'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/taskqueue.rb:56:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:ineach
withindex'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in
dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:inblock in perform
main_job'"]
2017-09-25 21:06:26 ERROR [1565] Error occured while provisioning:

<Astute::MClientError: #<RuntimeError: Could not find any hosts in

discovery data provided> >
2017-09-25 21:06:26 ERROR [1565] No more retries for MCollective client
instantiation after exception:
["/usr/share/gems/gems/mcollective-client-2.8.4/lib/mcollective/rpc/client.rb:507:in
discover'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:167:ininitializemclient'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:51:in
initialize'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:innew'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:421:in
run_shell_without_check'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:449:inupdatenodestatus'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgunhooks.rb:313:in
reboot_hook'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:38:inblock in process'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:26:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:inprocess'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/imageprovision.rb:117:in
reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:273:insoft
reboot'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:240:in
provision_piece'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:126:inblock
(3 levels) in provisionandwatchprogress'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:in call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:insleep
notgreaterthan'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:120:in block (2 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:inloop'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in block in provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:incatch'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in
provision_and_watch_progress'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:52:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:109:in
provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:46:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:37:in
image_provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:172:indispatchmessage'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:131:in
block in dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:incall'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task
queue.rb:64:in
block in each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:56:ineach'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/taskqueue.rb:56:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:ineach
withindex'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in
dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:inblock in perform
main_job'"]
2017-09-25 21:06:21 ERROR [1565] Retrying RPC client instantiation after
exception: #
2017-09-25 21:06:16 ERROR [1565] Retrying RPC client instantiation after
exception: #
Version: 10.0

-- Jim


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Oct 19, 2017 in openstack by Jim_Okken (480 points)   1 3

3 Responses

0 votes

also I should add, I dont have the original hard drives in the system so it
isn't because it is booting the old OS where these node names were set.
this is definitely the newly installed OS being given the wroing hostname

is there a database this is all kept in? maybe I could look around and find
where these old node names are being saved?

thanks!

-- Jim

On Mon, Sep 25, 2017 at 6:03 PM, Jim Okken jim@jokken.com wrote:

hi all,

I am using Fuel 10.

i have 2 nodes I am trying to deploy as compute nodes. at one time in the
past I was attempting to deploy them too. I assume back then their node
names were node-11 and node-20.

they were never successfully deploy and now I've worked out their hardware
issues and are attempting to deploy them again. now Fuel has given them the
names node-80 and node-81.
(i may be at 80 in my node names but I only have 17 nodes so far)

the deploy of these 2 nodes does not get past installing Ubuntu. The nodes
reboot after Ubuntu is installed and come up incorrectly as node-11 and
node-20. After that Fuel sits for a long while and then gives an error
(pasted at the end of email). I assume the nodes come up with the wrong
name/ip/ssh-key and Fuel can't contact them.

I'm a novice at using the fuel and fuel2 cli's but I've tried deleting
these nodes and removing from database. Then re-PXE boot the nodes and
start a fresh deploy just to have them named node11 and 20 again. Fuel cli
does show the correct host name for these nodes, but I've tried anyway to
(re)set the host name for these node with no affect.

If I try to delete node-11 and node-20 I get this error
404 Client Error: Not Found for url: http://10.20.243.1:8000/api/
v1/nodes/?ids=11 (NodeCollection not found)

what can I do to get past this please?

Errors from the Fuel Astute log:
2017-09-25 21:06:28 ERROR [1565] Error running provisioning:

,

trace: ["/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:178:in
rescue in initialize_mclient'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/mclient.rb:161:ininitializemclient'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:51:in
initialize'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:innew'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:421:in
run_shell_without_check'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/nailgun_hooks.rb:449:inupdatenodestatus'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgunhooks.rb:313:in
reboot_hook'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:38:inblock in process'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/nailgun
hooks.rb:26:in each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:inprocess'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/imageprovision.rb:117:in
reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:273:insoft
reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:240:in
provision_piece'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/provision.rb:126:inblock (3 levels) in
provisionandwatchprogress'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/provision.rb:309:in call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:insleep
notgreaterthan'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/provision.rb:120:in block (2 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/provision.rb:119:inloop'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in block in provision_and_watch_progress'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/provision.rb:118:incatch'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in
provision_and_watch_progress'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/provision.rb:52:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:109:in
provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:46:inprovision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:37:in
image_provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:172:indispatchmessage'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/server/server.rb:131:in block in dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:incall'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task
queue.rb:64:in
block in each'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/server/task_queue.rb:56:ineach'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/taskqueue.rb:56:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:ineach
withindex'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in
dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:inblock in perform
main_job'"]
2017-09-25 21:06:26 ERROR [1565] Error occured while provisioning:

<Astute::MClientError: #<RuntimeError: Could not find any hosts in

discovery data provided> >
2017-09-25 21:06:26 ERROR [1565] No more retries for MCollective client
instantiation after exception: ["/usr/share/gems/gems/
mcollective-client-2.8.4/lib/mcollective/rpc/client.rb:507:in
discover'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:167:ininitializemclient'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/mclient.rb:51:in initialize'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:innew'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:421:in
run_shell_without_check'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/nailgun_hooks.rb:449:inupdatenodestatus'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgunhooks.rb:313:in
reboot_hook'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:38:inblock in process'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/nailgun
hooks.rb:26:in each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:inprocess'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/imageprovision.rb:117:in
reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:273:insoft
reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:240:in
provision_piece'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/provision.rb:126:inblock (3 levels) in
provisionandwatchprogress'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/provision.rb:309:in call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:insleep
notgreaterthan'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/provision.rb:120:in block (2 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/provision.rb:119:inloop'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in block in provision_and_watch_progress'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/provision.rb:118:incatch'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in
provision_and_watch_progress'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/provision.rb:52:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:109:in
provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:46:inprovision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:37:in
image_provision'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:172:indispatchmessage'", "/usr/share/gems/gems/astute-
10.0.0/lib/astute/server/server.rb:131:in block in dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:incall'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task
queue.rb:64:in
block in each'", "/usr/share/gems/gems/astute- 10.0.0/lib/astute/server/task_queue.rb:56:ineach'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/taskqueue.rb:56:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:ineach
withindex'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in
dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:inblock in perform
main_job'"]
2017-09-25 21:06:21 ERROR [1565] Retrying RPC client instantiation after
exception: #
2017-09-25 21:06:16 ERROR [1565] Retrying RPC client instantiation after
exception: #
Version: 10.0

-- Jim


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 26, 2017 by Jim_Okken (480 points)   1 3
0 votes

I ran "fuel2 node update -H blade13 20" just to get out of the node-*
naming convention, as someone suggested

The deploy still names the node node-11 and provisioning fails.

digging a little more, i see it might have to do with the fuel-agent
cloud-init scripts.

in the cloud-init.log on the new node I see the node name being set to
node-11!

this isnt node-11. this was node-20, but I renamed it to blade13 with the
command "fuel2 node update -H blade13 20"

i also noted that after the cloud-init scripts ran at the end the first
boot of the new provisioned OS, that in the Fuel GUI, the FQDN field became
node-11.ourdomain.com (before it was bootstrap.ourdomain.com)

(in the same window Hostname still show as blade13)

But FQDN in the Fuel2 CLI output still shows node-20.ourdomain.com!!!

[fuel2 node show 20

| id |
20
|

| name | Untitled (68:58)
|

| status |
ready
|

| os_platform |
ubuntu
|

| roles |
[u'compute']
|

| kernel_params |
None
|

| pending_roles |
[]
|

| hostname |
node-20
|

| fqdn | node-20.dialogic.com
|

| platform_name | ProLiant BL460c
Gen9
|

where can i find the cloud init settings which are deploy to new nodes?

i guess this has something to do with this file:
/usr/share/fuel-agent/cloud-init-templates/cloudconfigubuntu.jinja2

in that file I see

hostname: {{ common.hostname }}

fqdn: {{ common.fqdn }}

please help me with an info you might have or let me know that populates
those 2 parts of the template?

Is there a database these values are all stored in on the fuel server?

Thanks

--Jim

-- Jim

On Tue, Sep 26, 2017 at 12:00 PM, Jim Okken jim@jokken.com wrote:

also I should add, I dont have the original hard drives in the system so
it isn't because it is booting the old OS where these node names were set.
this is definitely the newly installed OS being given the wroing hostname

is there a database this is all kept in? maybe I could look around and
find where these old node names are being saved?

thanks!

-- Jim

On Mon, Sep 25, 2017 at 6:03 PM, Jim Okken jim@jokken.com wrote:

hi all,

I am using Fuel 10.

i have 2 nodes I am trying to deploy as compute nodes. at one time in the
past I was attempting to deploy them too. I assume back then their node
names were node-11 and node-20.

they were never successfully deploy and now I've worked out their
hardware issues and are attempting to deploy them again. now Fuel has given
them the names node-80 and node-81.
(i may be at 80 in my node names but I only have 17 nodes so far)

the deploy of these 2 nodes does not get past installing Ubuntu. The
nodes reboot after Ubuntu is installed and come up incorrectly as node-11
and node-20. After that Fuel sits for a long while and then gives an error
(pasted at the end of email). I assume the nodes come up with the wrong
name/ip/ssh-key and Fuel can't contact them.

I'm a novice at using the fuel and fuel2 cli's but I've tried deleting
these nodes and removing from database. Then re-PXE boot the nodes and
start a fresh deploy just to have them named node11 and 20 again. Fuel cli
does show the correct host name for these nodes, but I've tried anyway to
(re)set the host name for these node with no affect.

If I try to delete node-11 and node-20 I get this error
404 Client Error: Not Found for url: http://10.20.243.1:8000/api/v1
/nodes/?ids=11 (NodeCollection not found)

what can I do to get past this please?

Errors from the Fuel Astute log:
2017-09-25 21:06:28 ERROR [1565] Error running provisioning:

,

trace: ["/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:178:in
rescue in initialize_mclient'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/mclient.rb:161:ininitializemclient'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:51:in
initialize'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/nailgun_hooks.rb:421:innew'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:421:in
run_shell_without_check'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/nailgun_hooks.rb:449:inupdatenodestatus'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgunhooks.rb:313:in
reboot_hook'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/nailgun_hooks.rb:38:inblock in process'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:26:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:inprocess'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/imageprovision.rb:117:in
reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:273:insoft
reboot'", "/usr/share/gems/gems/astute-1
0.0.0/lib/astute/provision.rb:240:in provision_piece'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:126:inblock (3 levels) in provisionandwatchprogress'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:in
call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:insleep
notgreaterthan'", "/usr/share/gems/gems/astute-1
0.0.0/lib/astute/provision.rb:120:in block (2 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/provision.rb:119:inloop'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in
block in provision_and_watch_progress'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/provision.rb:118:incatch'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in
provision_and_watch_progress'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/provision.rb:52:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:109:in
provision'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/server/dispatcher.rb:46:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:37:in
image_provision'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/server/server.rb:172:indispatchmessage'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:131:in
block in dispatch'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/server/task_queue.rb:64:incall'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task
queue.rb:64:in
block in each'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/server/task_queue.rb:56:ineach'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/taskqueue.rb:56:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:ineach
withindex'", "/usr/share/gems/gems/astute-1
0.0.0/lib/astute/server/server.rb:128:in dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:inblock in perform
main_job'"]
2017-09-25 21:06:26 ERROR [1565] Error occured while provisioning:

<Astute::MClientError: #<RuntimeError: Could not find any hosts in

discovery data provided> >
2017-09-25 21:06:26 ERROR [1565] No more retries for MCollective client
instantiation after exception: ["/usr/share/gems/gems/mcollec
tive-client-2.8.4/lib/mcollective/rpc/client.rb:507:in discover'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/mclient.rb:167:ininitializemclient'", "/usr/share/gems/gems/astute-1
0.0.0/lib/astute/mclient.rb:51:in initialize'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:421:innew'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:421:in
run_shell_without_check'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/nailgun_hooks.rb:449:inupdatenodestatus'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgunhooks.rb:313:in
reboot_hook'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/nailgun_hooks.rb:38:inblock in process'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun
hooks.rb:26:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/nailgun_hooks.rb:26:inprocess'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/imageprovision.rb:117:in
reboot'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:273:insoft
reboot'", "/usr/share/gems/gems/astute-1
0.0.0/lib/astute/provision.rb:240:in provision_piece'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:126:inblock (3 levels) in provisionandwatchprogress'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:in
call'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:309:insleep
notgreaterthan'", "/usr/share/gems/gems/astute-1
0.0.0/lib/astute/provision.rb:120:in block (2 levels) in provision_and_watch_progress'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/provision.rb:119:inloop'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:119:in
block in provision_and_watch_progress'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/provision.rb:118:incatch'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:118:in
provision_and_watch_progress'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/provision.rb:52:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:109:in
provision'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/server/dispatcher.rb:46:inprovision'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:37:in
image_provision'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/server/server.rb:172:indispatchmessage'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:131:in
block in dispatch'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/server/task_queue.rb:64:incall'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task
queue.rb:64:in
block in each'", "/usr/share/gems/gems/astute-1 0.0.0/lib/astute/server/task_queue.rb:56:ineach'",
"/usr/share/gems/gems/astute-10.0.0/lib/astute/server/taskqueue.rb:56:in
each'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:ineach
withindex'", "/usr/share/gems/gems/astute-1
0.0.0/lib/astute/server/server.rb:128:in dispatch'", "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:inblock in perform
main_job'"]
2017-09-25 21:06:21 ERROR [1565] Retrying RPC client instantiation after
exception: #
2017-09-25 21:06:16 ERROR [1565] Retrying RPC client instantiation after
exception: #
Version: 10.0

-- Jim


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Sep 28, 2017 by Jim_Okken (480 points)   1 3
0 votes

hi all,

please help us out with an issue we are seeing on multiple compute nodes
running Newton (Ubuntu 16.04.3 Kernel 4.4.0). After about 1 hour of running
our VOIP test application the instances become non-responsive and can't be
pinged as well do the compute nodes.

messages appear on the compute node console screens. a screen shot of that
is hosted here:

http://www.jokken.com/downloads/console.png

i'll try to attach it also.

The first compute node this was seen on was running 2 instances, the second
was running only 1 instance. They were using on a portion of the total 40
vCPUs available, and the load was moderate. Cold boot these nodes and all
is well again, until we run our application for about 1 hour.

please let us know what you think thanks!

not a lot is shown in DEBUG logging of Nova and Neutron on the compute node

these logs are here:

http://www.jokken.com/downloads/logs.zip

i'll try to attach them too.

https://ask.openstack.org/en/question/110748/soft-lockup-on-newton-compute-nodes/

/var/log/messages on the compute node shows many repeats of these messages:

2017-10-18T20:49:26.462309+00:00 node-58 kernel: [1297007.624935] Modules
linked in: binfmtmisc nfconntracknetlink vhostnet vhost macvtap macvlan
ip6tableraw xtmac xttcpudp xtphysdev brnetfilter xtset
ipsethashnet ipset nfnetlink veth ebtablefilter ebtables openvswitch
ocfs2 quota
tree ocfs2dlmfs ocfs2stacko2cb ocfs2dlm ocfs2nodemanager
ocfs2
stackglue configfs ip6tablefilter ip6tables xtmultiport
xt
conntrack iptablefilter xtcomment xtCT iptableraw iptables xtables
xfs ipmissif 8021q garp mrp intelrapl x86pkgtempthermal
intel
powerclamp coretemp crct10difpclmul crc32pclmul ghashclmulniintel
aesniintel aesx8664 lrw gf128mul gluehelper ablkhelper cryptd
serio
raw bridge stp llc sbedac edaccore hpilo ioatdma lpcich shpchp dca
ipmi
si 8250fintek ipmimsghandler acpipowermeter machid kvmintel kvm
irqbypass ibiser rdmacm iwcm ibcm ibsa ibmad ibcore ibaddr
iscsitcp libiscsitcp nfconntrackprotogre nfconntrackipv6
nf
defragipv6 nfconntrackipv4 nfdefragipv4 nfconntrack autofs4 raid10
raid456 asyncraid6recov asyncmemcpy asyncpq asyncxor asynctx xor
raid6pq libcrc32c raid1 raid0 multipath linear dmroundrobin ses
enclosure uas usb
storage psmouse ahci lpfc be2iscsi libahci be2net
iscsibootsysfs libiscsi vxlan scsitransportfc ip6udptunnel
scsitransportiscsi udptunnel wmi fjes scsidhemc scsidhrdac
scsi
dhalua dmmultipath

2017-10-18T20:49:26.462311+00:00 node-58 kernel: [1297007.625008] CPU: 27
PID: 860 Comm: qemu-system-x86 Not tainted 4.4.0-93-generic #116-Ubuntu

2017-10-18T20:49:26.462313+00:00 node-58 kernel: [1297007.625009] Hardware
name: HP ProLiant BL460c Gen9, BIOS I36 02/17/2017

2017-10-18T20:49:26.462314+00:00 node-58 kernel: [1297007.625010] task:
ffff881faaaa7000 ti: ffff881fa3a34000 task.ti: ffff881fa3a34000

2017-10-18T20:49:26.462315+00:00 node-58 kernel: [1297007.625011] RIP:
0010:[] []
nativequeuedspinlockslowpath+0x15c/0x170

2017-10-18T20:49:26.462316+00:00 node-58 kernel: [1297007.625018] RSP:
0018:ffff883fff143c30 EFLAGS: 00000202

2017-10-18T20:49:26.462317+00:00 node-58 kernel: [1297007.625019] RAX:
0000000000000101 RBX: ffff881f677603f0 RCX: 0000000000000001

2017-10-18T20:49:26.462337+00:00 node-58 kernel: [1297007.625020] RDX:
0000000000000101 RSI: 0000000000000001 RDI: ffff881f677603ec

2017-10-18T20:49:26.462340+00:00 node-58 kernel: [1297007.625020] RBP:
ffff883fff143c30 R08: 0000000000000101 R09: ffffffff81191e27

2017-10-18T20:49:26.462341+00:00 node-58 kernel: [1297007.625021] R10:
ffffea00ffb09780 R11: 0000000000000a00 R12: ffff881f677603ec

2017-10-18T20:49:26.462342+00:00 node-58 kernel: [1297007.625022] R13:
0000000000000a00 R14: 00000000000a5000 R15: 0000000000000a00

2017-10-18T20:49:26.462343+00:00 node-58 kernel: [1297007.625023] FS:
00007f0c53fb3c00(0000) GS:ffff883fff140000(0000) knlGS:0000000000000000

2017-10-18T20:49:26.462343+00:00 node-58 kernel: [1297007.625024] CS: 0010
DS: 0000 ES: 0000 CR0: 0000000080050033

2017-10-18T20:49:26.462344+00:00 node-58 kernel: [1297007.625025] CR2:
00007fe018e2547e CR3: 0000003ec0b75000 CR4: 00000000001426e0

2017-10-18T20:49:26.462345+00:00 node-58 kernel: [1297007.625026] Stack:

2017-10-18T20:49:26.462347+00:00 node-58 kernel: [1297007.625026]
ffff883fff143c40 ffffffff81842f71 ffff883fff143c60 ffffffff81841085

2017-10-18T20:49:26.462348+00:00 node-58 kernel: [1297007.625028]
ffff881dc609ac00 ffff881f677604b0 ffff883fff143c70 ffffffff818410cb

2017-10-18T20:49:26.462349+00:00 node-58 kernel: [1297007.625029]
ffff883fff143ca0 ffffffffc08c658d ffff883feff9d500 0000000000000a00

2017-10-18T20:49:26.462351+00:00 node-58 kernel: [1297007.625031] Call
Trace:

2017-10-18T20:49:26.462353+00:00 node-58 kernel: [1297007.625032]

2017-10-18T20:49:26.462354+00:00 node-58 kernel: [1297007.625039]
[] rawspin_lock+0x21/0x30

2017-10-18T20:49:26.462356+00:00 node-58 kernel: [1297007.625041]
[] __mutexunlockslowpath+0x25/0x50

2017-10-18T20:49:26.462356+00:00 node-58 kernel: [1297007.625042]
[] mutex_unlock+0x1b/0x20

2017-10-18T20:49:26.462357+00:00 node-58 kernel: [1297007.625076]
[] ocfs2dioend_io+0x6d/0x80 [ocfs2]

2017-10-18T20:49:26.462358+00:00 node-58 kernel: [1297007.625080]
[] dio_complete+0x11c/0x1c0

2017-10-18T20:49:26.462359+00:00 node-58 kernel: [1297007.625081]
[] diobioend_aio+0x73/0x100

2017-10-18T20:49:26.462361+00:00 node-58 kernel: [1297007.625085]
[] bio_endio+0x3f/0x60

2017-10-18T20:49:26.462362+00:00 node-58 kernel: [1297007.625087]
[] blkupdaterequest+0x87/0x310

2017-10-18T20:49:26.462363+00:00 node-58 kernel: [1297007.625091]
[] endclonebio+0x46/0x70

2017-10-18T20:49:26.462363+00:00 node-58 kernel: [1297007.625092]
[] bio_endio+0x3f/0x60

2017-10-18T20:49:26.462364+00:00 node-58 kernel: [1297007.625093]
[] blkupdaterequest+0x87/0x310

2017-10-18T20:49:26.462365+00:00 node-58 kernel: [1297007.625097]
[] scsiendrequest+0x33/0x1d0

2017-10-18T20:49:26.462367+00:00 node-58 kernel: [1297007.625100]
[] scsiiocompletion+0x1b6/0x690

2017-10-18T20:49:26.462368+00:00 node-58 kernel: [1297007.625104]
[] ? rebalance_domains+0x166/0x2d0

2017-10-18T20:49:26.462368+00:00 node-58 kernel: [1297007.625107]
[] scsifinishcommand+0xcf/0x120

2017-10-18T20:49:26.462377+00:00 node-58 kernel: [1297007.625109]
[] scsisoftirqdone+0x124/0x150

2017-10-18T20:49:26.462378+00:00 node-58 kernel: [1297007.625112]
[] blkdonesoftirq+0x87/0xb0

2017-10-18T20:49:26.462379+00:00 node-58 kernel: [1297007.625116]
[] __do_softirq+0x101/0x290

2017-10-18T20:49:26.462381+00:00 node-58 kernel: [1297007.625118]
[] irq_exit+0xa3/0xb0

2017-10-18T20:49:26.462382+00:00 node-58 kernel: [1297007.625121]
[] smpcallfunctionsingleinterrupt+0x33/0x40

2017-10-18T20:49:26.462382+00:00 node-58 kernel: [1297007.625124]
[] callfunctionsingle_interrupt+0x82/0x90

2017-10-18T20:49:26.462383+00:00 node-58 kernel: [1297007.625125]

2017-10-18T20:49:26.462383+00:00 node-58 kernel: [1297007.625127]
[] ? rawspin_lock+0x14/0x30

2017-10-18T20:49:26.462385+00:00 node-58 kernel: [1297007.625129]
[] __mutexlockslowpath+0x72/0x130

2017-10-18T20:49:26.462387+00:00 node-58 kernel: [1297007.625142]
[] ? ocfs2inodeunlock+0x119/0x120 [ocfs2]

2017-10-18T20:49:26.462387+00:00 node-58 kernel: [1297007.625143]
[] mutex_lock+0x1f/0x30

2017-10-18T20:49:26.462388+00:00 node-58 kernel: [1297007.625155]
[] ocfs2filewrite_iter+0x95a/0xdf0 [ocfs2]

2017-10-18T20:49:26.462388+00:00 node-58 kernel: [1297007.625158]
[] ? pollselectcopy_remaining+0x140/0x140

2017-10-18T20:49:26.462389+00:00 node-58 kernel: [1297007.625169]
[] ? ocfs2checkrangeforrefcount+0x150/0x150 [ocfs2]

2017-10-18T20:49:26.462391+00:00 node-58 kernel: [1297007.625171]
[] aioruniocb+0x26a/0x2d0

2017-10-18T20:49:26.462392+00:00 node-58 kernel: [1297007.625174]
[] ? __fget_light+0x25/0x60

2017-10-18T20:49:26.462394+00:00 node-58 kernel: [1297007.625175]
[] ? __fdget+0x13/0x20

2017-10-18T20:49:26.462395+00:00 node-58 kernel: [1297007.625177]
[] doiosubmit+0x25f/0x500

2017-10-18T20:49:26.462396+00:00 node-58 kernel: [1297007.625178]
[] SySiosubmit+0x10/0x20

2017-10-18T20:49:26.462398+00:00 node-58 kernel: [1297007.625181]
[] entrySYSCALL64_fastpath+0x16/0x71

2017-10-18T20:49:26.462399+00:00 node-58 kernel: [1297007.625181] Code: 01
48 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6 c7 40 08 01 00 00 00
e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 <8b> 07 84 c0 75 f8 b8
01 00 00 00 66 89 07 5d c3 0f 1f 40 00 0f


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded Oct 19, 2017 by Jim_Okken (480 points)   1 3
...