settingsLogin | Registersettings

[Openstack] Fuel-9.2 - Broken deployment on one node stops all other nodes too

0 votes

Hello All,

Trying to add one more node to the group, and it breaks with error:
Failed tasks: Task[fuel_pkgs/9] Stopping the deployment process!

But now the entire cluster is stopped - nova service on all nodes are down.
Is this expected behaviour? How do I get back my cluster active?

Please help me to resolve this problem.

Regards,
Raja.

--
:^)


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Nov 2, 2017 in openstack by Raja_T_Nair (1,020 points)   3 5

4 Responses

0 votes

Hi,

fuelpkgs/9 means that "fuelpkgs" task failed on node with id 9, find the
node with this id using "fuel node" (run it from Fuel node), ssh to the
nodes and look for errors in /var/log/puppet.log.

If Nova service was running before, it should be running even after
fuel_pkgs task failed on one of the nodes, see nova logs, to find what
happened to nova.

Thanks,

On Mon, Oct 30, 2017 at 12:23 AM, Raja T Nair rtnair@gmail.com wrote:

Hello All,

Trying to add one more node to the group, and it breaks with error:
Failed tasks: Task[fuel_pkgs/9] Stopping the deployment process!

But now the entire cluster is stopped - nova service on all nodes are down.
Is this expected behaviour? How do I get back my cluster active?

Please help me to resolve this problem.

Regards,
Raja.

--
:^)


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Oct 30, 2017 by Evgeniy_L (8,580 points)   1 2 4
0 votes

Hi,

I built the repos locally, using the instructions in this page:
https://docs.openstack.org/fuel-docs/latest/userdocs/fuel-install-guide/local-repo.html

Now, while trying to deploy to 2 newly added nodes, it fails with this
error:

2017-10-31
06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::Fuelpkgs/Package[fuel-ha-utils]/ensure)
E: Unable to locate package fuel-ha-utils
2017-10-31
06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::Fuelpkgs/Package[fuel-ha-utils]/ensure)
Reading state information...
2017-10-31
06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::Fuelpkgs/Package[fuel-ha-utils]/ensure)
Building dependency tree...
2017-10-31
06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::Fuelpkgs/Package[fuel-ha-utils]/ensure)
change from purged to present failed: Execution of '/usr/bin/apt-get -q -y
-o DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1
install fuel-ha-utils' returned 100: Reading package lists...
2017-10-31 06:45:11ERR/usr/bin/puppet:8:in <main>'2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:inexecute'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util/commandline.rb:146:in
run'2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:inrun'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util.rb:496:in
exit_on_fail' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:inblock in
run'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/application.rb:507:in
plugin_hook' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:inblock (2
levels) in run'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendorruby/puppet/application/apply.rb:159:in
run_command' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:inmain'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet.rb:246:in override' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:inoverride'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendorruby/puppet/application/apply.rb:236:in
block in main' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:inapply
catalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/configurer.rb:133:in
run' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet.rb:246:inoverride'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/context.rb:64:in
override' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:inblock in run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/configurer.rb:227:in
run_internal' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:inapply
catalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util.rb:160:in
benchmark' 2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:inrealtime'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/util.rb:161:in
block in benchmark' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:inblock in applycatalog'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/resource/catalog.rb:168:in
apply' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:inasloggingdestination'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util/log.rb:149:in
with_destination' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:inblock in apply'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/transaction.rb:138:in evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:intraverse'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendorruby/puppet/transaction.rb:147:in block in evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:inthinmark'
2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:in realtime' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:inblock in thinmark'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/transaction.rb:147:in block (2 levels) in evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:incall'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendorruby/puppet/transaction.rb:217:in
eval_resource' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:inapply'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/transaction/resourceharness.rb:20:in
evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:inperform
changes'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendorruby/puppet/transaction/resourceharness.rb:128:in
sync_if_needed' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:insync'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/property.rb:581:in
sync' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:inset'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/property.rb:197:in
call_valuemethod' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/type/package.rb:73:inblock (3
levels) in '
2017-10-31
06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/aptfuel.rb:72:in
install' 2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:72:ineach'
2017-10-31
06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt
fuel.rb:74:in
block in install' 2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/apt_fuel.rb:66:inwaitforlock'
2017-10-31
06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/provider/package/aptfuel.rb:75:in
block (2 levels) in install' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider/package/apt.rb:73:ininstall'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/provider.rb:463:in
block in create_class_and_instance_method' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:inblock in hascommand'
2017-10-31
06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/provider/command.rb:23:in
execute' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:219:inexecute'
2017-10-31 06:45:11ERRE: Unable to locate package fuel-ha-utils
2017-10-31 06:45:11ERRReading state information...
2017-10-31 06:45:11ERRBuilding dependency tree...

I can see fuel-ha-utils deb package in /var/www/ tree:

[root@fuel ~]# cd /var/www/nailgun/
[root@fuel nailgun]# find . -name fuel-ha-utils*
./mitaka-9.0/mos-centos/x8664/Packages/fuel-ha-utils-9.0.0-1.mos8460.noarch.rpm
./mitaka-9.0/ubuntu/x86
64/pool/main/f/fuel-library9.0/fuel-ha-utils9.0.0-1~u14.04+mos8460all.deb
[root@fuel nailgun]#

On a node which was installed before building local repo, I can see the
package fuel-ha-utils installed, but a different version i think:
root@node-4:~# dpkg -l fuel-ha-utils
Desired=Unknown/Install/Remove/Purge/Hold
|
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version
Architecture Description
+++-==============================-====================-====================-==================================================================
ii fuel-ha-utils 9.0.0-1~u14.04+mos87
all Fuel Library HA utils
root@node-4:~#

Is there a way to fix this problem? Or should I just remove all nodes and
start the deployment from scratch - since local repo was built after
deploying 5 nodes?

Please advise.

Best Regards,
Raja.

On 30 October 2017 at 19:27, Evgeniy L eli@mirantis.com wrote:

Hi,

fuelpkgs/9 means that "fuelpkgs" task failed on node with id 9, find the
node with this id using "fuel node" (run it from Fuel node), ssh to the
nodes and look for errors in /var/log/puppet.log.

If Nova service was running before, it should be running even after
fuel_pkgs task failed on one of the nodes, see nova logs, to find what
happened to nova.

Thanks,

On Mon, Oct 30, 2017 at 12:23 AM, Raja T Nair rtnair@gmail.com wrote:

Hello All,

Trying to add one more node to the group, and it breaks with error:
Failed tasks: Task[fuel_pkgs/9] Stopping the deployment process!

But now the entire cluster is stopped - nova service on all nodes are
down.
Is this expected behaviour? How do I get back my cluster active?

Please help me to resolve this problem.

Regards,
Raja.

--
:^)


Mailing list: http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack

--
:^)


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Oct 31, 2017 by Raja_T_Nair (1,020 points)   3 5
0 votes

Run "apt-cache policy fuel-ha-utils" and see from which repository does it
come from, if your mirrors configured correctly, the package should come
from your mirror.
It also shows what command it failed to run, "/usr/bin/apt-get -q -y -o
DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1 install
fuel-ha-utils" you can run it manually on the node and see what the error
is.

Thanks,

On Tue, Oct 31, 2017 at 12:27 AM, Raja T Nair rtnair@gmail.com wrote:

Hi,

I built the repos locally, using the instructions in this page:
https://docs.openstack.org/fuel-docs/latest/userdocs/
fuel-install-guide/local-repo.html

Now, while trying to deploy to 2 newly added nodes, it fails with this
error:

2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::
Fuel
pkgs/Package[fuel-ha-utils]/ensure) E: Unable to locate package
fuel-ha-utils
2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::
Fuel
pkgs/Package[fuel-ha-utils]/ensure) Reading state information...
2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::
Fuel
pkgs/Package[fuel-ha-utils]/ensure) Building dependency tree...
2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::
Fuel
pkgs/Package[fuel-ha-utils]/ensure) change from purged to present
failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold
-o APT::Get::AllowUnauthenticated=1 install fuel-ha-utils' returned 100:
Reading package lists...
2017-10-31 06:45:11ERR/usr/bin/puppet:8:in <main>'2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:inexecute'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util/commandline.rb:146:in
run'2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:inrun'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util.rb:496:in
exit_on_fail' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:inblock in run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/application.rb:507:in
plugin_hook' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:inblock (2 levels) in run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/application/apply.rb:159:in
run_command' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:inmain'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet.rb:246:in
override' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:inoverride'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/application/apply.rb:236:in
block in main' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:inapply
catalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/configurer.rb:133:in
run' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet.rb:246:inoverride'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/context.rb:64:in
override' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:inblock in run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/configurer.rb:227:in
run_internal' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:inapply
catalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util.rb:160:in
benchmark' 2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:inrealtime'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/util.rb:161:in
block in benchmark' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:inblock in applycatalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/resource/catalog.rb:168:in
apply' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:inasloggingdestination'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util/log.rb:149:in
with_destination' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:inblock in apply'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/transaction.rb:138:in
evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:intraverse'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/transaction.rb:147:in
block in evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:inthinmark'
2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:in realtime' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:inblock in thinmark'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/transaction.rb:147:in
block (2 levels) in evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:incall'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/transaction.rb:217:in
eval_resource' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:inapply'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/
transaction/resourceharness.rb:20:in evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/ transaction/resource_harness.rb:81:inperformchanges'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/
transaction/resource
harness.rb:128:in sync_if_needed' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/ transaction/resource_harness.rb:204:insync'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/property.rb:581:in
sync' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:inset'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/property.rb:197:in
call_valuemethod' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/type/package.rb:73:inblock (3 levels) in '
2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/
puppet/provider/package/aptfuel.rb:72:in install' 2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/ puppet/provider/package/apt_fuel.rb:72:ineach'
2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/
puppet/provider/package/apt
fuel.rb:74:in block in install' 2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/ puppet/provider/package/apt_fuel.rb:66:inwaitforlock'
2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/
puppet/provider/package/aptfuel.rb:75:in block (2 levels) in install' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider/package/apt.rb:73:ininstall'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/provider.rb:463:in
block in create_class_and_instance_method' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:inblock in hascommand'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/provider/command.rb:23:in
execute' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:219:inexecute'
2017-10-31 06:45:11ERRE: Unable to locate package fuel-ha-utils
2017-10-31 06:45:11ERRReading state information...
2017-10-31 06:45:11ERRBuilding dependency tree...

I can see fuel-ha-utils deb package in /var/www/ tree:

[root@fuel ~]# cd /var/www/nailgun/
[root@fuel nailgun]# find . -name fuel-ha-utils*
./mitaka-9.0/mos-centos/x8664/Packages/fuel-ha-utils-9.0.
0-1.mos8460.noarch.rpm
./mitaka-9.0/ubuntu/x86
64/pool/main/f/fuel-library9.0/
fuel-ha-utils9.0.0-1~u14.04+mos8460all.deb
[root@fuel nailgun]#

On a node which was installed before building local repo, I can see the
package fuel-ha-utils installed, but a different version i think:
root@node-4:~# dpkg -l fuel-ha-utils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/
trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version
Architecture Description
+++-==============================-====================-====
================-===========================================
=======================
ii fuel-ha-utils 9.0.0-1~u14.04+mos87
all Fuel Library HA utils
root@node-4:~#

Is there a way to fix this problem? Or should I just remove all nodes and
start the deployment from scratch - since local repo was built after
deploying 5 nodes?

Please advise.

Best Regards,
Raja.

On 30 October 2017 at 19:27, Evgeniy L eli@mirantis.com wrote:

Hi,

fuelpkgs/9 means that "fuelpkgs" task failed on node with id 9, find
the node with this id using "fuel node" (run it from Fuel node), ssh to the
nodes and look for errors in /var/log/puppet.log.

If Nova service was running before, it should be running even after
fuel_pkgs task failed on one of the nodes, see nova logs, to find what
happened to nova.

Thanks,

On Mon, Oct 30, 2017 at 12:23 AM, Raja T Nair rtnair@gmail.com wrote:

Hello All,

Trying to add one more node to the group, and it breaks with error:
Failed tasks: Task[fuel_pkgs/9] Stopping the deployment process!

But now the entire cluster is stopped - nova service on all nodes are
down.
Is this expected behaviour? How do I get back my cluster active?

Please help me to resolve this problem.

Regards,
Raja.

--
:^)


Mailing list: http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack

--
:^)


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Oct 31, 2017 by Evgeniy_L (8,580 points)   1 2 4
0 votes

Hi Evgeniy,

apt-cache policy fuel-ha-utils says that the package is not to be found.

Now I re-installed fuel on another node(as there was a lot of issues faced
on the old one), and created local repo using these commands:

fuel-mirror create -P ubuntu -G ubuntu
fuel-mirror apply -G ubuntu -I /usr/share/fuel-mirror/ubuntu.yaml --replace
--default
fuel-mirror apply -G mos -I /usr/share/fuel-mirror/ubuntu.yaml --default

my /usr/share/fuel-mirror directory looks like this:

[root@fuel92 nginx]# ll /usr/share/fuel-mirror/
total 12
-rwxr-xr-x. 1 root root 1344 May 24 07:47 centos.yaml
-rwxr-xr-x. 1 root root 4288 May 24 07:47 ubuntu.yaml
[root@fuel92 nginx]#

While doing the ``connectivity check'', it failed with following error
lines:
Verification failed.
Repo availability verification failed on following nodes Fuel master node,
Untitled (0b:34), Untitled (71:28), Untitled (2b:52), Untitled (ae:40),
Untitled (a8:73).
Following repos are not available -
http://172.22.66.11:8080/mirrors/mos-repos/ubuntu/9.0,
http://172.22.66.11:8080/mirrors/mos-repos/ubuntu/9.2.

nginx logs say this:
2017/11/02 10:22:47 [error] 1188#1188: *12256 open()
"/var/www/nailgun/mirrors/mos-repos/ubuntu/9.0" failed (2: No such file or
directory), client
: 172.22.66.24, server: localhost, request: "GET
/mirrors/mos-repos/ubuntu/9.0 HTTP/1.1", host: "172.22.66.11:8080"

And the directory mos-repos actually does not exist:
[root@fuel92 nginx]# ls /var/www/nailgun/mirrors/
ubuntu

I suppose I need to build the mos-repos repo also locally. If yes, how do I
do this? I thought it would have built it with the above
"fuel-mirror create -P ubuntu -G ubuntu" command.

Attaching screenshot of settings page, which shows currently configured
repos.

Please help me with this one.

Regards,
Raja.

On 31 October 2017 at 19:29, Evgeniy L eli@mirantis.com wrote:

Run "apt-cache policy fuel-ha-utils" and see from which repository does it
come from, if your mirrors configured correctly, the package should come
from your mirror.
It also shows what command it failed to run, "/usr/bin/apt-get -q -y -o
DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1
install fuel-ha-utils" you can run it manually on the node and see what the
error is.

Thanks,

On Tue, Oct 31, 2017 at 12:27 AM, Raja T Nair rtnair@gmail.com wrote:

Hi,

I built the repos locally, using the instructions in this page:
https://docs.openstack.org/fuel-docs/latest/userdocs/fuel-
install-guide/local-repo.html

Now, while trying to deploy to 2 newly added nodes, it fails with this
error:

2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::Fuel
pkgs/Package[fuel-ha-utils]/ensure) E: Unable to locate package
fuel-ha-utils
2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::Fuel
pkgs/Package[fuel-ha-utils]/ensure) Reading state information...
2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::Fuel
pkgs/Package[fuel-ha-utils]/ensure) Building dependency tree...
2017-10-31 06:45:11ERR(/Stage[main]/Osnailyfacter::Fuelpkgs::Fuel
pkgs/Package[fuel-ha-utils]/ensure) change from purged to present
failed: Execution of '/usr/bin/apt-get -q -y -o
DPkg::Options::=--force-confold -o APT::Get::AllowUnauthenticated=1
install fuel-ha-utils' returned 100: Reading package lists...
2017-10-31 06:45:11ERR/usr/bin/puppet:8:in <main>'2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:inexecute'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util/commandline.rb:146:in
run'2017-10-31 06:45:11ERR/usr/lib/ruby/vendo r_ruby/puppet/application.rb:381:inrun'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util.rb:496:in
exit_on_fail' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:inblock in run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/application.rb:507:in
plugin_hook' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:inblock (2 levels) in run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/application/apply.rb:159:in
run_command' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:inmain'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet.rb:246:in
override' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:inoverride'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/application/apply.rb:236:in
block in main' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:inapply
catalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/configurer.rb:133:in
run' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet.rb:246:inoverride'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/context.rb:64:in
override' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:inblock in run'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/configurer.rb:227:in
run_internal' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:inapply
catalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util.rb:160:in
benchmark' 2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:inrealtime'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/util.rb:161:in
block in benchmark' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:inblock in applycatalog'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/resource/catalog.rb:168:in
apply' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:inasloggingdestination'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/util/log.rb:149:in
with_destination' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:inblock in apply'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/transaction.rb:138:in
evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:intraverse'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/transaction.rb:147:in
block in evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:inthinmark'
2017-10-31 06:45:11ERR/usr/lib/ruby/1.9.1/benchmark.rb:295:in realtime' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:inblock in thinmark'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/transaction.rb:147:in
block (2 levels) in evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:incall'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/transaction.rb:217:in
eval_resource' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:inapply'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/transaction/resourceharness.rb:20:in
evaluate' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:inperform
changes'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/transaction/resourceharness.rb:128:in
sync_if_needed' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:insync'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendorruby/puppet/property.rb:581:in
sync' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/property.rb:498:inset'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/property.rb:197:in
call_valuemethod' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/type/package.rb:73:inblock (3 levels) in '
2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/
provider/package/aptfuel.rb:72:in install' 2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/ provider/package/apt_fuel.rb:72:ineach'
2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/
provider/package/apt
fuel.rb:74:in block in install' 2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/ provider/package/apt_fuel.rb:66:inwaitforlock'
2017-10-31 06:45:11ERR/etc/puppet/modules/osnailyfacter/lib/puppet/
provider/package/aptfuel.rb:75:in block (2 levels) in install' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider/package/apt.rb:73:ininstall'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/provider.rb:463:in
block in create_class_and_instance_method' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/provider.rb:237:inblock in hascommand'
2017-10-31 06:45:11ERR/usr/lib/ruby/vendor
ruby/puppet/provider/command.rb:23:in
execute' 2017-10-31 06:45:11ERR/usr/lib/ruby/vendor_ruby/puppet/util/execution.rb:219:inexecute'
2017-10-31 06:45:11ERRE: Unable to locate package fuel-ha-utils
2017-10-31 06:45:11ERRReading state information...
2017-10-31 06:45:11ERRBuilding dependency tree...

I can see fuel-ha-utils deb package in /var/www/ tree:

[root@fuel ~]# cd /var/www/nailgun/
[root@fuel nailgun]# find . -name fuel-ha-utils*
./mitaka-9.0/mos-centos/x8664/Packages/fuel-ha-utils-9.0.0-
1.mos8460.noarch.rpm
./mitaka-9.0/ubuntu/x86
64/pool/main/f/fuel-library9.0/fuel-
ha-utils9.0.0-1~u14.04+mos8460all.deb
[root@fuel nailgun]#

On a node which was installed before building local repo, I can see the
package fuel-ha-utils installed, but a different version i think:
root@node-4:~# dpkg -l fuel-ha-utils
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig
-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version
Architecture Description
+++-==============================-====================-====
================-===========================================
=======================
ii fuel-ha-utils 9.0.0-1~u14.04+mos87
all Fuel Library HA utils
root@node-4:~#

Is there a way to fix this problem? Or should I just remove all nodes and
start the deployment from scratch - since local repo was built after
deploying 5 nodes?

Please advise.

Best Regards,
Raja.

On 30 October 2017 at 19:27, Evgeniy L eli@mirantis.com wrote:

Hi,

fuelpkgs/9 means that "fuelpkgs" task failed on node with id 9, find
the node with this id using "fuel node" (run it from Fuel node), ssh to the
nodes and look for errors in /var/log/puppet.log.

If Nova service was running before, it should be running even after
fuel_pkgs task failed on one of the nodes, see nova logs, to find what
happened to nova.

Thanks,

On Mon, Oct 30, 2017 at 12:23 AM, Raja T Nair rtnair@gmail.com wrote:

Hello All,

Trying to add one more node to the group, and it breaks with error:
Failed tasks: Task[fuel_pkgs/9] Stopping the deployment process!

But now the entire cluster is stopped - nova service on all nodes are
down.
Is this expected behaviour? How do I get back my cluster active?

Please help me to resolve this problem.

Regards,
Raja.

--
:^)


Mailing list: http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack

--
:^)

--
:^)


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded Nov 2, 2017 by Raja_T_Nair (1,020 points)   3 5
...