settingsLogin | Registersettings

[Openstack] Horizon console WAY slow for an instance

0 votes

Does anyone have any idea why a console session's would be so slow from
horizon? After creating an instance, I go to Compute, Instance, then
Console tab where I click "Click here to show only console". From there,
it's EXTREMELY slow interacting with the instance itself... I can RDP or
ssh to the instance and speed is fine, so it seems like it's got something
to do with the console interaction with the instance. Is this a VNC issue?

Response time is fine for everything else with Horizon and Instances, it's
just interacting with the console that's slow.

Any help would be greatly appreciated!

Thanks


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Jun 29, 2016 in openstack by Sterdnot_Shaken (900 points)   2 4 10

6 Responses

0 votes

My rule-of-thumb for stuff that falls into the "really slow but works"
camp is, check DNS. If possible, disable reverse lookups, or else ensure
that the hosts involved all know how to reverse-resolve each other's IP
addresses.

That's where I'd start, anyway. Good luck!

-Ken

On 2016-06-28 20:28, Sterdnot Shaken wrote:

Does anyone have any idea why a console session's would be so slow from horizon? After creating an instance, I go to Compute, Instance, then Console tab where I click "Click here to show only console". From there, it's EXTREMELY slow interacting with the instance itself... I can RDP or ssh to the instance and speed is fine, so it seems like it's got something to do with the console interaction with the instance. Is this a VNC issue?

Response time is fine for everything else with Horizon and Instances, it's just interacting with the console that's slow.

Any help would be greatly appreciated!

Thanks


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]

Links:


[1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Jun 29, 2016 by Ken_D'Ambrosio (1,180 points)   1 7 13
0 votes

Hi all,

I have a Mitaka environment where glance, nova and cinder use ceph
(rbd) as storage backend, that works perfectly fine. Now I'm trying to
get meters from my rbd pool with ceilometer, but libvirt fails to read
information, the ceilometer-polling.log says

---cut here---
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
[req-823d46da-5a23-45bd-a0e9-596e7d4a9fc3 admin - - - -] Ignoring
instance instance-000002d7 (51d7bfdc-feec-4f13-ad0c-190dcfa2c62d) :
this function is not supported by the connection driver:
virDomainGetBlockInfo
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
Traceback (most recent call last):
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
File
"/usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/disk.py", line
625, in getsamples
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
instance,
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
File
"/usr/lib/python2.7/site-packages/ceilometer/compute/pollsters/disk.py", line
567, in _populate
cache
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
for disk, info in diskinfo:
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
File
"/usr/lib/python2.7/site-packages/ceilometer/compute/virt/libvirt/inspector.py", line 215, in
inspect
diskinfo
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
block
info = domain.blockInfo(device)
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 690, in
blockInfo
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
if ret is None: raise libvirtError ('virDomainGetBlockInfo()
failed', dom=self)
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
libvirtError: this function is not supported by the connection driver:
virDomainGetBlockInfo
2016-06-29 15:45:19.711 29483 ERROR ceilometer.compute.pollsters.disk
---cut here---

This bug [1] describes the issue, but it seems to be a libvirt issue,
not ceilometer.

According to [2] it should be possible to retrieve those meters.
I followed ceph docs to install the rados-gateway, I integrated
keystone authentication, at least I don't get any errors regarding
authentication and swift command seems to work.
Then I added the meters described in [2] to the /etc/ceilometer/pipeline.yaml

---cut here---
control1:~ # cat /etc/ceilometer/pipeline.yaml


sources:
[...]
- name: radosgwsource
interval: 600
meters:
- "radosgw.objects"
- "radosgw.objects.size"
- "radosgw.objects.containers"
- "radosgw.api.request"
- "radosgw.containers.objects"
- "radosgw.containers.objects.size"
sinks:
- meter
sink
[...]
---cut here---

I believe I have completed all required steps, but I still get the
libvirt errors in ceilometer (both kvm and xen hypervisor).
Now I'm starting to wonder if rados-gw really is the right choice
here. Has anybody figured out a way to retrieve rbd meters with
ceilometer?

Regards,
Eugen

[1] https://bugs.launchpad.net/ceilometer/+bug/1457440
[2]
http://docs.openstack.org/admin-guide/telemetry-measurements.html#ceph-object-storage

--
Eugen Block voice : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail : eblock@nde.ag

     Vorsitzende des Aufsichtsrates: Angelika Mozdzen
       Sitz und Registergericht: Hamburg, HRB 90934
               Vorstand: Jens-U. Mozdzen
                USt-IdNr. DE 814 013 983
responded Jun 29, 2016 by Eugen_Block (3,740 points)   2 2
0 votes

Hi Eugen,

Le 2016-06-29 16:30, Eugen Block a écrit :

This bug [1] describes the issue, but it seems to be a libvirt issue,
not ceilometer.

According to [2] it should be possible to retrieve those meters.
I followed ceph docs to install the rados-gateway, I integrated
keystone authentication, at least I don't get any errors regarding
authentication and swift command seems to work.
Then I added the meters described in [2] to the
/etc/ceilometer/pipeline.yaml

Ceph RBD and Ceph RadosGW are differents applications (both on top Ceph
librados API)

So from Ceilometer point of view, meters collected on each applications
are different meters, we currently have:

  • instances disks IOPS meters retrieved by Ceilometer throught libvirt.
    (And currently broken for rbd backed instances)
  • radosgw meters retrieved by ceilometer-polling-agent by polling the
    Ceph RadosGW API directly.

I believe I have completed all required steps, but I still get the
libvirt errors in ceilometer (both kvm and xen hypervisor).

If you got this error, that means you did the thing right, but it's
bugged due to [1] as you notice.

Now I'm starting to wonder if rados-gw really is the right choice
here.

Yes, rados-gw meters seems unrelated to your use-case.

Has anybody figured out a way to retrieve rbd meters with ceilometer?

That depends on the use-cases, if you want to get:

  • instances disks IOPS, ceilometer-compute-agent and libvirt have to be
    improved to retrieve them (that means fixing [1]).
  • the raw rbd meters (image size and utilization), a new
    ceilometer-polling-agent plugin needs to be written.
    (A poller that stat all rbd images directly have chance to be really
    really slow).

AFAICR, nobody is currently working those two points, so contributions
welcome.

Cheers,
--
Mehdi Abaakouk
mail: sileht@sileht.net
irc: sileht

responded Jun 29, 2016 by Mehdi_Abaakouk (4,040 points)   2 5
0 votes

Thanks for your reply!

I can totally check reverse lookups as a possible culprit. Has anyone else
experienced this where console access is slow in Horizon? We're using novnc
to view the console. I have noticed that although response time for a
Windows 10 VM using a mouse is still slow, using Firefox browser seems to
respond better (faster) then using something like Google Chrome for
example.

Any other ideas would be greatly appreciated.

Steve

On Tue, Jun 28, 2016 at 10:52 PM, Ken D'Ambrosio ken@jots.org wrote:

My rule-of-thumb for stuff that falls into the "really slow but works"
camp is, check DNS. If possible, disable reverse lookups, or else ensure
that the hosts involved all know how to reverse-resolve each other's IP
addresses.

That's where I'd start, anyway. Good luck!

-Ken

On 2016-06-28 20:28, Sterdnot Shaken wrote:

Does anyone have any idea why a console session's would be so slow from
horizon? After creating an instance, I go to Compute, Instance, then
Console tab where I click "Click here to show only console". From there,
it's EXTREMELY slow interacting with the instance itself... I can RDP or
ssh to the instance and speed is fine, so it seems like it's got something
to do with the console interaction with the instance. Is this a VNC issue?

Response time is fine for everything else with Horizon and Instances, it's
just interacting with the console that's slow.

Any help would be greatly appreciated!

Thanks


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Jun 30, 2016 by Sterdnot_Shaken (900 points)   2 4 10
0 votes

Hi,

I got the similar problem recently and reported it.
https://bugs.launchpad.net/horizon/+bug/1592323

As you said, it seems related to what kind of browser you are using.
For me, it is working well after changing the browser from Chrome to
Firefox.

Hope this helps.

Dongwon

On Thu, Jun 30, 2016 at 1:30 PM, Sterdnot Shaken sterdnotshaken@gmail.com
wrote:

Thanks for your reply!

I can totally check reverse lookups as a possible culprit. Has anyone else
experienced this where console access is slow in Horizon? We're using novnc
to view the console. I have noticed that although response time for a
Windows 10 VM using a mouse is still slow, using Firefox browser seems to
respond better (faster) then using something like Google Chrome for
example.

Any other ideas would be greatly appreciated.

Steve

On Tue, Jun 28, 2016 at 10:52 PM, Ken D'Ambrosio ken@jots.org wrote:

My rule-of-thumb for stuff that falls into the "really slow but works"
camp is, check DNS. If possible, disable reverse lookups, or else ensure
that the hosts involved all know how to reverse-resolve each other's IP
addresses.

That's where I'd start, anyway. Good luck!

-Ken

On 2016-06-28 20:28, Sterdnot Shaken wrote:

Does anyone have any idea why a console session's would be so slow from
horizon? After creating an instance, I go to Compute, Instance, then
Console tab where I click "Click here to show only console". From there,
it's EXTREMELY slow interacting with the instance itself... I can RDP or
ssh to the instance and speed is fine, so it seems like it's got something
to do with the console interaction with the instance. Is this a VNC issue?

Response time is fine for everything else with Horizon and Instances,
it's just interacting with the console that's slow.

Any help would be greatly appreciated!

Thanks


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Jun 30, 2016 by Dongwon_Cho (160 points)   1
0 votes

Thanks for the explanation, that actually helps a lot (and makes me
realise that I did that for nothing ;-) )!
I'll be out of office for the next three weeks, so any attempt to
contribute will start after my vacation ;-)

Regards,
Eugen

Zitat von Mehdi Abaakouk sileht@sileht.net:

Hi Eugen,

Le 2016-06-29 16:30, Eugen Block a écrit :

This bug [1] describes the issue, but it seems to be a libvirt issue,
not ceilometer.

According to [2] it should be possible to retrieve those meters.
I followed ceph docs to install the rados-gateway, I integrated
keystone authentication, at least I don't get any errors regarding
authentication and swift command seems to work.
Then I added the meters described in [2] to the
/etc/ceilometer/pipeline.yaml

Ceph RBD and Ceph RadosGW are differents applications (both on top
Ceph librados API)

So from Ceilometer point of view, meters collected on each
applications are different meters, we currently have:

  • instances disks IOPS meters retrieved by Ceilometer throught
    libvirt. (And currently broken for rbd backed instances)
  • radosgw meters retrieved by ceilometer-polling-agent by polling
    the Ceph RadosGW API directly.

I believe I have completed all required steps, but I still get the
libvirt errors in ceilometer (both kvm and xen hypervisor).

If you got this error, that means you did the thing right, but it's
bugged due to [1] as you notice.

Now I'm starting to wonder if rados-gw really is the right choice
here.

Yes, rados-gw meters seems unrelated to your use-case.

Has anybody figured out a way to retrieve rbd meters with ceilometer?

That depends on the use-cases, if you want to get:

  • instances disks IOPS, ceilometer-compute-agent and libvirt have to
    be improved to retrieve them (that means fixing [1]).
  • the raw rbd meters (image size and utilization), a new
    ceilometer-polling-agent plugin needs to be written.
    (A poller that stat all rbd images directly have chance to be
    really really slow).

AFAICR, nobody is currently working those two points, so
contributions welcome.

Cheers,
--
Mehdi Abaakouk
mail: sileht@sileht.net
irc: sileht

--
Eugen Block voice : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail : eblock@nde.ag

     Vorsitzende des Aufsichtsrates: Angelika Mozdzen
       Sitz und Registergericht: Hamburg, HRB 90934
               Vorstand: Jens-U. Mozdzen
                USt-IdNr. DE 814 013 983


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Jul 1, 2016 by Eugen_Block (3,740 points)   2 2
...