settingsLogin | Registersettings

[Openstack] guest rbd block device speed limit

0 votes

Hello:
I am using openstack + libvirt + qemu-kvm. The speed of copying files between virtual machines exceeds 300M Byte per second but the speed of dd command:
"watch dd oflag=direct,nonblock if=/dev/zero of=/opt/iotest1 bs=4M count=10" is around 20M byte per second. Every guest is the same, and there is no io tuning in libvirt. What can I do to unleash the speed of disk io speed inside the guest?
Thanks!

The libvirt vm xml is as follow:


instance-00000663
d9bda7d7-f68a-45ea-84a1-f50faa44b801



smtestiotune
2017-08-23 07:38:13

1024
50
0
0
1


admin
admin




1048576
1048576
1

1024


/machine



Fedora Project
OpenStack Nova
13.1.0-1.el7
0994820c-3e23-4cc0-9b2c-f34fb88dee4e
d9bda7d7-f68a-45ea-84a1-f50faa44b801
Virtual Machine



hvm




<acpi/>
<apic/>










destroy
restart
destroy

/usr/libexec/qemu-kvm







<backingStore/>































































chagg@foxmail.com


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Aug 24, 2017 in openstack by chagg_at_foxmail.com (320 points)   1

4 Responses

0 votes

On 08/24/2017 01:04 AM, chagg@foxmail.com wrote:
Hello:
I am using openstack + libvirt + qemu-kvm. The speed of copying files
between virtual machines exceeds 300M Byte per second but the speed of dd command:
"watch dd oflag=direct,nonblock if=/dev/zero of=/opt/iotest1 bs=4M count=10"
is around 20M byte per second. Every guest is the same, and there is no io
tuning in libvirt. What can I do to unleash the speed of disk io speed inside
the guest?
Thanks!

It looks like you're using ceph for your root disk. What sort of performance do
you get accessing the ceph volumes from the host?

Chris

responded Aug 24, 2017 by Chris_Friesen (20,420 points)   3 15 24
0 votes

This is my ceph performance data:

Total time run: 30.370630
Total writes made: 849
Write size: 4194304
Bandwidth (MB/sec): 111.819
Stddev Bandwidth: 21.6665
Max bandwidth (MB/sec): 124
Min bandwidth (MB/sec): 0
Average IOPS: 27
Average Latency(s): 0.572236
Stddev Latency(s): 0.421846
Max latency(s): 4.06547
Min latency(s): 0.121321

Thanks!

chagg@foxmail.com

From: Chris Friesen
Date: 2017-08-24 23:14
To: openstack@lists.openstack.org
Subject: Re: [Openstack] guest rbd block device speed limit
On 08/24/2017 01:04 AM, chagg@foxmail.com wrote:
Hello:
I am using openstack + libvirt + qemu-kvm. The speed of copying files
between virtual machines exceeds 300M Byte per second but the speed of dd command:
"watch dd oflag=direct,nonblock if=/dev/zero of=/opt/iotest1 bs=4M count=10"
is around 20M byte per second. Every guest is the same, and there is no io
tuning in libvirt. What can I do to unleash the speed of disk io speed inside
the guest?
Thanks!

It looks like you're using ceph for your root disk. What sort of performance do
you get accessing the ceph volumes from the host?

Chris


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Aug 25, 2017 by chagg_at_foxmail.com (320 points)   1
0 votes

I use this command to launch a VM. It disk io is also about 20M Bps,Why?

/usr/libexec/qemu-kvm -m 1024 -drive format=rbd,file=rbd:volumes/d9bda7d7-f68a-45ea-84a1-f50faa44b801_disk,cache=writeback -vnc :10 -usb -usbdevice tablet

Thanks!

chagg@foxmail.com

From: chagg@foxmail.com
Date: 2017-08-25 10:05
To: Chris Friesen; openstack
Subject: Re: Re: [Openstack] guest rbd block device speed limit
This is my ceph performance data:

Total time run: 30.370630
Total writes made: 849
Write size: 4194304
Bandwidth (MB/sec): 111.819
Stddev Bandwidth: 21.6665
Max bandwidth (MB/sec): 124
Min bandwidth (MB/sec): 0
Average IOPS: 27
Average Latency(s): 0.572236
Stddev Latency(s): 0.421846
Max latency(s): 4.06547
Min latency(s): 0.121321

Thanks!

chagg@foxmail.com

From: Chris Friesen
Date: 2017-08-24 23:14
To: openstack@lists.openstack.org
Subject: Re: [Openstack] guest rbd block device speed limit
On 08/24/2017 01:04 AM, chagg@foxmail.com wrote:
Hello:
I am using openstack + libvirt + qemu-kvm. The speed of copying files
between virtual machines exceeds 300M Byte per second but the speed of dd command:
"watch dd oflag=direct,nonblock if=/dev/zero of=/opt/iotest1 bs=4M count=10"
is around 20M byte per second. Every guest is the same, and there is no io
tuning in libvirt. What can I do to unleash the speed of disk io speed inside
the guest?
Thanks!

It looks like you're using ceph for your root disk. What sort of performance do
you get accessing the ceph volumes from the host?

Chris


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Aug 25, 2017 by chagg_at_foxmail.com (320 points)   1
0 votes

I'm afraid I can't give you any pointers...hopefully someone with ceph tuning
expertise could comment.

(You might try the ceph mailing list at ceph-users@ceph.com)

Chris

On 08/25/2017 01:30 AM, chagg@foxmail.com wrote:
I use this command to launch a VM. It disk io is also about 20M Bps,Why?

/usr/libexec/qemu-kvm -m 1024 -drive format=rbd,file=rbd:volumes/d9bda7d7-f68a-45ea-84a1-f50faa44b801_disk,cache=writeback -vnc :10 -usb -usbdevice tablet

Thanks!


chagg@foxmail.com
*From:* chagg@foxmail.com <mailto:chagg@foxmail.com>
*Date:* 2017-08-25 10:05
*To:* Chris Friesen <mailto:chris.friesen@windriver.com>; openstack
<mailto:openstack@lists.openstack.org>
*Subject:* Re: Re: [Openstack] guest rbd block device speed limit
This is my ceph performance data:

  Total time run:         30.370630
Total writes made:      849
Write size:             4194304
Bandwidth (MB/sec):     111.819
Stddev Bandwidth:       21.6665
Max bandwidth (MB/sec): 124
Min bandwidth (MB/sec): 0
Average IOPS:           27
Average Latency(s):     0.572236
Stddev Latency(s):      0.421846
Max latency(s):         4.06547
Min latency(s):         0.121321

Thanks!
--------------------------------------------------------------------------------
chagg@foxmail.com

    *From:* Chris Friesen <mailto:chris.friesen@windriver.com>
    *Date:* 2017-08-24 23:14
    *To:* openstack@lists.openstack.org <mailto:openstack@lists.openstack.org>
    *Subject:* Re: [Openstack] guest rbd block device speed limit
    On 08/24/2017 01:04 AM, chagg@foxmail.com wrote:
     > Hello:
     >      I  am using openstack + libvirt + qemu-kvm. The speed of copying
    files
     > between virtual machines exceeds 300M Byte per second but the speed
    of dd command:
     >   "watch dd oflag=direct,nonblock if=/dev/zero of=/opt/iotest1 bs=4M
    count=10"
     >   is around 20M byte per second. Every guest is the same, and there
    is no io
     > tuning in libvirt. What can I do to unleash the speed of disk io
    speed inside
     > the guest?
     >      Thanks!
    It looks like you're using ceph for your root disk.  What sort of
    performance do
    you get accessing the ceph volumes from the host?
    Chris
    _______________________________________________
    Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
    Post to     : openstack@lists.openstack.org
    Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Aug 25, 2017 by Chris_Friesen (20,420 points)   3 15 24
...