settingsLogin | Registersettings

[Openstack] GlusterFS and Openstack Newton

0 votes

Hello,

We have a new build going up in our DC of Openstack Newton. We wanted to
build in a shared storage solution and really liked the simplicity as well
as functions of glusterFS. This would allow us to perform live migrations
along with Geo replication. The issue we have been having is getting
nova-libvirt instances to run on the compute nodes with the glusterfs mount
point of /var/lib/nova/instances.

We have added all the required permissions on the volume share :

Volume Name: gfsimgstore

Type: Replicate

Volume ID: 768d161f-78ca-40dd-befc-ddf9de2ccb38

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: cloud304-node1:/bricks/imgstore1

Brick2: cloud304-node2:/bricks/imgstore1

Options Reconfigured:

cluster.data-self-heal-algorithm: full

features.shard: on

cluster.server-quorum-type: server

cluster.quorum-type: auto

network.remote-dio: enable

cluster.eager-lock: enable

performance.stat-prefetch: off

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

server.allow-insecure: on

storage.owner-gid: 162

storage.owner-uid: 162

transport.address-family: inet

performance.readdir-ahead: on

nfs.disable: on

We have modified permissions following what documentation we were able to
locate, but we still get errors when we try to create a VM. The errors are
a lot but this is the final error that stands out:

2017-01-23 18:29:25.798 12184 ERROR nova.compute.manager [instance:
c6634e67-b293-4424-96ec-f0c58b2bf081] libvirtError: Unable to open file:
/var/lib/nova/instances/c6634e67-b293-4424-96ec-f0c58b2bf081/console.log:
Permission denied 2017-01-23 18:29:25.798 12184 ERROR

I am hoping I can find someone running glusterfs and can offer some insight
to our issue.

James Fleet

James R. Fleet
Innovative Solutions Technology
888.809.0223 ext.702


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
asked Jan 24, 2017 in openstack by James_Fleet (340 points)   1 2

4 Responses

0 votes

Hi James,

I'm pretty new to OpenStack, but I'm working on setting up exactly the same
thing right now. I'm having some other issues a little before where you
are with my stonith device so I don't really have any insight on your exact
problem. If I get mine to work I'll share what I did.

However, my first thought is SELinux. If you've checked file permissions
and they look ok, is SELinux Enforcing? If so, you might try to
temporarily set it to permissive. If that fixes the problem then check
audit logs for what you're missing. It could be a missing context.

Just my two cents.

On Tue, Jan 24, 2017 at 9:51 AM, James Fleet jrfleet@istech-corp.com
wrote:

Hello,

We have a new build going up in our DC of Openstack Newton. We wanted to
build in a shared storage solution and really liked the simplicity as well
as functions of glusterFS. This would allow us to perform live migrations
along with Geo replication. The issue we have been having is getting
nova-libvirt instances to run on the compute nodes with the glusterfs mount
point of /var/lib/nova/instances.

We have added all the required permissions on the volume share :

Volume Name: gfsimgstore

Type: Replicate

Volume ID: 768d161f-78ca-40dd-befc-ddf9de2ccb38

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: cloud304-node1:/bricks/imgstore1

Brick2: cloud304-node2:/bricks/imgstore1

Options Reconfigured:

cluster.data-self-heal-algorithm: full

features.shard: on

cluster.server-quorum-type: server

cluster.quorum-type: auto

network.remote-dio: enable

cluster.eager-lock: enable

performance.stat-prefetch: off

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

server.allow-insecure: on

storage.owner-gid: 162

storage.owner-uid: 162

transport.address-family: inet

performance.readdir-ahead: on

nfs.disable: on

We have modified permissions following what documentation we were able to
locate, but we still get errors when we try to create a VM. The errors are
a lot but this is the final error that stands out:

2017-01-23 18:29:25.798 12184 ERROR nova.compute.manager [instance:
c6634e67-b293-4424-96ec-f0c58b2bf081] libvirtError: Unable to open file:
/var/lib/nova/instances/c6634e67-b293-4424-96ec-f0c58b2bf081/console.log:
Permission denied 2017-01-23 18:29:25.798 12184 ERROR

I am hoping I can find someone running glusterfs and can offer some
insight to our issue.

James Fleet

James R. Fleet
Innovative Solutions Technology
888.809.0223 ext.702 <(888)%20809-0223>


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Jan 24, 2017 by Peter_Kirby (460 points)   1 1 2
0 votes

Hello Peter,

Yes, the command to add it to SElinux is setsebool -P virtusefusefs on
which allows the client to connect using SElInux.

James R. Fleet
Innovative Solutions Technology
484 Williamsport Pike #135
Martinsburg, WV 25404
888.809.0223 ext.702

On Tue, Jan 24, 2017 at 11:17 AM, Peter Kirby peter.kirby@objectstream.com
wrote:

Hi James,

I'm pretty new to OpenStack, but I'm working on setting up exactly the
same thing right now. I'm having some other issues a little before where
you are with my stonith device so I don't really have any insight on your
exact problem. If I get mine to work I'll share what I did.

However, my first thought is SELinux. If you've checked file permissions
and they look ok, is SELinux Enforcing? If so, you might try to
temporarily set it to permissive. If that fixes the problem then check
audit logs for what you're missing. It could be a missing context.

Just my two cents.

On Tue, Jan 24, 2017 at 9:51 AM, James Fleet jrfleet@istech-corp.com
wrote:

Hello,

We have a new build going up in our DC of Openstack Newton. We wanted to
build in a shared storage solution and really liked the simplicity as well
as functions of glusterFS. This would allow us to perform live migrations
along with Geo replication. The issue we have been having is getting
nova-libvirt instances to run on the compute nodes with the glusterfs mount
point of /var/lib/nova/instances.

We have added all the required permissions on the volume share :

Volume Name: gfsimgstore

Type: Replicate

Volume ID: 768d161f-78ca-40dd-befc-ddf9de2ccb38

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: cloud304-node1:/bricks/imgstore1

Brick2: cloud304-node2:/bricks/imgstore1

Options Reconfigured:

cluster.data-self-heal-algorithm: full

features.shard: on

cluster.server-quorum-type: server

cluster.quorum-type: auto

network.remote-dio: enable

cluster.eager-lock: enable

performance.stat-prefetch: off

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

server.allow-insecure: on

storage.owner-gid: 162

storage.owner-uid: 162

transport.address-family: inet

performance.readdir-ahead: on

nfs.disable: on

We have modified permissions following what documentation we were able to
locate, but we still get errors when we try to create a VM. The errors are
a lot but this is the final error that stands out:

2017-01-23 18:29:25.798 12184 ERROR nova.compute.manager [instance:
c6634e67-b293-4424-96ec-f0c58b2bf081] libvirtError: Unable to open file:
/var/lib/nova/instances/c6634e67-b293-4424-96ec-f0c58b2bf081/console.log:
Permission denied 2017-01-23 18:29:25.798 12184 ERROR

I am hoping I can find someone running glusterfs and can offer some
insight to our issue.

James Fleet

James R. Fleet
Innovative Solutions Technology
888.809.0223 ext.702 <(888)%20809-0223>


Mailing list: http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Jan 24, 2017 by James_Fleet (340 points)   1 2
0 votes

If you use audit2why on your audit log, is there anything in there
suggesting SELinux is blocking the disk access?

On Tue, Jan 24, 2017 at 12:12 PM, James Fleet jrfleet@istech-corp.com
wrote:

Hello Peter,

Yes, the command to add it to SElinux is setsebool -P virtusefusefs on
which allows the client to connect using SElInux.

James R. Fleet
Innovative Solutions Technology
484 Williamsport Pike #135
Martinsburg, WV 25404
888.809.0223 ext.702 <(888)%20809-0223>

On Tue, Jan 24, 2017 at 11:17 AM, Peter Kirby <
peter.kirby@objectstream.com> wrote:

Hi James,

I'm pretty new to OpenStack, but I'm working on setting up exactly the
same thing right now. I'm having some other issues a little before where
you are with my stonith device so I don't really have any insight on your
exact problem. If I get mine to work I'll share what I did.

However, my first thought is SELinux. If you've checked file permissions
and they look ok, is SELinux Enforcing? If so, you might try to
temporarily set it to permissive. If that fixes the problem then check
audit logs for what you're missing. It could be a missing context.

Just my two cents.

On Tue, Jan 24, 2017 at 9:51 AM, James Fleet jrfleet@istech-corp.com
wrote:

Hello,

We have a new build going up in our DC of Openstack Newton. We wanted to
build in a shared storage solution and really liked the simplicity as well
as functions of glusterFS. This would allow us to perform live migrations
along with Geo replication. The issue we have been having is getting
nova-libvirt instances to run on the compute nodes with the glusterfs mount
point of /var/lib/nova/instances.

We have added all the required permissions on the volume share :

Volume Name: gfsimgstore

Type: Replicate

Volume ID: 768d161f-78ca-40dd-befc-ddf9de2ccb38

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: cloud304-node1:/bricks/imgstore1

Brick2: cloud304-node2:/bricks/imgstore1

Options Reconfigured:

cluster.data-self-heal-algorithm: full

features.shard: on

cluster.server-quorum-type: server

cluster.quorum-type: auto

network.remote-dio: enable

cluster.eager-lock: enable

performance.stat-prefetch: off

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

server.allow-insecure: on

storage.owner-gid: 162

storage.owner-uid: 162

transport.address-family: inet

performance.readdir-ahead: on

nfs.disable: on

We have modified permissions following what documentation we were able
to locate, but we still get errors when we try to create a VM. The errors
are a lot but this is the final error that stands out:

2017-01-23 18:29:25.798 12184 ERROR nova.compute.manager [instance:
c6634e67-b293-4424-96ec-f0c58b2bf081] libvirtError: Unable to open
file: /var/lib/nova/instances/c6634e67-b293-4424-96ec-f0c58b2bf081/console.log:
Permission denied 2017-01-23 18:29:25.798 12184 ERROR

I am hoping I can find someone running glusterfs and can offer some
insight to our issue.

James Fleet

James R. Fleet
Innovative Solutions Technology
888.809.0223 ext.702 <(888)%20809-0223>


Mailing list: http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Jan 24, 2017 by Peter_Kirby (460 points)   1 1 2
0 votes

That is a good idea Peter to troubleshoot the disk access its not making
sense I am able to mount the volume and write files to it but it seems a
permission thing with libvirt and Nova.

James Fleet

James R. Fleet
Innovative Solutions Technology
484 Williamsport Pike #135
Martinsburg, WV 25404
888.809.0223 ext.702

On Tue, Jan 24, 2017 at 2:44 PM, Peter Kirby peter.kirby@objectstream.com
wrote:

If you use audit2why on your audit log, is there anything in there
suggesting SELinux is blocking the disk access?

On Tue, Jan 24, 2017 at 12:12 PM, James Fleet jrfleet@istech-corp.com
wrote:

Hello Peter,

Yes, the command to add it to SElinux is setsebool -P virtusefusefs on
which allows the client to connect using SElInux.

James R. Fleet
Innovative Solutions Technology
484 Williamsport Pike #135
Martinsburg, WV 25404
888.809.0223 ext.702 <(888)%20809-0223>

On Tue, Jan 24, 2017 at 11:17 AM, Peter Kirby <
peter.kirby@objectstream.com> wrote:

Hi James,

I'm pretty new to OpenStack, but I'm working on setting up exactly the
same thing right now. I'm having some other issues a little before where
you are with my stonith device so I don't really have any insight on your
exact problem. If I get mine to work I'll share what I did.

However, my first thought is SELinux. If you've checked file
permissions and they look ok, is SELinux Enforcing? If so, you might try
to temporarily set it to permissive. If that fixes the problem then check
audit logs for what you're missing. It could be a missing context.

Just my two cents.

On Tue, Jan 24, 2017 at 9:51 AM, James Fleet jrfleet@istech-corp.com
wrote:

Hello,

We have a new build going up in our DC of Openstack Newton. We wanted
to build in a shared storage solution and really liked the simplicity as
well as functions of glusterFS. This would allow us to perform live
migrations along with Geo replication. The issue we have been having is
getting nova-libvirt instances to run on the compute nodes with the
glusterfs mount point of /var/lib/nova/instances.

We have added all the required permissions on the volume share :

Volume Name: gfsimgstore

Type: Replicate

Volume ID: 768d161f-78ca-40dd-befc-ddf9de2ccb38

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: cloud304-node1:/bricks/imgstore1

Brick2: cloud304-node2:/bricks/imgstore1

Options Reconfigured:

cluster.data-self-heal-algorithm: full

features.shard: on

cluster.server-quorum-type: server

cluster.quorum-type: auto

network.remote-dio: enable

cluster.eager-lock: enable

performance.stat-prefetch: off

performance.io-cache: off

performance.read-ahead: off

performance.quick-read: off

server.allow-insecure: on

storage.owner-gid: 162

storage.owner-uid: 162

transport.address-family: inet

performance.readdir-ahead: on

nfs.disable: on

We have modified permissions following what documentation we were able
to locate, but we still get errors when we try to create a VM. The errors
are a lot but this is the final error that stands out:

2017-01-23 18:29:25.798 12184 ERROR nova.compute.manager [instance:
c6634e67-b293-4424-96ec-f0c58b2bf081] libvirtError: Unable to open
file: /var/lib/nova/instances/c6634e67-b293-4424-96ec-f0c58b2bf081/console.log:
Permission denied 2017-01-23 18:29:25.798 12184 ERROR

I am hoping I can find someone running glusterfs and can offer some
insight to our issue.

James Fleet

James R. Fleet
Innovative Solutions Technology
888.809.0223 ext.702 <(888)%20809-0223>


Mailing list: http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi
-bin/mailman/listinfo/openstack


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded Jan 25, 2017 by James_Fleet (340 points)   1 2
...