settingsLogin | Registersettings

[Openstack] Multiple private nets and routing

0 votes

Hi,

I am setting up replica of my real-world deployment in terms of networks
within single instance of OpenStack. For that I have to create 3
networks (1 for each tier): web, middleware, db.

I have created those networks successfully then I've added the routers
between respective networks, yet my web tier can't reach middleware and
middleware can't reach DB using those private nets.

I have created a separate "public" network to which all those nets can
be routed (so that I can access VMs directly). This one works just fine.

Most details provided in gist:

https://gist.github.com/droopy4096/0008581552e63710341b

To simplify: this time around I've used dashboard to create network
infrastructure. So procedure I've followed:

  • created each network (front, mid, db, public) with according subnet.
  • created routers "bridging" specific pairs of network, e.g.:
    gbfrontmid_router is connecting front tier (web) and mid tier
    (middleware).
  • for each tier created secgroup with corresponding rules
  • created VMs for each tier assigned to specific private network with
    specific secgroups applied
  • checked that default secgroup seems to be allowing everything in (see
    gist)

pinging from systest-front to systest-mid (on 10.10/16 IPs) fails so
far. Same goes for pings from mid to db etc.

what am I missing? Why traffic from one private net can't reach another?
Anything needs to be added to configuration?

--
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245


Confidence is what you have before you understand the problem
Woody Allen

When in trouble when in doubt run in circles scream and shout
http://www.wordwizard.com/phpbb3/viewtopic.php?f=16&t=19330


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

asked May 21, 2015 in openstack by Dmitry_Makovey (700 points)   2 3

22 Responses

0 votes

Dmitry Makovey wrote:
I am setting up replica of my real-world deployment in terms of networks
within single instance of OpenStack. For that I have to create 3
networks (1 for each tier): web, middleware, db.

I have created those networks successfully then I've added the routers
between respective networks, yet my web tier can't reach middleware and
middleware can't reach DB using those private nets.

I have created a separate "public" network to which all those nets can
be routed (so that I can access VMs directly). This one works just fine.

Most details provided in gist:

https://gist.github.com/droopy4096/0008581552e63710341b

To simplify: this time around I've used dashboard to create network
infrastructure. So procedure I've followed:

  • created each network (front, mid, db, public) with according subnet.
  • created routers "bridging" specific pairs of network, e.g.:
    gbfrontmid_router is connecting front tier (web) and mid tier
    (middleware).
  • for each tier created secgroup with corresponding rules
  • created VMs for each tier assigned to specific private network with
    specific secgroups applied
  • checked that default secgroup seems to be allowing everything in (see
    gist)

pinging from systest-front to systest-mid (on10.10/16 IPs) fails so
far. Same goes for pings from mid to db etc.

what am I missing? Why traffic from one private net can't reach another?
Anything needs to be added to configuration?

Dmitry,

Could you do a neutron router-show on each of those routers? Maybe just
add it to the existing gist?

Regards,

Richard Raseley

SysOps Engineer @ Puppet Labs

responded May 22, 2015 by Richard_Raseley (3,060 points)   4 5
0 votes

On 05/22/2015 11:43 AM, Richard Raseley wrote:

Could you do a neutron router-show on each of those routers? Maybe just
add it to the existing gist?

absolutely - I have just updated Gist to include router-show results
(second file there)

Regards,

Richard Raseley

SysOps Engineer @ Puppet Labs

--
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245


Confidence is what you have before you understand the problem
Woody Allen

When in trouble when in doubt run in circles scream and shout
http://www.wordwizard.com/phpbb3/viewtopic.php?f=16&t=19330


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded May 22, 2015 by Dmitry_Makovey (700 points)   2 3
0 votes

Dmitry Makovey wrote:
absolutely - I have just updated Gist to include router-show results
(second file there)

Dmitry,

After trying to take the data you've dumped here and trying to
whiteboard it out (to internalize the model) I have to say it isn't
totally clear to me how all these bits are talking to each other. To
help me visualize, can you do a screen-grab of the Horizon network overview?

Alternately, is this provisioned via a Heat template such that I could
replicate it in my environment?

Regards,

Richard


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded May 22, 2015 by Richard_Raseley (3,060 points)   4 5
0 votes

On 05/22/2015 12:29 PM, Richard Raseley wrote:
Dmitry Makovey wrote:

absolutely - I have just updated Gist to include router-show results
(second file there)

Dmitry,

After trying to take the data you've dumped here and trying to
whiteboard it out (to internalize the model) I have to say it isn't
totally clear to me how all these bits are talking to each other. To
help me visualize, can you do a screen-grab of the Horizon network
overview?

done. Link to image attached to gist

Alternately, is this provisioned via a Heat template such that I could
replicate it in my environment?

no, this is a manual implementation - no HEAT template.

--
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245


Confidence is what you have before you understand the problem
Woody Allen

When in trouble when in doubt run in circles scream and shout
http://www.wordwizard.com/phpbb3/viewtopic.php?f=16&t=19330


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded May 22, 2015 by Dmitry_Makovey (700 points)   2 3
0 votes

Dmitry Makovey wrote:
done. Link to image attached to gist

Thank you, this helps me better understand.

Can you share what the routes on your instances look like? You'll
obviously need to let the instance in each network how to get to the
other networks over the 'secondary' router (assuming your addressing
them by their RFC5735 addresses), otherwise the packets will hit the
default gateway (the routers connected to your ext_net) and get dropped.

Regards,

Richard


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded May 22, 2015 by Richard_Raseley (3,060 points)   4 5
0 votes

On 05/22/2015 01:30 PM, Dmitry Makovey wrote:
done. Link to image attached to gist

note: connections to "public" network are "optional" as per my original
email - I use them only to get direct access to VMs

Alternately, is this provisioned via a Heat template such that I could
replicate it in my environment?

no, this is a manual implementation - no HEAT template.

--
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245


Confidence is what you have before you understand the problem
Woody Allen

When in trouble when in doubt run in circles scream and shout
http://www.wordwizard.com/phpbb3/viewtopic.php?f=16&t=19330


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded May 22, 2015 by Dmitry_Makovey (700 points)   2 3
0 votes

On 05/22/2015 01:39 PM, Richard Raseley wrote:
Dmitry Makovey wrote:

done. Link to image attached to gist

Thank you, this helps me better understand.

first of all - thank you very much for bearing with me on this one. ;)

Can you share what the routes on your instances look like? You'll
obviously need to let the instance in each network how to get to the
other networks over the 'secondary' router (assuming your addressing
them by their RFC5735 addresses), otherwise the packets will hit the
default gateway (the routers connected to your ext_net) and get dropped.

I think you're onto something here as out of my own ignorance I've
assumed that OpenStack/neutron will handle routing auto-magically as I
set up this kind of infrastructure:

$ for i in 123.54.67.{141,144,145} ; do ssh cloud-user@${i} netstat -nar
; done
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt
Iface
10.10.31.0 0.0.0.0 255.255.255.0 U 0 0 0
eth0
0.0.0.0 10.10.31.1 0.0.0.0 UG 0 0 0
eth0
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt
Iface
10.10.25.0 0.0.0.0 255.255.255.0 U 0 0 0
eth0
0.0.0.0 10.10.25.1 0.0.0.0 UG 0 0 0
eth0
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt
Iface
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0
eth0
0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0
eth0

So how shall I go about setting it up? do I need to spin up some
"dual-nic" VM that would act as a router or can I use some other
OpenStack facilities for that?

--
Dmitry Makovey
Web Systems Administrator
Athabasca University
(780) 675-6245


Confidence is what you have before you understand the problem
Woody Allen

When in trouble when in doubt run in circles scream and shout
http://www.wordwizard.com/phpbb3/viewtopic.php?f=16&t=19330


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

responded May 22, 2015 by Dmitry_Makovey (700 points)   2 3
0 votes

Dmitry Makovey wrote:
first of all - thank you very much for bearing with me on this one.;)

My pleasure, happy to help!

So how shall I go about setting it up? do I need to spin up some
"dual-nic" VM that would act as a router or can I use some other
OpenStack facilities for that?

You won't need a dual NIC VM, but rather to just add a route on each VM
that tells it that, for hosts which have addresses in the other
network(s), to not use the default gateway (which is the ext_net
attached router), but to send their traffic via the other ('private')
router's interface.

So, for example on the instance which you have attached to the
'privategbdb_net' network you would have two routes:

  • ip route add -net 10.10.31.0 netmask 255.255.255.0 gw 10.10.10.4

  • ip route add -net 10.10.25.0 netmask 255.255.255.0 gw 10.10.10.6

As of now, the default gateway is the only route they know of, so while
it is true that the Neutron routers themselves have automatic knowledge
of their attached networks, that is separate from the instance's
knowledge of which routers are available.

I hope that helps.

Regards,

Richard


Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
responded May 22, 2015 by Richard_Raseley (3,060 points)   4 5
0 votes

Hi List,

We're trying to install Kilo on RHEL7, when we get to creating service
endpoints, this command:

openstack service create
--name keystone --description "OpenStack Identity" identity

Consistently returns this error:

ERROR: openstack An unexpected error prevented the server from
fulfilling your request. (HTTP 500)

Looking into keystone.log we see that the final error reads:

(OperationalError) (1045, "Access denied for user 'keystone'@'localhost'
(using password: YES)") None None

Three of our team have tried reinstalling as per the instructions, we've
triple checked our config and database settings but are still lost.

Apologies if this question is terribly backward, but we've exhausted all
other ideas.

Thanks,

MikeL.

responded May 23, 2015 by michael_at_tropyx.co (380 points)   1 2 2
0 votes

----- Original Message -----
From: michael@tropyx.com
To: openstack@lists.openstack.org

Hi List,

We're trying to install Kilo on RHEL7, when we get to creating service
endpoints, this command:

openstack service create
--name keystone --description "OpenStack Identity" identity

Consistently returns this error:

ERROR: openstack An unexpected error prevented the server from
fulfilling your request. (HTTP 500)

Looking into keystone.log we see that the final error reads:

(OperationalError) (1045, "Access denied for user 'keystone'@'localhost'
(using password: YES)") None None

Three of our team have tried reinstalling as per the instructions, we've
triple checked our config and database settings but are still lost.

Apologies if this question is terribly backward, but we've exhausted all
other ideas.

Thanks,

MikeL.

It seems likely you are running into the issue Matt is trying to resolve here:

https://review.openstack.org/#/c/185120/1/doc/install-guide/section_keystone-verify.xml

Though I think the proper fix is a packaging one.

Thanks,

Steve

responded May 23, 2015 by Steve_Gordon (9,680 points)   2 5 6
...