settingsLogin | Registersettings

[openstack-dev] Supporting SSH host certificates

0 votes

Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates): without
host certificates, when clients ssh to a host for the first time (or after
the host has been re-installed), they have to hope that there's no man in
the middle and that the public key being presented actually belongs to the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the possibility of
Man-in-the-Middle attack: a Certificate Authority public key is distributed
to clients (and written to their known
hosts file with a special syntax and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its SSH
host certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key to
my CA server, sign it, and then write the certificate back into the host
(again via VNC). I cannot ssh without risking a MITM attack. What about
using Nova user-data? User-data is exposed via the metadata service.
Metadata is queried via http (reply transmitted in the clear, susceptible
to snooping), and any compute node can query for any instance's
meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I know
cloud-init allows specifying SSH private keys (both for users and for SSH
service). I have not yet studied how such information is securely injected
into an instance. I assume it should only be made available via ConfigDrive
rather than metadata-service (again, that service transmits in the clear).

What about providing SSH host certificates as a service in OpenStack? Let's
keep out of scope issues around choosing and storing the CA keys, but the
CA key is per project. What design supports setting up the SSH host
certificate automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public key
out; and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova
compute instance boot code. Nova compute can securely query a CA service
asking for a triplet (private key, public key, SSH certificate) for the
specific hostname. It can then inject the triplet using ConfigDrive. I
believe this securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can think of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 10, 2017 in openstack-dev by Giuseppe_de_Candia (340 points)  

15 Responses

0 votes

Giuseppe ,

I'm pretty sure this is the project you want ot look into:

http://git.openstack.org/cgit/openstack/barbican/

"Barbican is a ReST API designed for the secure storage, provisioning
and management of secrets, including in OpenStack environments."

-Jon

On Fri, Sep 29, 2017 at 02:21:06PM -0500, Giuseppe de Candia wrote:
:Hi Folks,
:
:
:
:My intent in this e-mail is to solicit advice for how to inject SSH host
:certificates into VM instances, with minimal or no burden on users.
:
:
:
:Background (skip if you're already familiar with SSH certificates): without
:host certificates, when clients ssh to a host for the first time (or after
:the host has been re-installed), they have to hope that there's no man in
:the middle and that the public key being presented actually belongs to the
:host they're trying to reach. The host's public key is stored in the
:client's knownhosts file. SSH host certicates eliminate the possibility of
:Man-in-the-Middle attack: a Certificate Authority public key is distributed
:to clients (and written to their known
hosts file with a special syntax and
:options); the host public key is signed by the CA, generating an SSH
:certificate that contains the hostname and validity period (among other
:things). When negotiating the ssh connection, the host presents its SSH
:host certificate and the client verifies that it was signed by the CA.
:
:
:
:How to support SSH host certificates in OpenStack?
:
:
:
:First, let's consider doing it by hand, instance by instance. The only
:solution I can think of is to VNC to the instance, copy the public key to
:my CA server, sign it, and then write the certificate back into the host
:(again via VNC). I cannot ssh without risking a MITM attack. What about
:using Nova user-data? User-data is exposed via the metadata service.
:Metadata is queried via http (reply transmitted in the clear, susceptible
:to snooping), and any compute node can query for any instance's
:meta-data/user-data.
:
:
:
:At this point I have to admit I'm ignorant of details of cloud-init. I know
:cloud-init allows specifying SSH private keys (both for users and for SSH
:service). I have not yet studied how such information is securely injected
:into an instance. I assume it should only be made available via ConfigDrive
:rather than metadata-service (again, that service transmits in the clear).
:
:
:
:What about providing SSH host certificates as a service in OpenStack? Let's
:keep out of scope issues around choosing and storing the CA keys, but the
:CA key is per project. What design supports setting up the SSH host
:certificate automatically for every VM instance?
:
:
:
:I have looked at Vendor Data and I don't see a way to use that, mainly
:because 1) it doesn't take parameters, so you can't pass the public key
:out; and 2) it's queried over http, not https.
:
:
:
:Just as a feasibility argument, one solution would be to modify Nova
:compute instance boot code. Nova compute can securely query a CA service
:asking for a triplet (private key, public key, SSH certificate) for the
:specific hostname. It can then inject the triplet using ConfigDrive. I
:believe this securely gets the private key into the instance.
:
:
:
:I cannot figure out how to get the equivalent functionality without
:modifying Nova compute and the boot process. Every solution I can think of
:risks either exposing the private key or vulnerability to a MITM attack
:during the signing process.
:
:
:
:Your help is appreciated.
:
:
:
:--Pino

:__________________________________________________________________________
:OpenStack Development Mailing List (not for usage questions)
:Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 29, 2017 by jon_at_csail.mit.edu (4,720 points)   1 4 6
0 votes

What you describe (at least the use case) seems to resemble
https://review.openstack.org/#/c/456394/ This work never moved
anywhere since the spec was posted though. You may want to revive the
discussion in scope of the spec.

Ihar

On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia
giuseppe.decandia@gmail.com wrote:
Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates): without
host certificates, when clients ssh to a host for the first time (or after
the host has been re-installed), they have to hope that there's no man in
the middle and that the public key being presented actually belongs to the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the possibility of
Man-in-the-Middle attack: a Certificate Authority public key is distributed
to clients (and written to their known
hosts file with a special syntax and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its SSH host
certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key to my
CA server, sign it, and then write the certificate back into the host (again
via VNC). I cannot ssh without risking a MITM attack. What about using Nova
user-data? User-data is exposed via the metadata service. Metadata is
queried via http (reply transmitted in the clear, susceptible to snooping),
and any compute node can query for any instance's meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I know
cloud-init allows specifying SSH private keys (both for users and for SSH
service). I have not yet studied how such information is securely injected
into an instance. I assume it should only be made available via ConfigDrive
rather than metadata-service (again, that service transmits in the clear).

What about providing SSH host certificates as a service in OpenStack? Let's
keep out of scope issues around choosing and storing the CA keys, but the CA
key is per project. What design supports setting up the SSH host certificate
automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public key out;
and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova compute
instance boot code. Nova compute can securely query a CA service asking for
a triplet (private key, public key, SSH certificate) for the specific
hostname. It can then inject the triplet using ConfigDrive. I believe this
securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can think of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 29, 2017 by Ihar_Hrachyshka (35,300 points)   3 9 16
0 votes

Ihar, thanks for pointing that out - I'll definitely take a close look.

Jon, I'm not very familiar with Barbican, but I did assume the full
implementation would use Barbican to store private keys. However, in terms
of actually getting a private key (or SSH host cert) into a VM instance,
Barbican doesn't help. The instance needs permission to access secrets
stored in Barbican. The main question of my e-mail is: how do you inject a
credential in an automated but secure way? I'd love to hear ideas - in the
meantime I'll study Ihar's link.

thanks,
Pino

On Fri, Sep 29, 2017 at 2:49 PM, Ihar Hrachyshka ihrachys@redhat.com
wrote:

What you describe (at least the use case) seems to resemble
https://review.openstack.org/#/c/456394/ This work never moved
anywhere since the spec was posted though. You may want to revive the
discussion in scope of the spec.

Ihar

On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia
giuseppe.decandia@gmail.com wrote:

Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates):
without
host certificates, when clients ssh to a host for the first time (or
after
the host has been re-installed), they have to hope that there's no man in
the middle and that the public key being presented actually belongs to
the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the possibility
of
Man-in-the-Middle attack: a Certificate Authority public key is
distributed
to clients (and written to their known
hosts file with a special syntax
and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its SSH
host
certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key
to my
CA server, sign it, and then write the certificate back into the host
(again
via VNC). I cannot ssh without risking a MITM attack. What about using
Nova
user-data? User-data is exposed via the metadata service. Metadata is
queried via http (reply transmitted in the clear, susceptible to
snooping),
and any compute node can query for any instance's meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I
know
cloud-init allows specifying SSH private keys (both for users and for SSH
service). I have not yet studied how such information is securely
injected
into an instance. I assume it should only be made available via
ConfigDrive
rather than metadata-service (again, that service transmits in the
clear).

What about providing SSH host certificates as a service in OpenStack?
Let's
keep out of scope issues around choosing and storing the CA keys, but
the CA
key is per project. What design supports setting up the SSH host
certificate
automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public key
out;
and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova
compute
instance boot code. Nova compute can securely query a CA service asking
for
a triplet (private key, public key, SSH certificate) for the specific
hostname. It can then inject the triplet using ConfigDrive. I believe
this
securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can think
of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:
unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 29, 2017 by Giuseppe_de_Candia (340 points)  
0 votes

Hi Ihar,

I have reviewed https://review.openstack.org/#/c/456394/ (Fetch hostkey
from port) and noted that:
1) that discussion is likely to stay among the Neutron developers only
(whereas I would like a wider audience, especially including Nova
developers)
2) that proposal does not consider SSH certificates, which are becoming a
standard approach in the industry (see [1] and [2])

Therefore, I would like to keep the discussion here.

thanks,
Pino

[1] -
https://code.facebook.com/posts/365787980419535/scalable-and-secure-access-with-ssh/
[2] -
https://medium.com/uber-security-privacy/introducing-the-uber-ssh-certificate-authority-4f840839c5cc

On Fri, Sep 29, 2017 at 3:05 PM, Giuseppe de Candia <
giuseppe.decandia@gmail.com> wrote:

Ihar, thanks for pointing that out - I'll definitely take a close look.

Jon, I'm not very familiar with Barbican, but I did assume the full
implementation would use Barbican to store private keys. However, in terms
of actually getting a private key (or SSH host cert) into a VM instance,
Barbican doesn't help. The instance needs permission to access secrets
stored in Barbican. The main question of my e-mail is: how do you inject a
credential in an automated but secure way? I'd love to hear ideas - in the
meantime I'll study Ihar's link.

thanks,
Pino

On Fri, Sep 29, 2017 at 2:49 PM, Ihar Hrachyshka ihrachys@redhat.com
wrote:

What you describe (at least the use case) seems to resemble
https://review.openstack.org/#/c/456394/ This work never moved
anywhere since the spec was posted though. You may want to revive the
discussion in scope of the spec.

Ihar

On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia
giuseppe.decandia@gmail.com wrote:

Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates):
without
host certificates, when clients ssh to a host for the first time (or
after
the host has been re-installed), they have to hope that there's no man
in
the middle and that the public key being presented actually belongs to
the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the
possibility of
Man-in-the-Middle attack: a Certificate Authority public key is
distributed
to clients (and written to their known
hosts file with a special syntax
and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its SSH
host
certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key
to my
CA server, sign it, and then write the certificate back into the host
(again
via VNC). I cannot ssh without risking a MITM attack. What about using
Nova
user-data? User-data is exposed via the metadata service. Metadata is
queried via http (reply transmitted in the clear, susceptible to
snooping),
and any compute node can query for any instance's meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I
know
cloud-init allows specifying SSH private keys (both for users and for
SSH
service). I have not yet studied how such information is securely
injected
into an instance. I assume it should only be made available via
ConfigDrive
rather than metadata-service (again, that service transmits in the
clear).

What about providing SSH host certificates as a service in OpenStack?
Let's
keep out of scope issues around choosing and storing the CA keys, but
the CA
key is per project. What design supports setting up the SSH host
certificate
automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public key
out;
and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova
compute
instance boot code. Nova compute can securely query a CA service asking
for
a triplet (private key, public key, SSH certificate) for the specific
hostname. It can then inject the triplet using ConfigDrive. I believe
this
securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can think
of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 29, 2017 by Giuseppe_de_Candia (340 points)  
0 votes

https://review.openstack.org/#/c/222293/


From: Giuseppe de Candia [giuseppe.decandia@gmail.com]
Sent: Friday, September 29, 2017 1:05 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Supporting SSH host certificates

Ihar, thanks for pointing that out - I'll definitely take a close look.

Jon, I'm not very familiar with Barbican, but I did assume the full implementation would use Barbican to store private keys. However, in terms of actually getting a private key (or SSH host cert) into a VM instance, Barbican doesn't help. The instance needs permission to access secrets stored in Barbican. The main question of my e-mail is: how do you inject a credential in an automated but secure way? I'd love to hear ideas - in the meantime I'll study Ihar's link.

thanks,
Pino

On Fri, Sep 29, 2017 at 2:49 PM, Ihar Hrachyshka ihrachys@redhat.com wrote:
What you describe (at least the use case) seems to resemble
https://review.openstack.org/#/c/456394/ This work never moved
anywhere since the spec was posted though. You may want to revive the
discussion in scope of the spec.

Ihar

On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia
giuseppe.decandia@gmail.com wrote:
Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates): without
host certificates, when clients ssh to a host for the first time (or after
the host has been re-installed), they have to hope that there's no man in
the middle and that the public key being presented actually belongs to the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the possibility of
Man-in-the-Middle attack: a Certificate Authority public key is distributed
to clients (and written to their known
hosts file with a special syntax and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its SSH host
certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key to my
CA server, sign it, and then write the certificate back into the host (again
via VNC). I cannot ssh without risking a MITM attack. What about using Nova
user-data? User-data is exposed via the metadata service. Metadata is
queried via http (reply transmitted in the clear, susceptible to snooping),
and any compute node can query for any instance's meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I know
cloud-init allows specifying SSH private keys (both for users and for SSH
service). I have not yet studied how such information is securely injected
into an instance. I assume it should only be made available via ConfigDrive
rather than metadata-service (again, that service transmits in the clear).

What about providing SSH host certificates as a service in OpenStack? Let's
keep out of scope issues around choosing and storing the CA keys, but the CA
key is per project. What design supports setting up the SSH host certificate
automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public key out;
and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova compute
instance boot code. Nova compute can securely query a CA service asking for
a triplet (private key, public key, SSH certificate) for the specific
hostname. It can then inject the triplet using ConfigDrive. I believe this
securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can think of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 30, 2017 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

Hey Pino,

mriedem pointed me to the vendordata code [1] which shows some fields are
passed (such as project ID) and that SSL is supported. So that's good.

The docs on vendordata suck. But I think it'll do what you're looking for.
Michael Still wrote up a helpful post titled "Nova vendordata deployment,
an excessively detailed guide"[2] and he's written a vendordata service
example[3] which even shows keystone integration.

At Oath, we have a system that provides a unique x509 certificate for each
host, including the ability to sign host SSH keys against an HSM. In our
case what we do is have Nova call the service, which generates and returns
a signed (and time limited) host bootstrap document, which is injected into
the instance. When the instance boots it calls our identity service and
provides its bootstrap document as a bearer certificate. The identity
service trusts this one-time document to attest the instance, and will then
provide an x509 certificate as well as sign the hosts SSH keys. After the
initial bootstrap the host will rotate its keys frequently, by providing
its last certificate in exchange for a new one. The service tracks all host
document and certificate IDs which have been exchanged until their expiry,
so that a cert cannot be re-used.

This infrastructure relies on Athenz [4] as the AuthNG system for all
principals (users, services, roles, domains, etc) as well as an internal
signatory service which signs x509 certificates and SSH host keys using an
HSM infrastructure.

Instead, you could write a vendordata service which, when called, would
generate an ssh host keypair, sign it, and return those files as encoded
data, which can be expanded into files in the correct locations on first
boot. I strongly suggest using not only using keystone auth, but that you
ensure all calls from vendordata to the microservice are encrypted with TLS
mutual auth.

-James

1:
https://github.com/openstack/nova/blob/master/nova/api/metadata/vendordata_dynamic.py#L77
2: https://www.stillhq.com/openstack/000022.html
3: https://github.com/mikalstill/vendordata
4: https://athenz.io

On Fri, Sep 29, 2017 at 5:17 PM, Fox, Kevin M Kevin.Fox@pnnl.gov wrote:

https://review.openstack.org/#/c/222293/


From: Giuseppe de Candia [giuseppe.decandia@gmail.com]
Sent: Friday, September 29, 2017 1:05 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Supporting SSH host certificates

Ihar, thanks for pointing that out - I'll definitely take a close look.

Jon, I'm not very familiar with Barbican, but I did assume the full
implementation would use Barbican to store private keys. However, in terms
of actually getting a private key (or SSH host cert) into a VM instance,
Barbican doesn't help. The instance needs permission to access secrets
stored in Barbican. The main question of my e-mail is: how do you inject a
credential in an automated but secure way? I'd love to hear ideas - in the
meantime I'll study Ihar's link.

thanks,
Pino

On Fri, Sep 29, 2017 at 2:49 PM, Ihar Hrachyshka ihrachys@redhat.com
wrote:

What you describe (at least the use case) seems to resemble
https://review.openstack.org/#/c/456394/ This work never moved
anywhere since the spec was posted though. You may want to revive the
discussion in scope of the spec.

Ihar

On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia
giuseppe.decandia@gmail.com wrote:

Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates):
without
host certificates, when clients ssh to a host for the first time (or
after
the host has been re-installed), they have to hope that there's no man
in
the middle and that the public key being presented actually belongs to
the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the
possibility of
Man-in-the-Middle attack: a Certificate Authority public key is
distributed
to clients (and written to their known
hosts file with a special syntax
and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its SSH
host
certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key
to my
CA server, sign it, and then write the certificate back into the host
(again
via VNC). I cannot ssh without risking a MITM attack. What about using
Nova
user-data? User-data is exposed via the metadata service. Metadata is
queried via http (reply transmitted in the clear, susceptible to
snooping),
and any compute node can query for any instance's meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I
know
cloud-init allows specifying SSH private keys (both for users and for
SSH
service). I have not yet studied how such information is securely
injected
into an instance. I assume it should only be made available via
ConfigDrive
rather than metadata-service (again, that service transmits in the
clear).

What about providing SSH host certificates as a service in OpenStack?
Let's
keep out of scope issues around choosing and storing the CA keys, but
the CA
key is per project. What design supports setting up the SSH host
certificate
automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public key
out;
and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova
compute
instance boot code. Nova compute can securely query a CA service asking
for
a triplet (private key, public key, SSH certificate) for the specific
hostname. It can then inject the triplet using ConfigDrive. I believe
this
securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can think
of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 5, 2017 by jpenick_at_gmail.com (860 points)  
0 votes

A related bug that hasn't seen any love for some time:
https://bugs.launchpad.net/nova/+bug/1613199

On 6 October 2017 at 07:47, James Penick jpenick@gmail.com wrote:

Hey Pino,

mriedem pointed me to the vendordata code [1] which shows some fields are
passed (such as project ID) and that SSL is supported. So that's good.

The docs on vendordata suck. But I think it'll do what you're looking for.
Michael Still wrote up a helpful post titled "Nova vendordata deployment,
an excessively detailed guide"[2] and he's written a vendordata service
example[3] which even shows keystone integration.

At Oath, we have a system that provides a unique x509 certificate for each
host, including the ability to sign host SSH keys against an HSM. In our
case what we do is have Nova call the service, which generates and returns
a signed (and time limited) host bootstrap document, which is injected into
the instance. When the instance boots it calls our identity service and
provides its bootstrap document as a bearer certificate. The identity
service trusts this one-time document to attest the instance, and will then
provide an x509 certificate as well as sign the hosts SSH keys. After the
initial bootstrap the host will rotate its keys frequently, by providing
its last certificate in exchange for a new one. The service tracks all host
document and certificate IDs which have been exchanged until their expiry,
so that a cert cannot be re-used.

This infrastructure relies on Athenz [4] as the AuthNG system for all
principals (users, services, roles, domains, etc) as well as an internal
signatory service which signs x509 certificates and SSH host keys using an
HSM infrastructure.

Instead, you could write a vendordata service which, when called, would
generate an ssh host keypair, sign it, and return those files as encoded
data, which can be expanded into files in the correct locations on first
boot. I strongly suggest using not only using keystone auth, but that you
ensure all calls from vendordata to the microservice are encrypted with TLS
mutual auth.

-James

1: https://github.com/openstack/nova/blob/master/nova/api/
metadata/vendordata_dynamic.py#L77
2: https://www.stillhq.com/openstack/000022.html
3: https://github.com/mikalstill/vendordata
4: https://athenz.io

On Fri, Sep 29, 2017 at 5:17 PM, Fox, Kevin M Kevin.Fox@pnnl.gov wrote:

https://review.openstack.org/#/c/222293/

From: Giuseppe de Candia [giuseppe.decandia@gmail.com]
Sent: Friday, September 29, 2017 1:05 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Supporting SSH host certificates

Ihar, thanks for pointing that out - I'll definitely take a close look.

Jon, I'm not very familiar with Barbican, but I did assume the full
implementation would use Barbican to store private keys. However, in terms
of actually getting a private key (or SSH host cert) into a VM instance,
Barbican doesn't help. The instance needs permission to access secrets
stored in Barbican. The main question of my e-mail is: how do you inject a
credential in an automated but secure way? I'd love to hear ideas - in the
meantime I'll study Ihar's link.

thanks,
Pino

On Fri, Sep 29, 2017 at 2:49 PM, Ihar Hrachyshka ihrachys@redhat.com
wrote:

What you describe (at least the use case) seems to resemble
https://review.openstack.org/#/c/456394/ This work never moved
anywhere since the spec was posted though. You may want to revive the
discussion in scope of the spec.

Ihar

On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia
giuseppe.decandia@gmail.com wrote:

Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH
host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates):
without
host certificates, when clients ssh to a host for the first time (or
after
the host has been re-installed), they have to hope that there's no man
in
the middle and that the public key being presented actually belongs to
the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the
possibility of
Man-in-the-Middle attack: a Certificate Authority public key is
distributed
to clients (and written to their known
hosts file with a special
syntax and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its
SSH host
certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key
to my
CA server, sign it, and then write the certificate back into the host
(again
via VNC). I cannot ssh without risking a MITM attack. What about using
Nova
user-data? User-data is exposed via the metadata service. Metadata is
queried via http (reply transmitted in the clear, susceptible to
snooping),
and any compute node can query for any instance's meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I
know
cloud-init allows specifying SSH private keys (both for users and for
SSH
service). I have not yet studied how such information is securely
injected
into an instance. I assume it should only be made available via
ConfigDrive
rather than metadata-service (again, that service transmits in the
clear).

What about providing SSH host certificates as a service in OpenStack?
Let's
keep out of scope issues around choosing and storing the CA keys, but
the CA
key is per project. What design supports setting up the SSH host
certificate
automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public
key out;
and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova
compute
instance boot code. Nova compute can securely query a CA service
asking for
a triplet (private key, public key, SSH certificate) for the specific
hostname. It can then inject the triplet using ConfigDrive. I believe
this
securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can
think of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Cheers,
~Blairo


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 6, 2017 by Blair_Bethwaite (4,080 points)   1 3 5
0 votes

A long time ago, a few Canonical employees (Scott Moser was one of them,
forget who else was doing it, maybe Dave Walker and/or Dustin Kirkland)
worked out a scheme for general usage that doesn't require extra plumbing:

  • Client generates a small SSH host key locally and pushes it into
    user data in a way which causes the image to boot and install this
    key from user_data as the host SSH key.
  • Client SSH's in with the strict requirement that the host key be the
    one they just generated and pushed into the instance.
  • Client now triggers new host key generation, and copies new public
    key into client known_hosts.

With this system you don't have to scrape console logs for SSH keys or
build your system on hope.

Excerpts from Giuseppe de Candia's message of 2017-09-29 14:21:06 -0500:

Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates): without
host certificates, when clients ssh to a host for the first time (or after
the host has been re-installed), they have to hope that there's no man in
the middle and that the public key being presented actually belongs to the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the possibility of
Man-in-the-Middle attack: a Certificate Authority public key is distributed
to clients (and written to their known
hosts file with a special syntax and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its SSH
host certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key to
my CA server, sign it, and then write the certificate back into the host
(again via VNC). I cannot ssh without risking a MITM attack. What about
using Nova user-data? User-data is exposed via the metadata service.
Metadata is queried via http (reply transmitted in the clear, susceptible
to snooping), and any compute node can query for any instance's
meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I know
cloud-init allows specifying SSH private keys (both for users and for SSH
service). I have not yet studied how such information is securely injected
into an instance. I assume it should only be made available via ConfigDrive
rather than metadata-service (again, that service transmits in the clear).

What about providing SSH host certificates as a service in OpenStack? Let's
keep out of scope issues around choosing and storing the CA keys, but the
CA key is per project. What design supports setting up the SSH host
certificate automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public key
out; and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova
compute instance boot code. Nova compute can securely query a CA service
asking for a triplet (private key, public key, SSH certificate) for the
specific hostname. It can then inject the triplet using ConfigDrive. I
believe this securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can think of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 6, 2017 by Clint_Byrum (40,940 points)   4 5 9
0 votes

James, thanks for all the information! I will study, try some of this out
and get back to the thread with any more questions or findings.

One topic that may be worth discussing now, is that if we do pass ssh host
keys (including private) via vendordata, then there needs to be a way to
signal to Nova either:
- that specific data is private and should not be exposed via metadata API
- or for each vendordata choose what cloud-init data source it should be
offered on.

The goal would be to make sure that the private key is only offered to the
instance via config drive.

Does that make sense?

thanks,
Pino

On Thu, Oct 5, 2017 at 3:47 PM, James Penick jpenick@gmail.com wrote:

Hey Pino,

mriedem pointed me to the vendordata code [1] which shows some fields are
passed (such as project ID) and that SSL is supported. So that's good.

The docs on vendordata suck. But I think it'll do what you're looking for.
Michael Still wrote up a helpful post titled "Nova vendordata deployment,
an excessively detailed guide"[2] and he's written a vendordata service
example[3] which even shows keystone integration.

At Oath, we have a system that provides a unique x509 certificate for each
host, including the ability to sign host SSH keys against an HSM. In our
case what we do is have Nova call the service, which generates and returns
a signed (and time limited) host bootstrap document, which is injected into
the instance. When the instance boots it calls our identity service and
provides its bootstrap document as a bearer certificate. The identity
service trusts this one-time document to attest the instance, and will then
provide an x509 certificate as well as sign the hosts SSH keys. After the
initial bootstrap the host will rotate its keys frequently, by providing
its last certificate in exchange for a new one. The service tracks all host
document and certificate IDs which have been exchanged until their expiry,
so that a cert cannot be re-used.

This infrastructure relies on Athenz [4] as the AuthNG system for all
principals (users, services, roles, domains, etc) as well as an internal
signatory service which signs x509 certificates and SSH host keys using an
HSM infrastructure.

Instead, you could write a vendordata service which, when called, would
generate an ssh host keypair, sign it, and return those files as encoded
data, which can be expanded into files in the correct locations on first
boot. I strongly suggest using not only using keystone auth, but that you
ensure all calls from vendordata to the microservice are encrypted with TLS
mutual auth.

-James

1: https://github.com/openstack/nova/blob/master/nova/api/
metadata/vendordata_dynamic.py#L77
2: https://www.stillhq.com/openstack/000022.html
3: https://github.com/mikalstill/vendordata
4: https://athenz.io

On Fri, Sep 29, 2017 at 5:17 PM, Fox, Kevin M Kevin.Fox@pnnl.gov wrote:

https://review.openstack.org/#/c/222293/

From: Giuseppe de Candia [giuseppe.decandia@gmail.com]
Sent: Friday, September 29, 2017 1:05 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] Supporting SSH host certificates

Ihar, thanks for pointing that out - I'll definitely take a close look.

Jon, I'm not very familiar with Barbican, but I did assume the full
implementation would use Barbican to store private keys. However, in terms
of actually getting a private key (or SSH host cert) into a VM instance,
Barbican doesn't help. The instance needs permission to access secrets
stored in Barbican. The main question of my e-mail is: how do you inject a
credential in an automated but secure way? I'd love to hear ideas - in the
meantime I'll study Ihar's link.

thanks,
Pino

On Fri, Sep 29, 2017 at 2:49 PM, Ihar Hrachyshka ihrachys@redhat.com
wrote:

What you describe (at least the use case) seems to resemble
https://review.openstack.org/#/c/456394/ This work never moved
anywhere since the spec was posted though. You may want to revive the
discussion in scope of the spec.

Ihar

On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia
giuseppe.decandia@gmail.com wrote:

Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH
host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates):
without
host certificates, when clients ssh to a host for the first time (or
after
the host has been re-installed), they have to hope that there's no man
in
the middle and that the public key being presented actually belongs to
the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the
possibility of
Man-in-the-Middle attack: a Certificate Authority public key is
distributed
to clients (and written to their known
hosts file with a special
syntax and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its
SSH host
certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key
to my
CA server, sign it, and then write the certificate back into the host
(again
via VNC). I cannot ssh without risking a MITM attack. What about using
Nova
user-data? User-data is exposed via the metadata service. Metadata is
queried via http (reply transmitted in the clear, susceptible to
snooping),
and any compute node can query for any instance's meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I
know
cloud-init allows specifying SSH private keys (both for users and for
SSH
service). I have not yet studied how such information is securely
injected
into an instance. I assume it should only be made available via
ConfigDrive
rather than metadata-service (again, that service transmits in the
clear).

What about providing SSH host certificates as a service in OpenStack?
Let's
keep out of scope issues around choosing and storing the CA keys, but
the CA
key is per project. What design supports setting up the SSH host
certificate
automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public
key out;
and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova
compute
instance boot code. Nova compute can securely query a CA service
asking for
a triplet (private key, public key, SSH certificate) for the specific
hostname. It can then inject the triplet using ConfigDrive. I believe
this
securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can
think of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.op
enstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscrib
e
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 6, 2017 by Giuseppe_de_Candia (340 points)  
0 votes

Hi Clint,

Isn't user-data by definition available via the Metadata API, which isn't
considered secure:
https://wiki.openstack.org/wiki/OSSN/OSSN-0074

Or is there a way to specify that certain user-data should only be
available via config-drive (and not metadata api)?

Otherwise, the only difference I see compared to using Meta-data is that
the process you describe is driven by the user vs. automated.

Regarding the extra plumbing, I'm not trying to avoid it. I'm thinking to
eventually tie this all into Keystone. For example, a project should have
Host CA and User CA keys. Let's assume OpenStack manages these for now,
later we can consider OpenStack simply proxying signature requests and
vouching that a public key does actually belong to a specific instance (and
host-name) or Keystone user. So what I think should happen is when a
Project is enabled for SSHaaS support, any VM instance automatically gets
host certificate, authorized principal files based on Keystone roles for
the project, and users can call an API (or Dashboard form) to get a public
key signed (and assigned appropriate SSH principals).

cheers,
Pino

On Fri, Oct 6, 2017 at 2:37 AM, Clint Byrum clint@fewbar.com wrote:

A long time ago, a few Canonical employees (Scott Moser was one of them,
forget who else was doing it, maybe Dave Walker and/or Dustin Kirkland)
worked out a scheme for general usage that doesn't require extra plumbing:

  • Client generates a small SSH host key locally and pushes it into
    user data in a way which causes the image to boot and install this
    key from user_data as the host SSH key.
  • Client SSH's in with the strict requirement that the host key be the
    one they just generated and pushed into the instance.
  • Client now triggers new host key generation, and copies new public
    key into client known_hosts.

With this system you don't have to scrape console logs for SSH keys or
build your system on hope.

Excerpts from Giuseppe de Candia's message of 2017-09-29 14:21:06 -0500:

Hi Folks,

My intent in this e-mail is to solicit advice for how to inject SSH host
certificates into VM instances, with minimal or no burden on users.

Background (skip if you're already familiar with SSH certificates):
without
host certificates, when clients ssh to a host for the first time (or
after
the host has been re-installed), they have to hope that there's no man in
the middle and that the public key being presented actually belongs to
the
host they're trying to reach. The host's public key is stored in the
client's knownhosts file. SSH host certicates eliminate the possibility
of
Man-in-the-Middle attack: a Certificate Authority public key is
distributed
to clients (and written to their known
hosts file with a special syntax
and
options); the host public key is signed by the CA, generating an SSH
certificate that contains the hostname and validity period (among other
things). When negotiating the ssh connection, the host presents its SSH
host certificate and the client verifies that it was signed by the CA.

How to support SSH host certificates in OpenStack?

First, let's consider doing it by hand, instance by instance. The only
solution I can think of is to VNC to the instance, copy the public key to
my CA server, sign it, and then write the certificate back into the host
(again via VNC). I cannot ssh without risking a MITM attack. What about
using Nova user-data? User-data is exposed via the metadata service.
Metadata is queried via http (reply transmitted in the clear, susceptible
to snooping), and any compute node can query for any instance's
meta-data/user-data.

At this point I have to admit I'm ignorant of details of cloud-init. I
know
cloud-init allows specifying SSH private keys (both for users and for SSH
service). I have not yet studied how such information is securely
injected
into an instance. I assume it should only be made available via
ConfigDrive
rather than metadata-service (again, that service transmits in the
clear).

What about providing SSH host certificates as a service in OpenStack?
Let's
keep out of scope issues around choosing and storing the CA keys, but the
CA key is per project. What design supports setting up the SSH host
certificate automatically for every VM instance?

I have looked at Vendor Data and I don't see a way to use that, mainly
because 1) it doesn't take parameters, so you can't pass the public key
out; and 2) it's queried over http, not https.

Just as a feasibility argument, one solution would be to modify Nova
compute instance boot code. Nova compute can securely query a CA service
asking for a triplet (private key, public key, SSH certificate) for the
specific hostname. It can then inject the triplet using ConfigDrive. I
believe this securely gets the private key into the instance.

I cannot figure out how to get the equivalent functionality without
modifying Nova compute and the boot process. Every solution I can think
of
risks either exposing the private key or vulnerability to a MITM attack
during the signing process.

Your help is appreciated.

--Pino


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 6, 2017 by Giuseppe_de_Candia (340 points)  
...