settingsLogin | Registersettings

[openstack-dev] [Keystone] [Horizon] Pagination support for Identity dashboard entities

0 votes

Hello, Keystone folks!

I've just discovered an unfortunate fact that Horizon pagination for
Tenants/Projects list that worked with Keystone v2 doesn't work with
Keysone v3 anymore - its API call simply lacks the 'marker' and 'limit'
parameters [1] that Horizon is relying upon. Meanwhile having looked
through [2] and [3] I'm a bit confused: while Keystone v3 API states it
supports [2] pagination for every kind of entities (by using 'page' and
'per_page' parameters), the related blueprint [3] is not yet approved,
meaning that most likely the API implementation did not make it into
existing Keystone codebase. So I wonder whether there are some plans to
implement pagination for Keystone API calls that list its entities?

P.S. I'm aware of SearchLight project that tries to help Horizon with other
OpenStack APIs (part of its mission), what I'm trying to understand here is
are we going to have some fallback pagination mechanism for Horizon's
Identity in a short-term/mid-term perspective.

[1] http://developer.openstack.org/api-ref-identity-admin-v2.html
[2] http://developer.openstack.org/api-ref-identity-v3.html
[3] https://blueprints.launchpad.net/keystone/+spec/pagination


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Aug 14, 2015 in openstack-dev by Timur_Sufiev (3,480 points)   5 6

24 Responses

0 votes

As a quick note the api-ref you are linking to has some gaps/has not been kept in sync with the official api specifications.

The official API specification is located at http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3 sections at the top) and there is a known open bug to work with the docs team to get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity backend where pagination just does not work (or works completely unreliably) such as utilizing the ldap driver. Either a cursor must be maintained (problematic in REST) or the results could be wildly different every single request meaning each page is not really guaranteed to be the "next page" it could be the same/show inconsistent results. The second issue is that the pagination is not a good UX even where it works - the simple question is: if you can filter the results how many pages deep do you go before refining the query; think of your use of search engines.

In light of these issues Keystone has opted for a filter / limit (config). If the results exceed the limit there is a truncation that occurs and it is recommended extra filtering be applied to reduce the total number of results.

This discussion has gone around a few times, pagination in keystone is not currently on the roadmap. In addition to the above doc bug, we should work to better socialize this filter-over-paginate methodology.

--Morgan

Sent via mobile

On Aug 14, 2015, at 05:46, Timur Sufiev tsufiev@mirantis.com wrote:

Hello, Keystone folks!

I've just discovered an unfortunate fact that Horizon pagination for Tenants/Projects list that worked with Keystone v2 doesn't work with Keysone v3 anymore - its API call simply lacks the 'marker' and 'limit' parameters [1] that Horizon is relying upon. Meanwhile having looked through [2] and [3] I'm a bit confused: while Keystone v3 API states it supports [2] pagination for every kind of entities (by using 'page' and 'per_page' parameters), the related blueprint [3] is not yet approved, meaning that most likely the API implementation did not make it into existing Keystone codebase. So I wonder whether there are some plans to implement pagination for Keystone API calls that list its entities?

P.S. I'm aware of SearchLight project that tries to help Horizon with other OpenStack APIs (part of its mission), what I'm trying to understand here is are we going to have some fallback pagination mechanism for Horizon's Identity in a short-term/mid-term perspective.

[1] http://developer.openstack.org/api-ref-identity-admin-v2.html
[2] http://developer.openstack.org/api-ref-identity-v3.html
[3] https://blueprints.launchpad.net/keystone/+spec/pagination


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Morgan_Fainberg (17,320 points)   2 6 9
0 votes

On 08/14/2015 09:14 AM, Morgan Fainberg wrote:
As a quick note the api-ref you are linking to has some gaps/has not
been kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3 sections
at the top) and there is a known open bug to work with the docs team to
get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly
different every single request meaning each page is not really
guaranteed to be the "next page" it could be the same/show inconsistent
results. The second issue is that the pagination is not a good UX even
where it works - the simple question is: if you can filter the results
how many pages deep do you go before refining the query; think of your
use of search engines.

In light of these issues Keystone has opted for a filter / limit
(config). If the results exceed the limit there is a truncation that
occurs and it is recommended extra filtering be applied to reduce the
total number of results.

This discussion has gone around a few times, pagination in keystone is
not currently on the roadmap. In addition to the above doc bug, we
should work to better socialize this filter-over-paginate methodology.

I understand all the things you write above about the problems that
Keystone's underlying architecture (driver-based, not always able to do
pagination in the individual drivers). However, it really does mean that
Keystone is the only project in OpenStack that behaves this way. All
other services provide limit/marker paginations, AFAIK, which is
efficient and, while not the same UX as a filtering methodology, is
entirely compatible and complementary to filtering.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Jay_Pipes (59,760 points)   3 11 14
0 votes

For the identity (users and groups) backend as long as we support LDAP (and as side note federated users never show up in this list anyway) and with the drive towards pushing all user management out of keystone itself to ldap or other tools that do it better, I don't see pagination as something we should be providing. Providing an inconsistent user experience based on leaking underlying implementation details is something I am very against. This stance ensures that horizon and other tools like it will not need to know underlying implementation details to provide a consistent user experience. Unfortunately here we do need to cater to the lowest common denominator and filtering/searching/limiting is the clear common mechanism

With regards to resources (projects, domains, etc) since we no longer support using LDAP (deprecated with removal in mitaka) I have less strong feelings towards and wouldn't block efforts to implement if it is not already available (if not available this is likely a mitaka goal).

--Morgan

Sent via mobile

On Aug 14, 2015, at 07:39, Jay Pipes jaypipes@gmail.com wrote:

On 08/14/2015 09:14 AM, Morgan Fainberg wrote:
As a quick note the api-ref you are linking to has some gaps/has not
been kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3 sections
at the top) and there is a known open bug to work with the docs team to
get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly
different every single request meaning each page is not really
guaranteed to be the "next page" it could be the same/show inconsistent
results. The second issue is that the pagination is not a good UX even
where it works - the simple question is: if you can filter the results
how many pages deep do you go before refining the query; think of your
use of search engines.

In light of these issues Keystone has opted for a filter / limit
(config). If the results exceed the limit there is a truncation that
occurs and it is recommended extra filtering be applied to reduce the
total number of results.

This discussion has gone around a few times, pagination in keystone is
not currently on the roadmap. In addition to the above doc bug, we
should work to better socialize this filter-over-paginate methodology.

I understand all the things you write above about the problems that Keystone's underlying architecture (driver-based, not always able to do pagination in the individual drivers). However, it really does mean that Keystone is the only project in OpenStack that behaves this way. All other services provide limit/marker paginations, AFAIK, which is efficient and, while not the same UX as a filtering methodology, is entirely compatible and complementary to filtering.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Morgan_Fainberg (17,320 points)   2 6 9
0 votes

Morgan,

Your reasoning is perfectly fine from the Keystone point of view. Yet I
believe this approach is harmful for both Horizon and the whole OpenStack
ecosystem.

It is harmful for the ecosystem, because it breaks API uniformity in one of
the few areas where this uniformity could be achieved. Imagine if Nova or
Cinder start saying the same thing: "we have too much drivers/backends to
provide the uniform interface for all of them, let's delegate the choice of
handling them differently to our consumers". It'll propagate the knowledge
of different backends throughout the stack and it's obviously not good.

Not having pagination on Identity->Users page means that even with
filtering being fully supported there will be problems. At least the first
time the Users page with all the Users piped from production-grade LDAP
through Keystone is shown in Horizon, it takes a lot time to render them
all (before an unhappy admin had any chance to narrow the list), which
eventually may result in connection being dropped by some HA balancer. We
did these kinds of tests, the results weren't reassuring. Well, I might
miss some of new Horizon angularization steps, so please regard this
paragraph as my personal opinion - I don't think Horizon could be lighting
fast on its own (i.e. without additional services) with a lot of data
without pagination.

On Fri, Aug 14, 2015 at 6:03 PM Morgan Fainberg morgan.fainberg@gmail.com
wrote:

For the identity (users and groups) backend as long as we support LDAP
(and as side note federated users never show up in this list anyway) and
with the drive towards pushing all user management out of keystone itself
to ldap or other tools that do it better, I don't see pagination as
something we should be providing. Providing an inconsistent user experience
based on leaking underlying implementation details is something I am very
against. This stance ensures that horizon and other tools like it will not
need to know underlying implementation details to provide a consistent user
experience. Unfortunately here we do need to cater to the lowest common
denominator and filtering/searching/limiting is the clear common mechanism

With regards to resources (projects, domains, etc) since we no longer
support using LDAP (deprecated with removal in mitaka) I have less strong
feelings towards and wouldn't block efforts to implement if it is not
already available (if not available this is likely a mitaka goal).

--Morgan

Sent via mobile

On Aug 14, 2015, at 07:39, Jay Pipes jaypipes@gmail.com wrote:

On 08/14/2015 09:14 AM, Morgan Fainberg wrote:
As a quick note the api-ref you are linking to has some gaps/has not
been kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3
sections
at the top) and there is a known open bug to work with the docs team to
get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly
different every single request meaning each page is not really
guaranteed to be the "next page" it could be the same/show inconsistent
results. The second issue is that the pagination is not a good UX even
where it works - the simple question is: if you can filter the results
how many pages deep do you go before refining the query; think of your
use of search engines.

In light of these issues Keystone has opted for a filter / limit
(config). If the results exceed the limit there is a truncation that
occurs and it is recommended extra filtering be applied to reduce the
total number of results.

This discussion has gone around a few times, pagination in keystone is
not currently on the roadmap. In addition to the above doc bug, we
should work to better socialize this filter-over-paginate methodology.

I understand all the things you write above about the problems that
Keystone's underlying architecture (driver-based, not always able to do
pagination in the individual drivers). However, it really does mean that
Keystone is the only project in OpenStack that behaves this way. All other
services provide limit/marker paginations, AFAIK, which is efficient and,
while not the same UX as a filtering methodology, is entirely compatible
and complementary to filtering.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Timur_Sufiev (3,480 points)   5 6
0 votes

Surely ldap supports some form of pagination/searching natively. If any storage system of users needs to scale up to large numbers of users, its ldap...

Thanks,
Kevin


From: Timur Sufiev [tsufiev@mirantis.com]
Sent: Friday, August 14, 2015 9:20 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Keystone] [Horizon] Pagination support for Identity dashboard entities

Morgan,

Your reasoning is perfectly fine from the Keystone point of view. Yet I believe this approach is harmful for both Horizon and the whole OpenStack ecosystem.

It is harmful for the ecosystem, because it breaks API uniformity in one of the few areas where this uniformity could be achieved. Imagine if Nova or Cinder start saying the same thing: "we have too much drivers/backends to provide the uniform interface for all of them, let's delegate the choice of handling them differently to our consumers". It'll propagate the knowledge of different backends throughout the stack and it's obviously not good.

Not having pagination on Identity->Users page means that even with filtering being fully supported there will be problems. At least the first time the Users page with all the Users piped from production-grade LDAP through Keystone is shown in Horizon, it takes a lot time to render them all (before an unhappy admin had any chance to narrow the list), which eventually may result in connection being dropped by some HA balancer. We did these kinds of tests, the results weren't reassuring. Well, I might miss some of new Horizon angularization steps, so please regard this paragraph as my personal opinion - I don't think Horizon could be lighting fast on its own (i.e. without additional services) with a lot of data without pagination.

On Fri, Aug 14, 2015 at 6:03 PM Morgan Fainberg morgan.fainberg@gmail.com wrote:
For the identity (users and groups) backend as long as we support LDAP (and as side note federated users never show up in this list anyway) and with the drive towards pushing all user management out of keystone itself to ldap or other tools that do it better, I don't see pagination as something we should be providing. Providing an inconsistent user experience based on leaking underlying implementation details is something I am very against. This stance ensures that horizon and other tools like it will not need to know underlying implementation details to provide a consistent user experience. Unfortunately here we do need to cater to the lowest common denominator and filtering/searching/limiting is the clear common mechanism

With regards to resources (projects, domains, etc) since we no longer support using LDAP (deprecated with removal in mitaka) I have less strong feelings towards and wouldn't block efforts to implement if it is not already available (if not available this is likely a mitaka goal).

--Morgan

Sent via mobile

On Aug 14, 2015, at 07:39, Jay Pipes jaypipes@gmail.com wrote:

On 08/14/2015 09:14 AM, Morgan Fainberg wrote:
As a quick note the api-ref you are linking to has some gaps/has not
been kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3 sections
at the top) and there is a known open bug to work with the docs team to
get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly
different every single request meaning each page is not really
guaranteed to be the "next page" it could be the same/show inconsistent
results. The second issue is that the pagination is not a good UX even
where it works - the simple question is: if you can filter the results
how many pages deep do you go before refining the query; think of your
use of search engines.

In light of these issues Keystone has opted for a filter / limit
(config). If the results exceed the limit there is a truncation that
occurs and it is recommended extra filtering be applied to reduce the
total number of results.

This discussion has gone around a few times, pagination in keystone is
not currently on the roadmap. In addition to the above doc bug, we
should work to better socialize this filter-over-paginate methodology.

I understand all the things you write above about the problems that Keystone's underlying architecture (driver-based, not always able to do pagination in the individual drivers). However, it really does mean that Keystone is the only project in OpenStack that behaves this way. All other services provide limit/marker paginations, AFAIK, which is efficient and, while not the same UX as a filtering methodology, is entirely compatible and complementary to filtering.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Fox,_Kevin_M (29,360 points)   1 3 4
0 votes

I don't have a horse in the "What should keystone support" race. I do,
however, need to point out that any UX argument made about how a UI should
work should, at this point, ask the OpenStack UX program for help! Thus
I've changed the topic of this email to make sure Piet and the UX teams get
a chance to respond.

It feels like we have four UX assumptions, which I feel should be converted
into questions:
1- Do users want to page through search results?
2- Do users want to page through filter results? (do they use filter
results?)
3- If they want to page, do they want to be able to go back a page and/or
know their current page?
4- How much do users care about small data inconsistencies (i.e. ordering
of record sets changing from page-to-page).

Also, from personal experience, it is impossible to make a "search"
implementation that users, especially enterprise users, trust. I personally
blame Sharepoint for that.

Michael

On Fri, Aug 14, 2015 at 6:17 AM Morgan Fainberg morgan.fainberg@gmail.com
wrote:

As a quick note the api-ref you are linking to has some gaps/has not been
kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3 sections
at the top) and there is a known open bug to work with the docs team to get
this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly different
every single request meaning each page is not really guaranteed to be the
"next page" it could be the same/show inconsistent results. The second
issue is that the pagination is not a good UX even where it works - the
simple question is: if you can filter the results how many pages deep do
you go before refining the query; think of your use of search engines.

In light of these issues Keystone has opted for a filter / limit (config).
If the results exceed the limit there is a truncation that occurs and it is
recommended extra filtering be applied to reduce the total number of
results.

This discussion has gone around a few times, pagination in keystone is not
currently on the roadmap. In addition to the above doc bug, we should work
to better socialize this filter-over-paginate methodology.

--Morgan

Sent via mobile

On Aug 14, 2015, at 05:46, Timur Sufiev tsufiev@mirantis.com wrote:

Hello, Keystone folks!

I've just discovered an unfortunate fact that Horizon pagination for
Tenants/Projects list that worked with Keystone v2 doesn't work with
Keysone v3 anymore - its API call simply lacks the 'marker' and 'limit'
parameters [1] that Horizon is relying upon. Meanwhile having looked
through [2] and [3] I'm a bit confused: while Keystone v3 API states it
supports [2] pagination for every kind of entities (by using 'page' and
'per_page' parameters), the related blueprint [3] is not yet approved,
meaning that most likely the API implementation did not make it into
existing Keystone codebase. So I wonder whether there are some plans to
implement pagination for Keystone API calls that list its entities?

P.S. I'm aware of SearchLight project that tries to help Horizon with
other OpenStack APIs (part of its mission), what I'm trying to understand
here is are we going to have some fallback pagination mechanism for
Horizon's Identity in a short-term/mid-term perspective.

[1] http://developer.openstack.org/api-ref-identity-admin-v2.html
[2] http://developer.openstack.org/api-ref-identity-v3.html
[3] https://blueprints.launchpad.net/keystone/+spec/pagination


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Michael_Krotscheck (4,420 points)   3 3
0 votes

I understand the reasoning, but there are use cases for indexing (re:
searchlight) and auditing that are completely unsupported in keystone v3.
As from keystone, I have no way to exhaustively list who has accounts in my
cloud using OpenStack APIs. That seems like a hole that should be filled.

Not to mention API consistency, which others have already brought up.

David

On Fri, Aug 14, 2015 at 9:02 AM, Morgan Fainberg morgan.fainberg@gmail.com
wrote:

For the identity (users and groups) backend as long as we support LDAP
(and as side note federated users never show up in this list anyway) and
with the drive towards pushing all user management out of keystone itself
to ldap or other tools that do it better, I don't see pagination as
something we should be providing. Providing an inconsistent user experience
based on leaking underlying implementation details is something I am very
against. This stance ensures that horizon and other tools like it will not
need to know underlying implementation details to provide a consistent user
experience. Unfortunately here we do need to cater to the lowest common
denominator and filtering/searching/limiting is the clear common mechanism

With regards to resources (projects, domains, etc) since we no longer
support using LDAP (deprecated with removal in mitaka) I have less strong
feelings towards and wouldn't block efforts to implement if it is not
already available (if not available this is likely a mitaka goal).

--Morgan

Sent via mobile

On Aug 14, 2015, at 07:39, Jay Pipes jaypipes@gmail.com wrote:

On 08/14/2015 09:14 AM, Morgan Fainberg wrote:
As a quick note the api-ref you are linking to has some gaps/has not
been kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3
sections
at the top) and there is a known open bug to work with the docs team to
get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly
different every single request meaning each page is not really
guaranteed to be the "next page" it could be the same/show inconsistent
results. The second issue is that the pagination is not a good UX even
where it works - the simple question is: if you can filter the results
how many pages deep do you go before refining the query; think of your
use of search engines.

In light of these issues Keystone has opted for a filter / limit
(config). If the results exceed the limit there is a truncation that
occurs and it is recommended extra filtering be applied to reduce the
total number of results.

This discussion has gone around a few times, pagination in keystone is
not currently on the roadmap. In addition to the above doc bug, we
should work to better socialize this filter-over-paginate methodology.

I understand all the things you write above about the problems that
Keystone's underlying architecture (driver-based, not always able to do
pagination in the individual drivers). However, it really does mean that
Keystone is the only project in OpenStack that behaves this way. All other
services provide limit/marker paginations, AFAIK, which is efficient and,
while not the same UX as a filtering methodology, is entirely compatible
and complementary to filtering.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by David_Lyle (2,780 points)   2 3
0 votes

Pagination in ldap requires holding a cursor open. You would have to map the requests to the same cursor each time. It costs memory and holds a client connected to the ldap server. In a REST api it is a bad idea. With regard to searching it can be done, but each query can be a different set of objects (order is not guaranteed). It isn't straight forward.

To put is bluntly, we are working to push user management to the tools that are better at this than keystone. The LDAP servers or AD have far better tools than the keystone API. And federated users are managed externally as well. The SQL table to manage users is not a good solution and we are making strides to eliminate the needs for even service users to exist here.

The question about roles and grants can be queried and appropriately paginated/limited/searched (again same statement about resource for project/domain where if it doesn't exist i wouldn't block it but it is likely a mitaka target).

--Morgan

Sent via mobile

On Aug 14, 2015, at 09:42, Fox, Kevin M Kevin.Fox@pnnl.gov wrote:

Surely ldap supports some form of pagination/searching natively. If any storage system of users needs to scale up to large numbers of users, its ldap...

Thanks,
Kevin
From: Timur Sufiev [tsufiev@mirantis.com]
Sent: Friday, August 14, 2015 9:20 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Keystone] [Horizon] Pagination support for Identity dashboard entities

Morgan,

Your reasoning is perfectly fine from the Keystone point of view. Yet I believe this approach is harmful for both Horizon and the whole OpenStack ecosystem.

It is harmful for the ecosystem, because it breaks API uniformity in one of the few areas where this uniformity could be achieved. Imagine if Nova or Cinder start saying the same thing: "we have too much drivers/backends to provide the uniform interface for all of them, let's delegate the choice of handling them differently to our consumers". It'll propagate the knowledge of different backends throughout the stack and it's obviously not good.

Not having pagination on Identity->Users page means that even with filtering being fully supported there will be problems. At least the first time the Users page with all the Users piped from production-grade LDAP through Keystone is shown in Horizon, it takes a lot time to render them all (before an unhappy admin had any chance to narrow the list), which eventually may result in connection being dropped by some HA balancer. We did these kinds of tests, the results weren't reassuring. Well, I might miss some of new Horizon angularization steps, so please regard this paragraph as my personal opinion - I don't think Horizon could be lighting fast on its own (i.e. without additional services) with a lot of data without pagination.

On Fri, Aug 14, 2015 at 6:03 PM Morgan Fainberg morgan.fainberg@gmail.com wrote:
For the identity (users and groups) backend as long as we support LDAP (and as side note federated users never show up in this list anyway) and with the drive towards pushing all user management out of keystone itself to ldap or other tools that do it better, I don't see pagination as something we should be providing. Providing an inconsistent user experience based on leaking underlying implementation details is something I am very against. This stance ensures that horizon and other tools like it will not need to know underlying implementation details to provide a consistent user experience. Unfortunately here we do need to cater to the lowest common denominator and filtering/searching/limiting is the clear common mechanism

With regards to resources (projects, domains, etc) since we no longer support using LDAP (deprecated with removal in mitaka) I have less strong feelings towards and wouldn't block efforts to implement if it is not already available (if not available this is likely a mitaka goal).

--Morgan

Sent via mobile

On Aug 14, 2015, at 07:39, Jay Pipes jaypipes@gmail.com wrote:

On 08/14/2015 09:14 AM, Morgan Fainberg wrote:
As a quick note the api-ref you are linking to has some gaps/has not
been kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3 sections
at the top) and there is a known open bug to work with the docs team to
get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly
different every single request meaning each page is not really
guaranteed to be the "next page" it could be the same/show inconsistent
results. The second issue is that the pagination is not a good UX even
where it works - the simple question is: if you can filter the results
how many pages deep do you go before refining the query; think of your
use of search engines.

In light of these issues Keystone has opted for a filter / limit
(config). If the results exceed the limit there is a truncation that
occurs and it is recommended extra filtering be applied to reduce the
total number of results.

This discussion has gone around a few times, pagination in keystone is
not currently on the roadmap. In addition to the above doc bug, we
should work to better socialize this filter-over-paginate methodology.

I understand all the things you write above about the problems that Keystone's underlying architecture (driver-based, not always able to do pagination in the individual drivers). However, it really does mean that Keystone is the only project in OpenStack that behaves this way. All other services provide limit/marker paginations, AFAIK, which is efficient and, while not the same UX as a filtering methodology, is entirely compatible and complementary to filtering.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Morgan_Fainberg (17,320 points)   2 6 9
0 votes

On Aug 14, 2015, at 12:19, Morgan Fainberg morgan.fainberg@gmail.com wrote:

Pagination in ldap requires holding a cursor open. You would have to map the requests to the same cursor each time. It costs memory and holds a client connected to the ldap server. In a REST api it is a bad idea. With regard to searching it can be done, but each query can be a different set of objects (order is not guaranteed). It isn't straight forward.

To put is bluntly, we are working to push user management to the tools that are better at this than keystone. The LDAP servers or AD have far better tools than the keystone API. And federated users are managed externally as well. The SQL table to manage users is not a good solution and we are making strides to eliminate the needs for even service users to exist here.

The question about roles and grants can be queried and appropriately paginated/limited/searched (again same statement about resource for project/domain where if it doesn't exist i wouldn't block it but it is likely a mitaka target).

That is to say "list all users that could have a role in keystone" is not a good question as it highlights all the aforementioned problems. Asking for a list of active assignments is a reasonable answer as it is tied to a backend that can support what you're asking for (it isnt directly tied to identity, but to the assignment/role backend)

--Morgan

Sent via mobile

On Aug 14, 2015, at 09:42, Fox, Kevin M Kevin.Fox@pnnl.gov wrote:

Surely ldap supports some form of pagination/searching natively. If any storage system of users needs to scale up to large numbers of users, its ldap...

Thanks,
Kevin
From: Timur Sufiev [tsufiev@mirantis.com]
Sent: Friday, August 14, 2015 9:20 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Keystone] [Horizon] Pagination support for Identity dashboard entities

Morgan,

Your reasoning is perfectly fine from the Keystone point of view. Yet I believe this approach is harmful for both Horizon and the whole OpenStack ecosystem.

It is harmful for the ecosystem, because it breaks API uniformity in one of the few areas where this uniformity could be achieved. Imagine if Nova or Cinder start saying the same thing: "we have too much drivers/backends to provide the uniform interface for all of them, let's delegate the choice of handling them differently to our consumers". It'll propagate the knowledge of different backends throughout the stack and it's obviously not good.

Not having pagination on Identity->Users page means that even with filtering being fully supported there will be problems. At least the first time the Users page with all the Users piped from production-grade LDAP through Keystone is shown in Horizon, it takes a lot time to render them all (before an unhappy admin had any chance to narrow the list), which eventually may result in connection being dropped by some HA balancer. We did these kinds of tests, the results weren't reassuring. Well, I might miss some of new Horizon angularization steps, so please regard this paragraph as my personal opinion - I don't think Horizon could be lighting fast on its own (i.e. without additional services) with a lot of data without pagination.

On Fri, Aug 14, 2015 at 6:03 PM Morgan Fainberg morgan.fainberg@gmail.com wrote:
For the identity (users and groups) backend as long as we support LDAP (and as side note federated users never show up in this list anyway) and with the drive towards pushing all user management out of keystone itself to ldap or other tools that do it better, I don't see pagination as something we should be providing. Providing an inconsistent user experience based on leaking underlying implementation details is something I am very against. This stance ensures that horizon and other tools like it will not need to know underlying implementation details to provide a consistent user experience. Unfortunately here we do need to cater to the lowest common denominator and filtering/searching/limiting is the clear common mechanism

With regards to resources (projects, domains, etc) since we no longer support using LDAP (deprecated with removal in mitaka) I have less strong feelings towards and wouldn't block efforts to implement if it is not already available (if not available this is likely a mitaka goal).

--Morgan

Sent via mobile

On Aug 14, 2015, at 07:39, Jay Pipes jaypipes@gmail.com wrote:

On 08/14/2015 09:14 AM, Morgan Fainberg wrote:
As a quick note the api-ref you are linking to has some gaps/has not
been kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3 sections
at the top) and there is a known open bug to work with the docs team to
get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly
different every single request meaning each page is not really
guaranteed to be the "next page" it could be the same/show inconsistent
results. The second issue is that the pagination is not a good UX even
where it works - the simple question is: if you can filter the results
how many pages deep do you go before refining the query; think of your
use of search engines.

In light of these issues Keystone has opted for a filter / limit
(config). If the results exceed the limit there is a truncation that
occurs and it is recommended extra filtering be applied to reduce the
total number of results.

This discussion has gone around a few times, pagination in keystone is
not currently on the roadmap. In addition to the above doc bug, we
should work to better socialize this filter-over-paginate methodology.

I understand all the things you write above about the problems that Keystone's underlying architecture (driver-based, not always able to do pagination in the individual drivers). However, it really does mean that Keystone is the only project in OpenStack that behaves this way. All other services provide limit/marker paginations, AFAIK, which is efficient and, while not the same UX as a filtering methodology, is entirely compatible and complementary to filtering.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Morgan_Fainberg (17,320 points)   2 6 9
0 votes

So I was one of the keystone folks who looked at pagination (hell, I even had an implementation - and the framework for it still exists in keystone). However, I think it is true to say that there were as many people (external to keystone) who thought pagination was a bad idea, as thought it was a good one. At the time, there was a drive to answer this debate corss-project so we would have a new consensus (as opposed to just assume that what we did before should be replicated everywhere). I’m actually unclear if that happened. We then add the complication of federation where keystone physically does not have access to the users (it only knows about users who are active "right now” and even that is pretty tenuous).

As Morgan has outlined, although there are solutions to at least the “traditional LDAP” backed keystone…they aren’t very pretty - and don’t sit easily with a REST API.

It’s really a dichotomy - we have grown up thinking that keystone can serve up users and groups…whereas the future of large enterprise systems (where you might think you need pagination) is one where keystone will probably NOT have access to the users.

Henry

On 14 Aug 2015, at 20:19, Morgan Fainberg morgan.fainberg@gmail.com wrote:

Pagination in ldap requires holding a cursor open. You would have to map the requests to the same cursor each time. It costs memory and holds a client connected to the ldap server. In a REST api it is a bad idea. With regard to searching it can be done, but each query can be a different set of objects (order is not guaranteed). It isn't straight forward.

To put is bluntly, we are working to push user management to the tools that are better at this than keystone. The LDAP servers or AD have far better tools than the keystone API. And federated users are managed externally as well. The SQL table to manage users is not a good solution and we are making strides to eliminate the needs for even service users to exist here.

The question about roles and grants can be queried and appropriately paginated/limited/searched (again same statement about resource for project/domain where if it doesn't exist i wouldn't block it but it is likely a mitaka target).

--Morgan

Sent via mobile

On Aug 14, 2015, at 09:42, Fox, Kevin M <Kevin.Fox@pnnl.gov Kevin.Fox@pnnl.gov> wrote:

Surely ldap supports some form of pagination/searching natively. If any storage system of users needs to scale up to large numbers of users, its ldap...

Thanks,
Kevin
From: Timur Sufiev [tsufiev@mirantis.com tsufiev@mirantis.com]
Sent: Friday, August 14, 2015 9:20 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Keystone] [Horizon] Pagination support for Identity dashboard entities

Morgan,

Your reasoning is perfectly fine from the Keystone point of view. Yet I believe this approach is harmful for both Horizon and the whole OpenStack ecosystem.

It is harmful for the ecosystem, because it breaks API uniformity in one of the few areas where this uniformity could be achieved. Imagine if Nova or Cinder start saying the same thing: "we have too much drivers/backends to provide the uniform interface for all of them, let's delegate the choice of handling them differently to our consumers". It'll propagate the knowledge of different backends throughout the stack and it's obviously not good.

Not having pagination on Identity->Users page means that even with filtering being fully supported there will be problems. At least the first time the Users page with all the Users piped from production-grade LDAP through Keystone is shown in Horizon, it takes a lot time to render them all (before an unhappy admin had any chance to narrow the list), which eventually may result in connection being dropped by some HA balancer. We did these kinds of tests, the results weren't reassuring. Well, I might miss some of new Horizon angularization steps, so please regard this paragraph as my personal opinion - I don't think Horizon could be lighting fast on its own (i.e. without additional services) with a lot of data without pagination.

On Fri, Aug 14, 2015 at 6:03 PM Morgan Fainberg <morgan.fainberg@gmail.com morgan.fainberg@gmail.com> wrote:
For the identity (users and groups) backend as long as we support LDAP (and as side note federated users never show up in this list anyway) and with the drive towards pushing all user management out of keystone itself to ldap or other tools that do it better, I don't see pagination as something we should be providing. Providing an inconsistent user experience based on leaking underlying implementation details is something I am very against. This stance ensures that horizon and other tools like it will not need to know underlying implementation details to provide a consistent user experience. Unfortunately here we do need to cater to the lowest common denominator and filtering/searching/limiting is the clear common mechanism

With regards to resources (projects, domains, etc) since we no longer support using LDAP (deprecated with removal in mitaka) I have less strong feelings towards and wouldn't block efforts to implement if it is not already available (if not available this is likely a mitaka goal).

--Morgan

Sent via mobile

On Aug 14, 2015, at 07:39, Jay Pipes <jaypipes@gmail.com jaypipes@gmail.com> wrote:

On 08/14/2015 09:14 AM, Morgan Fainberg wrote:
As a quick note the api-ref you are linking to has some gaps/has not
been kept in sync with the official api specifications.

The official API specification is located at
http://specs.openstack.org/openstack/keystone-specs/ (v2 and v3 sections
at the top) and there is a known open bug to work with the docs team to
get this in sync (somehow).

Unfortunately there are a number of cases especially with the identity
backend where pagination just does not work (or works completely
unreliably) such as utilizing the ldap driver. Either a cursor must be
maintained (problematic in REST) or the results could be wildly
different every single request meaning each page is not really
guaranteed to be the "next page" it could be the same/show inconsistent
results. The second issue is that the pagination is not a good UX even
where it works - the simple question is: if you can filter the results
how many pages deep do you go before refining the query; think of your
use of search engines.

In light of these issues Keystone has opted for a filter / limit
(config). If the results exceed the limit there is a truncation that
occurs and it is recommended extra filtering be applied to reduce the
total number of results.

This discussion has gone around a few times, pagination in keystone is
not currently on the roadmap. In addition to the above doc bug, we
should work to better socialize this filter-over-paginate methodology.

I understand all the things you write above about the problems that Keystone's underlying architecture (driver-based, not always able to do pagination in the individual drivers). However, it really does mean that Keystone is the only project in OpenStack that behaves this way. All other services provide limit/marker paginations, AFAIK, which is efficient and, while not the same UX as a filtering methodology, is entirely compatible and complementary to filtering.

Best,
-jay


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Aug 14, 2015 by Henry_Nash (1,080 points)   2
...