settingsLogin | Registersettings

[OpenStack-docs] Autogenerate the API Reference

0 votes

Hi all,

I'm writing to you because I think we should seriously consider
generating the API reference at least partially from the code itself,
rather than continuing to maintain a separate set of documents that has
no relationship with the code.

No-one can argue that it's an enormous amount of work to maintain an API
reference. Despite our best efforts, our API reference is incomplete and
inaccurate - you need only check the bug list[1] to know this.

The move to swagger[2] will help some of this, but it's not enough. Even
with new tooling, we are still stuck in a world of manual updates and
effort - spending a large amount of time converting a python
method/class definition into another format.

We do add great value in the descriptions of parameters, groupings and
overall display of the information. Diane is simply amazing here. So,
wouldn't it be great if we could focus just on that, without needing to
worry about changing this python docstring:

"""
Move your app forward with the Uber API
"""

into this swagger description:

info:
description: Move your app forward with the Uber API

and instead fix up the string itself ?

The move-to-swagger spec at the moment says: "We don't really want
autogenerated API documentation, because then how do you know if the
code is correct?". Which is really quite odd - because looking at the
code is exactly what developers, SDK makers and users are doing right
now when confronted with our incomplete, inaccurate documentation :) The
code is what defines the API - it's what actually processes the requests
and responses. If there's a problem where the implementation of that
processing is incorrect - we fix that in the code, not by telling users
something completely different in a document :)

We may yet get to the point where OpenStack APIs are
designed-in-document first, then implemented, but we're far from that at
the moment and have much more pressing problems to solve.

There are a number of python integrations that swagger supports[3].
Essentially, what these would allow us to do is use portions of the code
to generate swagger specs. This means that our documentation would
always be complete and accurate to what users actually experience when
they interact with OpenStack APIs.

The automation of option tables in the configuration reference is a good
parallel here. We used to have very incomplete and often outdated
configuration tables. After the automation, literally hundreds of bugs
have been solved with very little human intervention. The problem just
doesn't exist anymore, and we've clawed back some of our precious resources.

In the config-ref there are two types of auto-magic: non-swift, where
all text lives in the code and updates should be done there; and swift,
where the option name comes from the code, but the descriptive text must
be manually entered - overwriting a default label of "No help text
available for this option".

Applying that to API documents, I think it would be amazing to try the
former: it basically means that developers write their own API
documentation, happily, in the code, in a format they are familiar with.
We can take that and generate swagger for the API-ref and display it
appropriately.

However, that might not be possible, or might be too much work for a
single release. So, the second option is still a very good one - we use
automation to extract some information (eg it could be just class
titles, doc strings, parameter names) to generate swagger specs with the
placeholder text that we-as-the-docs-team can fill in with our best
efforts - which are excellent, since we're the pros at strings :)

The important thing is that this should be done continuously[4], as with
the config-ref, to ensure accuracy and completeness. Just looking at the
first 75 bugs in api-site[5], this approach would solve almost all of
them. This isn't something you can say about simply switching formats ;)

Anyway, thank you if you've managed to get to this point. I think we
have a great opportunity here to make a very significant change to the
way we do API documentation, and free up our previous docs resources to
really focus on the things we do best, rather than spending a lot of
time changing one arcane syntax into another :)

Regards,

Tom

[1] https://bugs.launchpad.net/openstack-api-site
[2] https://review.openstack.org/#/c/177934/
[3] https://github.com/swagger-api/swagger-spec#python , I found there's
a worked example showing the output of the swagger UI for
django-rest-swagger at:
http://django-rest-swagger.readthedocs.org/en/latest/examples.html
[4] Kinda once a milestone is how we're doing it with config-ref at the
moment, but this could in theory be fully automated
[5] https://bugs.launchpad.net/openstack-api-site/+bugs?orderby=id&start=0

-------- Forwarded Message --------
Subject: [Blueprint autogenerate-api-reference] Autogenerate API Reference
Date: Mon, 27 Apr 2015 15:14:41 -0000
From: Anne Gentle anne@openstack.org
Reply-To: Anne Gentle anne@openstack.org
To: tom@openstack.org

Blueprint changed by Anne Gentle:

Whiteboard set to:
We don't really want autogenerated API documentation, because then how
do you know if the code is correct? How can a quality engineer or SDK
dev trust the output? Autogeneration is fine for the first set of docs
but after that need to be treated as the "contract" for SDK developers
to trust.

--
Autogenerate API Reference
https://blueprints.launchpad.net/openstack-manuals/+spec/autogenerate-api-reference


OpenStack-docs mailing list
OpenStack-docs@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
asked Apr 29, 2015 in openstack-docs by Tom_Fifield (13,880 points)   2 3 5

2 Responses

0 votes

hi all, i missed the original email due to not being on this list so i'm
going to copy-paste and quote inline.

firstly, thanks for bringing this up Tom.

I'm writing to you because I think we should seriously consider
generating the API reference at least partially from the code itself,
rather than continuing to maintain a separate set of documents that has
no relationship with the code.

i agree, i think it would be great if projects could start from an
autogenerated base for their api-ref.

No-one can argue that it's an enormous amount of work to maintain an API
reference. Despite our best efforts, our API reference is incomplete and
inaccurate - you need only check the bug list[1] to know this.

The move to swagger[2] will help some of this, but it's not enough. Even
with new tooling, we are still stuck in a world of manual updates and
effort - spending a large amount of time converting a python
method/class definition into another format.

i'd love to see the density of workload shift slightly more on to the
project teams for the api-ref doc side of things. maybe it will help
ease some of the manual updating pressure on the doc team.

the sahara team recently did their api-ref wadls and i know it was a
hard fought battle with several of us participating. also big props to
Diane for helping us through that process.

We do add great value in the descriptions of parameters, groupings and
overall display of the information. Diane is simply amazing here. So,
wouldn't it be great if we could focus just on that, without needing to
worry about changing this python docstring:

"""
Move your app forward with the Uber API
"""

into this swagger description:

info:
description: Move your app forward with the Uber API

and instead fix up the string itself ?

i'm a big fan of increasing the usability of the docstrings to become
more informative. i realize this might be a slightly selfish desire as i
enjoy seeing them in the code, but i think it could pay off well if we
were to harvest the documentation to create something useful for users.

i would love to see some sort of oslo-ish package that would allow us to
have a neutral format in the docstrings that could then be transformed
into swagger or another format if necessary.

The move-to-swagger spec at the moment says: "We don't really want
autogenerated API documentation, because then how do you know if the
code is correct?". Which is really quite odd - because looking at the
code is exactly what developers, SDK makers and users are doing right
now when confronted with our incomplete, inaccurate documentation :) The
code is what defines the API - it's what actually processes the requests
and responses. If there's a problem where the implementation of that
processing is incorrect - we fix that in the code, not by telling users
something completely different in a document :)

+1, improving documentation in the code is a noble effort =)

We may yet get to the point where OpenStack APIs are
designed-in-document first, then implemented, but we're far from that at
the moment and have much more pressing problems to solve.

i'm completely intrigued by this process and i love what swagger is
proposing/doing in this problem space. i'm unsure that we would ever
reach this point with openstack projects but i think it would provide
interesting options for new projects if we had some sort of path for
this type of development.

i can imagine a world where a new project is able to get their wsgi
server/api up and running quickly by defining a nice swagger doc that
would transform into a code skeleton. imo, this would rock. it wouldn't
be a one-stop-shop solution, but a nice solid toolkit to start from.

There are a number of python integrations that swagger supports[3].
Essentially, what these would allow us to do is use portions of the code
to generate swagger specs. This means that our documentation would
always be complete and accurate to what users actually experience when
they interact with OpenStack APIs.

assuming we had the extra markup to make complete swagger docs it would
always be up to date. i think the sticking point is getting to the level
where the autogenerated output contains all the necessary information.

i think it would a great first step to have projects creating the
baseline for their docs from an autogenerated format, then marking up as
necessary. hopefully backporting the added information into their code,
or just improving it to begin with.

The automation of option tables in the configuration reference is a good
parallel here. We used to have very incomplete and often outdated
configuration tables. After the automation, literally hundreds of bugs
have been solved with very little human intervention. The problem just
doesn't exist anymore, and we've clawed back some of our precious resources.

In the config-ref there are two types of auto-magic: non-swift, where
all text lives in the code and updates should be done there; and swift,
where the option name comes from the code, but the descriptive text must
be manually entered - overwriting a default label of "No help text
available for this option".

Applying that to API documents, I think it would be amazing to try the
former: it basically means that developers write their own API
documentation, happily, in the code, in a format they are familiar with.
We can take that and generate swagger for the API-ref and display it
appropriately.

+1

i'd like to take it a step further and say that the projects themselves
could be responsible for creating the swagger and then the upstream
api-ref could grab those documents for aggregation into the doc team
managed site. this will make more work for the project teams, but i like
the idea of allowing the teams to have more control over their api output.

this might make it more difficult for the doc team to organize
consistency as they might have to reach back to the project teams if
there are issues with their generated docs, and i don't really have a
good answer for that at this point. perhaps this could be an area where
the team liaisons could help?

However, that might not be possible, or might be too much work for a
single release. So, the second option is still a very good one - we use
automation to extract some information (eg it could be just class
titles, doc strings, parameter names) to generate swagger specs with the
placeholder text that we-as-the-docs-team can fill in with our best
efforts - which are excellent, since we're the pros at strings :)

+1, this would be an awesome starting point

The important thing is that this should be done continuously[4], as with
the config-ref, to ensure accuracy and completeness. Just looking at the
first 75 bugs in api-site[5], this approach would solve almost all of
them. This isn't something you can say about simply switching formats ;)

Anyway, thank you if you've managed to get to this point. I think we
have a great opportunity here to make a very significant change to the
way we do API documentation, and free up our previous docs resources to
really focus on the things we do best, rather than spending a lot of
time changing one arcane syntax into another :)

i agree, i find the prospect of this really encouraging =)

mike

Regards,

Tom

[1] https://bugs.launchpad.net/openstack-api-site
[2] https://review.openstack.org/#/c/177934/
[3] https://github.com/swagger-api/swagger-spec#python , I found there's
a worked example showing the output of the swagger UI for
django-rest-swagger at:
http://django-rest-swagger.readthedocs.org/en/latest/examples.html
[4] Kinda once a milestone is how we're doing it with config-ref at the
moment, but this could in theory be fully automated
[5] https://bugs.launchpad.net/openstack-api-site/+bugs?orderby=id&start=0

-------- Forwarded Message --------
Subject: [Blueprint autogenerate-api-reference] Autogenerate API Reference
Date: Mon, 27 Apr 2015 15:14:41 -0000
From: Anne Gentle
Reply-To: Anne Gentle
To: tom at openstack.org

Blueprint changed by Anne Gentle:

Whiteboard set to:
We don't really want autogenerated API documentation, because then how
do you know if the code is correct? How can a quality engineer or SDK
dev trust the output? Autogeneration is fine for the first set of docs
but after that need to be treated as the "contract" for SDK developers
to trust.

--
Autogenerate API Reference
https://blueprints.launchpad.net/openstack-manuals/+spec/autogenerate-api-reference


OpenStack-docs mailing list
OpenStack-docs@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
responded Apr 30, 2015 by michael_mccune (5,080 points)   3 4
0 votes

On Tue, Apr 28, 2015 at 9:01 PM, Tom Fifield tom@openstack.org wrote:

Hi all,

I'm writing to you because I think we should seriously consider
generating the API reference at least partially from the code itself,
rather than continuing to maintain a separate set of documents that has
no relationship with the code.

I definitely want to seriously consider autogenerating, but for the
reference portion only. It's important to bring up the quality of the API
docs by not purely autogenerating (I know you're not advocating for that)
but for strategically scraping. I like the ideas you have; more below.

No-one can argue that it's an enormous amount of work to maintain an API
reference. Despite our best efforts, our API reference is incomplete and
inaccurate - you need only check the bug list[1] to know this.

The move to swagger[2] will help some of this, but it's not enough. Even
with new tooling, we are still stuck in a world of manual updates and
effort - spending a large amount of time converting a python
method/class definition into another format.

We do add great value in the descriptions of parameters, groupings and
overall display of the information. Diane is simply amazing here. So,
wouldn't it be great if we could focus just on that, without needing to
worry about changing this python docstring:

"""
Move your app forward with the Uber API
"""

into this swagger description:

info:
description: Move your app forward with the Uber API

and instead fix up the string itself ?

The move-to-swagger spec at the moment says: "We don't really want
autogenerated API documentation, because then how do you know if the
code is correct?". Which is really quite odd - because looking at the
code is exactly what developers, SDK makers and users are doing right
now when confronted with our incomplete, inaccurate documentation :) The
code is what defines the API - it's what actually processes the requests
and responses. If there's a problem where the implementation of that
processing is incorrect - we fix that in the code, not by telling users
something completely different in a document :)

Well, we certainly won't solve the problem of inaccuracy by relying on
inaccurate code! But what I think will help is the API Working Group's
reviews and ensuring no backwards compatible changes get introduced. So the
tactic here is to ask the API WG to tackle the "contractual agreement"
portion of reviews that have APIImpact. Plus, keeping reviews with the core
teams will help immensely in keeping up with change.

We may yet get to the point where OpenStack APIs are
designed-in-document first, then implemented, but we're far from that at
the moment and have much more pressing problems to solve.

Agreed, and we have years of data to back that up.

There are a number of python integrations that swagger supports[3].
Essentially, what these would allow us to do is use portions of the code
to generate swagger specs. This means that our documentation would
always be complete and accurate to what users actually experience when
they interact with OpenStack APIs.

The automation of option tables in the configuration reference is a good
parallel here. We used to have very incomplete and often outdated
configuration tables. After the automation, literally hundreds of bugs
have been solved with very little human intervention. The problem just
doesn't exist anymore, and we've clawed back some of our precious
resources.

In the config-ref there are two types of auto-magic: non-swift, where
all text lives in the code and updates should be done there; and swift,
where the option name comes from the code, but the descriptive text must
be manually entered - overwriting a default label of "No help text
available for this option".

Applying that to API documents, I think it would be amazing to try the
former: it basically means that developers write their own API
documentation, happily, in the code, in a format they are familiar with.
We can take that and generate swagger for the API-ref and display it
appropriately.

However, that might not be possible, or might be too much work for a
single release. So, the second option is still a very good one - we use
automation to extract some information (eg it could be just class
titles, doc strings, parameter names) to generate swagger specs with the
placeholder text that we-as-the-docs-team can fill in with our best
efforts - which are excellent, since we're the pros at strings :)

I think this is a great first step, and I'd like to scope it to
infrastructure services only as well to keep the scope do-able.

The important thing is that this should be done continuously[4], as with
the config-ref, to ensure accuracy and completeness. Just looking at the
first 75 bugs in api-site[5], this approach would solve almost all of
them. This isn't something you can say about simply switching formats ;)

True. I do not want to switch formats only. I want to paint a picture of
the best future for this doc set. The overarching goals are:
- better process for keeping docs updated
- more trust in the accuracy of the API reference docs

I think by moving sourcing closer to the project repos themselves it'll
help with both goals. We'll just have to place accuracy and consumer
advocacy high in the review priority.

These new developer guides should enable app devs to make powerful apps on
OpenStack clouds. Those developer guides will be reviewed by the API WG,
the service devs themselves, and dev doc writers. Let's automate where it
makes sense to. The approach is not purely auto-generation -- that only
gets us halfway there. We also have to source docs where the best reviewers
live. We can insert narrative, helpful content that is heavily reviewed,
vetted, and offers high quality. The contract agreement burden can happen
at review time (for this iteration, we can tackle gated validation next if
we like).

Let's scrape code for resources, parameters, headers, requests and
responses, that'll definitely help. In doing so we can bake-in the
versioning of the API reference, does this run on icehouse? sorts of
questions.

I'll update the docs-spec, and I think I can get some great wireframes in
time for the Summit that show how awesome the future can be. I'll need web
dev, Python development, and time, but this is definitely shaping into
something do-able. Sweet.

I appreciate the API WG letting me discuss and shape with them at their
meeting this week also; feel free to read that log if you want to catch up.
[1]

Anne

1.
http://eavesdrop.openstack.org/meetings/api_wg/2015/api_wg.2015-04-30-00.01.log.html

Anyway, thank you if you've managed to get to this point. I think we
have a great opportunity here to make a very significant change to the
way we do API documentation, and free up our previous docs resources to
really focus on the things we do best, rather than spending a lot of
time changing one arcane syntax into another :)

Regards,

Tom

[1] https://bugs.launchpad.net/openstack-api-site
[2] https://review.openstack.org/#/c/177934/
[3] https://github.com/swagger-api/swagger-spec#python , I found there's
a worked example showing the output of the swagger UI for
django-rest-swagger at:
http://django-rest-swagger.readthedocs.org/en/latest/examples.html
[4] Kinda once a milestone is how we're doing it with config-ref at the
moment, but this could in theory be fully automated
[5] https://bugs.launchpad.net/openstack-api-site/+bugs?orderby=id&start=0

-------- Forwarded Message --------
Subject: [Blueprint autogenerate-api-reference] Autogenerate API Reference
Date: Mon, 27 Apr 2015 15:14:41 -0000
From: Anne Gentle anne@openstack.org
Reply-To: Anne Gentle anne@openstack.org
To: tom@openstack.org

Blueprint changed by Anne Gentle:

Whiteboard set to:
We don't really want autogenerated API documentation, because then how
do you know if the code is correct? How can a quality engineer or SDK
dev trust the output? Autogeneration is fine for the first set of docs
but after that need to be treated as the "contract" for SDK developers
to trust.

--
Autogenerate API Reference

https://blueprints.launchpad.net/openstack-manuals/+spec/autogenerate-api-reference


OpenStack-docs mailing list
OpenStack-docs@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs

--
Anne Gentle
annegentle@justwriteclick.com


OpenStack-docs mailing list
OpenStack-docs@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
responded May 1, 2015 by annegentle_at_justwr (9,780 points)   2 5 6
...