settingsLogin | Registersettings

[openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs

0 votes

Hey everybody,

Oh goodie, I can hear you say, let's definitely spend some time
bikeshedding about specific command invocations related to building docs
and tarballs!!!

tl;dr I want to change the PTI for docs and tarball building to be less
OpenStack-specific

The Problem
===========

As we work on Zuul v3, there are a set of job definitions that are
rather fundamental that can totally be directly shared between Zuul
installations whether those Zuuls are working with OpenStack content or
not. As an example "tox -epy27" is a fairly standard thing, so a Zuul
job called "tox-py27" has no qualities specific to OpenStack and could
realistically be used by anyone who uses tox to manage their project.

Docs and Tarballs builds for us, however, are the following:

tox -evenv -- python setup.py sdist
tox -evenv -- python setup.py build_sphinx

Neither of those are things that are likely to work outside of
OpenStack. (The 'venv' tox environment is not a default tox thing)

I'm going to argue neither of them are actually providing us with much
value.

Tarball Creation
================

Tarball creation is super simple. setup_requires is already handled out
of band of everything else. Go clone nova into a completely empty system
and run python setup.py sdist ... and it works. (actually, nova is big.
use something smaller like gertty ...)

docker run -it --rm python bash -c 'git clone \
  https://git.openstack.org/openstack/gertty && cd gertty \
  && python setup.py sdist'

There is not much value in that tox wrapper - and it's not like it's
making it EASIER to run the command. In fact, it's more typing.

I propose we change the PTI from:

tox -evenv python setup.py sdist

to:

python setup.py sdist

and then change the gate jobs to use the non-tox form of the command.

I'd also like to further change it to be explicit that we also build
wheels. So the ACTUAL commands that the project should support are:

python setup.py sdist
python setup.py bdist_wheel

All of our projects support this already, so this should be a no-op.

Notes:

  • Python projects that need to build C extensions might need their pip
    requirements (and bindep requirements) installed in order to run
    bdist_wheel. We do not support that broadly at the moment ANYWAY - so
    I'd like to leave that as an outlier and handle it when we need to
    handle it.

  • It's possible that somewhere we have a repo that has somehow done
    something that would cause python setup.py sdist or python setup.py
    bdist_wheel to not work without pip requirements installed. I believe we
    should consider that a bug and fix it in the project if we find such a
    thing - but since we use pbr in all of the OpenStack projects, I find it
    extremely unlikely.

Governance patch submitted: https://review.openstack.org/508693

Sphinx Documentation
====================

Doc builds are more complex - but I think there is a high amount of
value in changing how we invoke them for a few reasons.

a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
b) we decided to use sphinx for go and javascript - but we invoke sphinx
differently for each of those (since they naturally don't have tox),
meaning we can't just have a "build-sphinx-docs" job and even share it
with ourselves.
c) readthedocs.org is an excellent Open Source site that builds and
hosts sphinx docs for projects. They have an interface for docs
requirements documented and defined that we can align. By aligning,
projects can use migrate between docs.o.o and readthedocs.org and still
have a consistent experience.

The PTI I'd like to propose for this is more complex, so I'd like to
describe it in terms of:

  • OpenStack organizational requirements
  • helper sugar for developers with per-language recommendations

I believe the result can be a single in-tree doc PTI that applies to
python, go and javascript - and a single Zuul job that applies to all of
our projects AND non-OpenStack projects as well.

Organizational Requirements


Following readthedocs.org logic we can actually support a wider range of
schemes technically, but there is human value in having consistency on
these topics across our OpenStack repos.

  • docs live in doc/source
  • Python requirements needed by Sphinx to build the docs live in
    doc/requirements.txt

If the project is python:

  • doc/requirements.txt can assume the project will have been installed
  • The following should be set in setup.cfg:

    [build_sphinx]
    source-dir = doc/source
    build-dir = doc/build

Doing the above allows the following commands to work cleanly in
automation no matter what the language is:

[ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
sphinx-build -b doc/source doc/build

No additional commands should be required.

The setup.cfg stanza allows:

python setup.py build_sphinx

to continue to work. (also, everyone already has one)

Helper sugar for developers


The following are recommended ways to provide in-tree developer convenience:

  • tox -edocs # recommended for Python projects
  • make docs # recommended for golang projects
  • npm run document # recommended for javascript projects

These will not be used by Zuul - they are there purely for local
developer convenience.

Rolling it Out


First, the governance patch:

https://review.openstack.org/508694

I've also written a patch for pbr to add support for finding a
doc/requirements.txt file, if one exists, and adding the contents to a
"docs" extra:

https://review.openstack.org/#/c/492620

That will let people do this in their tox.ini files:

[testenv:docs]
extras = docs
commands=
python setup.py build_sphinx

And a WIP job for Zuul v3 written to provide a non-tox based build:

https://review.openstack.org/#/c/492709/

(that job can be improved a little if we land the pbr docs extras patch)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
asked Oct 9, 2017 in openstack-dev by Monty_Taylor (22,780 points)   2 4 7

5 Responses

0 votes

On 2017-09-30 10:20:08 -0500 (-0500), Monty Taylor wrote:
[...]
I want to change the PTI for docs and tarball building to be less
OpenStack-specific
[...]

Looks great! Anything to make OpenStack less snowflake-like and our
build processes more familiar to developers from other communities.
--
Jeremy Stanley


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

responded Sep 30, 2017 by Jeremy_Stanley (56,700 points)   3 5 7
0 votes

Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:

Hey everybody,

Oh goodie, I can hear you say, let's definitely spend some time
bikeshedding about specific command invocations related to building docs
and tarballs!!!

tl;dr I want to change the PTI for docs and tarball building to be less
OpenStack-specific

The Problem
===========

As we work on Zuul v3, there are a set of job definitions that are
rather fundamental that can totally be directly shared between Zuul
installations whether those Zuuls are working with OpenStack content or
not. As an example "tox -epy27" is a fairly standard thing, so a Zuul
job called "tox-py27" has no qualities specific to OpenStack and could
realistically be used by anyone who uses tox to manage their project.

Docs and Tarballs builds for us, however, are the following:

tox -evenv -- python setup.py sdist
tox -evenv -- python setup.py build_sphinx

Neither of those are things that are likely to work outside of
OpenStack. (The 'venv' tox environment is not a default tox thing)

I'm going to argue neither of them are actually providing us with much
value.

Tarball Creation
================

Tarball creation is super simple. setup_requires is already handled out
of band of everything else. Go clone nova into a completely empty system
and run python setup.py sdist ... and it works. (actually, nova is big.
use something smaller like gertty ...)

docker run -it --rm python bash -c 'git clone \
  https://git.openstack.org/openstack/gertty && cd gertty \
  && python setup.py sdist'

There is not much value in that tox wrapper - and it's not like it's
making it EASIER to run the command. In fact, it's more typing.

I propose we change the PTI from:

tox -evenv python setup.py sdist

to:

python setup.py sdist

and then change the gate jobs to use the non-tox form of the command.

I'd also like to further change it to be explicit that we also build
wheels. So the ACTUAL commands that the project should support are:

python setup.py sdist
python setup.py bdist_wheel

All of our projects support this already, so this should be a no-op.

Notes:

  • Python projects that need to build C extensions might need their pip
    requirements (and bindep requirements) installed in order to run
    bdist_wheel. We do not support that broadly at the moment ANYWAY - so
    I'd like to leave that as an outlier and handle it when we need to
    handle it.

  • It's possible that somewhere we have a repo that has somehow done
    something that would cause python setup.py sdist or python setup.py
    bdist_wheel to not work without pip requirements installed. I believe we
    should consider that a bug and fix it in the project if we find such a
    thing - but since we use pbr in all of the OpenStack projects, I find it
    extremely unlikely.

Governance patch submitted: https://review.openstack.org/508693

Sphinx Documentation
====================

Doc builds are more complex - but I think there is a high amount of
value in changing how we invoke them for a few reasons.

a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
b) we decided to use sphinx for go and javascript - but we invoke sphinx
differently for each of those (since they naturally don't have tox),
meaning we can't just have a "build-sphinx-docs" job and even share it
with ourselves.
c) readthedocs.org is an excellent Open Source site that builds and
hosts sphinx docs for projects. They have an interface for docs
requirements documented and defined that we can align. By aligning,
projects can use migrate between docs.o.o and readthedocs.org and still
have a consistent experience.

The PTI I'd like to propose for this is more complex, so I'd like to
describe it in terms of:

  • OpenStack organizational requirements
  • helper sugar for developers with per-language recommendations

I believe the result can be a single in-tree doc PTI that applies to
python, go and javascript - and a single Zuul job that applies to all of
our projects AND non-OpenStack projects as well.

Organizational Requirements


Following readthedocs.org logic we can actually support a wider range of
schemes technically, but there is human value in having consistency on
these topics across our OpenStack repos.

  • docs live in doc/source
  • Python requirements needed by Sphinx to build the docs live in
    doc/requirements.txt

If the project is python:

  • doc/requirements.txt can assume the project will have been installed
  • The following should be set in setup.cfg:

    [build_sphinx]
    source-dir = doc/source
    build-dir = doc/build

Doing the above allows the following commands to work cleanly in
automation no matter what the language is:

[ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
sphinx-build -b doc/source doc/build

I suspect you mean "sphinx-build -b html doc/source doc/build" (the
"html" arg was missing).

We should also include a PDF build, since it is useful for
users who do not have good access to docs.o.o (especially when
governments block the site).

We should also formalize how we are going to handle translations
for both HTML and PDF. I think some of that is implemented, but I'm
not sure if it's actually documented anywhere.

Ian started a separate thread about this [1] that didn't see much
in the way of response because everyone was involved in the zuul
migration work.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122461.html

No additional commands should be required.

The setup.cfg stanza allows:

python setup.py build_sphinx

to continue to work. (also, everyone already has one)

Helper sugar for developers


The following are recommended ways to provide in-tree developer convenience:

  • tox -edocs # recommended for Python projects
  • make docs # recommended for golang projects
  • npm run document # recommended for javascript projects

These will not be used by Zuul - they are there purely for local
developer convenience.

Rolling it Out


First, the governance patch:

https://review.openstack.org/508694

I've also written a patch for pbr to add support for finding a
doc/requirements.txt file, if one exists, and adding the contents to a
"docs" extra:

https://review.openstack.org/#/c/492620

That will let people do this in their tox.ini files:

[testenv:docs]
extras = docs
commands=
python setup.py build_sphinx

And a WIP job for Zuul v3 written to provide a non-tox based build:

https://review.openstack.org/#/c/492709/

(that job can be improved a little if we land the pbr docs extras patch)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 30, 2017 by Doug_Hellmann (87,520 points)   3 4 7
0 votes

Having a PDF (or similar offline copy) was also requested during OpenStack UK days event during the executive Q&A with jbryce.

Tim

-----Original Message-----
From: Doug Hellmann doug@doughellmann.com
Reply-To: "OpenStack Development Mailing List (not for usage questions)" openstack-dev@lists.openstack.org
Date: Saturday, 30 September 2017 at 17:44
To: openstack-dev openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [tc][docs][release] Updating the PTI for docs and tarballs

Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:
> Hey everybody,
> 
> Oh goodie, I can hear you say, let's definitely spend some time 
> bikeshedding about specific command invocations related to building docs 
> and tarballs!!!
> 
> tl;dr I want to change the PTI for docs and tarball building to be less 
> OpenStack-specific
> 
> The Problem
> ===========
> 
> As we work on Zuul v3, there are a set of job definitions that are 
> rather fundamental that can totally be directly shared between Zuul 
> installations whether those Zuuls are working with OpenStack content or 
> not. As an example "tox -epy27" is a fairly standard thing, so a Zuul 
> job called "tox-py27" has no qualities specific to OpenStack and could 
> realistically be used by anyone who uses tox to manage their project.
> 
> Docs and Tarballs builds for us, however, are the following:
> 
> tox -evenv -- python setup.py sdist
> tox -evenv -- python setup.py build_sphinx
> 
> Neither of those are things that are likely to work outside of 
> OpenStack. (The 'venv' tox environment is not a default tox thing)
> 
> I'm going to argue neither of them are actually providing us with much 
> value.
> 
> Tarball Creation
> ================
> 
> Tarball creation is super simple. setup_requires is already handled out 
> of band of everything else. Go clone nova into a completely empty system 
> and run python setup.py sdist ... and it works. (actually, nova is big. 
> use something smaller like gertty ...)
> 
>     docker run -it --rm python bash -c 'git clone \
>       https://git.openstack.org/openstack/gertty && cd gertty \
>       && python setup.py sdist'
> 
> There is not much value in that tox wrapper - and it's not like it's 
> making it EASIER to run the command. In fact, it's more typing.
> 
> I propose we change the PTI from:
> 
>    tox -evenv python setup.py sdist
> 
> to:
> 
>    python setup.py sdist
> 
> and then change the gate jobs to use the non-tox form of the command.
> 
> I'd also like to further change it to be explicit that we also build 
> wheels. So the ACTUAL commands that the project should support are:
> 
>    python setup.py sdist
>    python setup.py bdist_wheel
> 
> All of our projects support this already, so this should be a no-op.
> 
> Notes:
> 
> * Python projects that need to build C extensions might need their pip 
> requirements (and bindep requirements) installed in order to run 
> bdist_wheel. We do not support that broadly at the moment ANYWAY - so 
> I'd like to leave that as an outlier and handle it when we need to 
> handle it.
> 
> * It's *possible* that somewhere we have a repo that has somehow done 
> something that would cause python setup.py sdist or python setup.py 
> bdist_wheel to not work without pip requirements installed. I believe we 
> should consider that a bug and fix it in the project if we find such a 
> thing - but since we use pbr in all of the OpenStack projects, I find it 
> extremely unlikely.
> 
> Governance patch submitted: https://review.openstack.org/508693
> 
> Sphinx Documentation
> ====================
> 
> Doc builds are more complex - but I think there is a high amount of 
> value in changing how we invoke them for a few reasons.
> 
> a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
> b) we decided to use sphinx for go and javascript - but we invoke sphinx 
> differently for each of those (since they naturally don't have tox), 
> meaning we can't just have a "build-sphinx-docs" job and even share it 
> with ourselves.
> c) readthedocs.org is an excellent Open Source site that builds and 
> hosts sphinx docs for projects. They have an interface for docs 
> requirements documented and defined that we can align. By aligning, 
> projects can use migrate between docs.o.o and readthedocs.org and still 
> have a consistent experience.
> 
> The PTI I'd like to propose for this is more complex, so I'd like to 
> describe it in terms of:
> 
> - OpenStack organizational requirements
> - helper sugar for developers with per-language recommendations
> 
> I believe the result can be a single in-tree doc PTI that applies to 
> python, go and javascript - and a single Zuul job that applies to all of 
> our projects AND non-OpenStack projects as well.
> 
> Organizational Requirements
> ---------------------------
> 
> Following readthedocs.org logic we can actually support a wider range of 
> schemes technically, but there is human value in having consistency on 
> these topics across our OpenStack repos.
> 
> * docs live in doc/source
> * Python requirements needed by Sphinx to build the docs live in 
> doc/requirements.txt
> 
> If the project is python:
> 
> * doc/requirements.txt can assume the project will have been installed
> * The following should be set in setup.cfg:
> 
>    [build_sphinx]
>    source-dir = doc/source
>    build-dir = doc/build
> 
> Doing the above allows the following commands to work cleanly in 
> automation no matter what the language is:
> 
>    [ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
>    sphinx-build -b doc/source doc/build
I suspect you mean "sphinx-build -b html doc/source doc/build" (the
"html" arg was missing).

We should also include a PDF build, since it is useful for
users who do not have good access to docs.o.o (especially when
governments block the site).

We should also formalize how we are going to handle translations
for both HTML and PDF. I think some of that is implemented, but I'm
not sure if it's actually documented anywhere.

Ian started a separate thread about this [1] that didn't see much
in the way of response because everyone was involved in the zuul
migration work.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122461.html
> 
> No additional commands should be required.
> 
> The setup.cfg stanza allows:
> 
>    python setup.py build_sphinx
> 
> to continue to work. (also, everyone already has one)
> 
> Helper sugar for developers
> ---------------------------
> 
> The following are recommended ways to provide in-tree developer convenience:
> 
> * tox -edocs  # recommended for Python projects
> * make docs  # recommended for golang projects
> * npm run document  # recommended for javascript projects
> 
> These will not be used by Zuul - they are there purely for local 
> developer convenience.
> 
> Rolling it Out
> --------------
> 
> First, the governance patch:
> 
>    https://review.openstack.org/508694
> 
> I've also written a patch for pbr to add support for finding a 
> doc/requirements.txt file, if one exists, and adding the contents to a 
> "docs" extra:
> 
>    https://review.openstack.org/#/c/492620
> 
> That will let people do this in their tox.ini files:
> 
>    [testenv:docs]
>    extras = docs
>    commands=
>      python setup.py build_sphinx
> 
> And a WIP job for Zuul v3 written to provide a non-tox based build:
> 
>    https://review.openstack.org/#/c/492709/
> 
> (that job can be improved a little if we land the pbr docs extras patch)
> 
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Sep 30, 2017 by Tim_Bell (16,440 points)   1 5 10
0 votes

On 09/30/2017 10:40 AM, Doug Hellmann wrote:
Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:

Hey everybody,

Oh goodie, I can hear you say, let's definitely spend some time
bikeshedding about specific command invocations related to building docs
and tarballs!!!

tl;dr I want to change the PTI for docs and tarball building to be less
OpenStack-specific

The Problem
===========

As we work on Zuul v3, there are a set of job definitions that are
rather fundamental that can totally be directly shared between Zuul
installations whether those Zuuls are working with OpenStack content or
not. As an example "tox -epy27" is a fairly standard thing, so a Zuul
job called "tox-py27" has no qualities specific to OpenStack and could
realistically be used by anyone who uses tox to manage their project.

Docs and Tarballs builds for us, however, are the following:

tox -evenv -- python setup.py sdist
tox -evenv -- python setup.py build_sphinx

Neither of those are things that are likely to work outside of
OpenStack. (The 'venv' tox environment is not a default tox thing)

I'm going to argue neither of them are actually providing us with much
value.

Tarball Creation
================

Tarball creation is super simple. setup_requires is already handled out
of band of everything else. Go clone nova into a completely empty system
and run python setup.py sdist ... and it works. (actually, nova is big.
use something smaller like gertty ...)

 docker run -it --rm python bash -c 'git clone \
   https://git.openstack.org/openstack/gertty && cd gertty \
   && python setup.py sdist'

There is not much value in that tox wrapper - and it's not like it's
making it EASIER to run the command. In fact, it's more typing.

I propose we change the PTI from:

tox -evenv python setup.py sdist

to:

python setup.py sdist

and then change the gate jobs to use the non-tox form of the command.

I'd also like to further change it to be explicit that we also build
wheels. So the ACTUAL commands that the project should support are:

python setup.py sdist
python setup.py bdist_wheel

All of our projects support this already, so this should be a no-op.

Notes:

  • Python projects that need to build C extensions might need their pip
    requirements (and bindep requirements) installed in order to run
    bdist_wheel. We do not support that broadly at the moment ANYWAY - so
    I'd like to leave that as an outlier and handle it when we need to
    handle it.

  • It's possible that somewhere we have a repo that has somehow done
    something that would cause python setup.py sdist or python setup.py
    bdist_wheel to not work without pip requirements installed. I believe we
    should consider that a bug and fix it in the project if we find such a
    thing - but since we use pbr in all of the OpenStack projects, I find it
    extremely unlikely.

Governance patch submitted: https://review.openstack.org/508693

Sphinx Documentation
====================

Doc builds are more complex - but I think there is a high amount of
value in changing how we invoke them for a few reasons.

a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
b) we decided to use sphinx for go and javascript - but we invoke sphinx
differently for each of those (since they naturally don't have tox),
meaning we can't just have a "build-sphinx-docs" job and even share it
with ourselves.
c) readthedocs.org is an excellent Open Source site that builds and
hosts sphinx docs for projects. They have an interface for docs
requirements documented and defined that we can align. By aligning,
projects can use migrate between docs.o.o and readthedocs.org and still
have a consistent experience.

The PTI I'd like to propose for this is more complex, so I'd like to
describe it in terms of:

  • OpenStack organizational requirements
  • helper sugar for developers with per-language recommendations

I believe the result can be a single in-tree doc PTI that applies to
python, go and javascript - and a single Zuul job that applies to all of
our projects AND non-OpenStack projects as well.

Organizational Requirements


Following readthedocs.org logic we can actually support a wider range of
schemes technically, but there is human value in having consistency on
these topics across our OpenStack repos.

  • docs live in doc/source
  • Python requirements needed by Sphinx to build the docs live in
    doc/requirements.txt

If the project is python:

  • doc/requirements.txt can assume the project will have been installed
  • The following should be set in setup.cfg:

    [build_sphinx]
    source-dir = doc/source
    build-dir = doc/build

Doing the above allows the following commands to work cleanly in
automation no matter what the language is:

[ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
sphinx-build -b doc/source doc/build

I suspect you mean "sphinx-build -b html doc/source doc/build" (the
"html" arg was missing).

Yes! Please amend all code examples with the things to make them
actually work. :)

We should also include a PDF build, since it is useful for
users who do not have good access to docs.o.o (especially when
governments block the site).

YES

We should also formalize how we are going to handle translations
for both HTML and PDF. I think some of that is implemented, but I'm
not sure if it's actually documented anywhere.

We've got translation handling connected to sphinx builds integrated
into the releasenotes job. I was just saying to AJaeger that I would
love to see that extracted into a Sphinx plugin perhaps - but whatever
it is, I could not agree more.

It also occurs to me (just was poking at the v3 releasenotes job updates
mnaser did) - that we should perhaps:

a) add reno releasenotes to the PTI for all languages
b) similar to doc/requirements.txt add support for
releasenotes/requirements.txt - so that we could share much of the same
logic between releasenotes and docs jobs.

and ...

c) probably add 'use openstackdocstheme" to the docs PTI.

Ian started a separate thread about this [1] that didn't see much
in the way of response because everyone was involved in the zuul
migration work.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122461.html

Awesome. I'll respond over there too.

No additional commands should be required.

The setup.cfg stanza allows:

python setup.py build_sphinx

to continue to work. (also, everyone already has one)

Helper sugar for developers


The following are recommended ways to provide in-tree developer convenience:

  • tox -edocs # recommended for Python projects
  • make docs # recommended for golang projects
  • npm run document # recommended for javascript projects

These will not be used by Zuul - they are there purely for local
developer convenience.

Rolling it Out


First, the governance patch:

https://review.openstack.org/508694

I've also written a patch for pbr to add support for finding a
doc/requirements.txt file, if one exists, and adding the contents to a
"docs" extra:

https://review.openstack.org/#/c/492620

That will let people do this in their tox.ini files:

[testenv:docs]
extras = docs
commands=
  python setup.py build_sphinx

And a WIP job for Zuul v3 written to provide a non-tox based build:

https://review.openstack.org/#/c/492709/

(that job can be improved a little if we land the pbr docs extras patch)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 5, 2017 by Monty_Taylor (22,780 points)   2 4 7
0 votes

Excerpts from Monty Taylor's message of 2017-10-05 08:45:52 -0500:

On 09/30/2017 10:40 AM, Doug Hellmann wrote:

Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:

Hey everybody,

Oh goodie, I can hear you say, let's definitely spend some time
bikeshedding about specific command invocations related to building docs
and tarballs!!!

tl;dr I want to change the PTI for docs and tarball building to be less
OpenStack-specific

The Problem
===========

As we work on Zuul v3, there are a set of job definitions that are
rather fundamental that can totally be directly shared between Zuul
installations whether those Zuuls are working with OpenStack content or
not. As an example "tox -epy27" is a fairly standard thing, so a Zuul
job called "tox-py27" has no qualities specific to OpenStack and could
realistically be used by anyone who uses tox to manage their project.

Docs and Tarballs builds for us, however, are the following:

tox -evenv -- python setup.py sdist
tox -evenv -- python setup.py build_sphinx

Neither of those are things that are likely to work outside of
OpenStack. (The 'venv' tox environment is not a default tox thing)

I'm going to argue neither of them are actually providing us with much
value.

Tarball Creation
================

Tarball creation is super simple. setup_requires is already handled out
of band of everything else. Go clone nova into a completely empty system
and run python setup.py sdist ... and it works. (actually, nova is big.
use something smaller like gertty ...)

 docker run -it --rm python bash -c 'git clone \
   https://git.openstack.org/openstack/gertty && cd gertty \
   && python setup.py sdist'

There is not much value in that tox wrapper - and it's not like it's
making it EASIER to run the command. In fact, it's more typing.

I propose we change the PTI from:

tox -evenv python setup.py sdist

to:

python setup.py sdist

and then change the gate jobs to use the non-tox form of the command.

I'd also like to further change it to be explicit that we also build
wheels. So the ACTUAL commands that the project should support are:

python setup.py sdist
python setup.py bdist_wheel

All of our projects support this already, so this should be a no-op.

Notes:

  • Python projects that need to build C extensions might need their pip
    requirements (and bindep requirements) installed in order to run
    bdist_wheel. We do not support that broadly at the moment ANYWAY - so
    I'd like to leave that as an outlier and handle it when we need to
    handle it.

  • It's possible that somewhere we have a repo that has somehow done
    something that would cause python setup.py sdist or python setup.py
    bdist_wheel to not work without pip requirements installed. I believe we
    should consider that a bug and fix it in the project if we find such a
    thing - but since we use pbr in all of the OpenStack projects, I find it
    extremely unlikely.

Governance patch submitted: https://review.openstack.org/508693

Sphinx Documentation
====================

Doc builds are more complex - but I think there is a high amount of
value in changing how we invoke them for a few reasons.

a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
b) we decided to use sphinx for go and javascript - but we invoke sphinx
differently for each of those (since they naturally don't have tox),
meaning we can't just have a "build-sphinx-docs" job and even share it
with ourselves.
c) readthedocs.org is an excellent Open Source site that builds and
hosts sphinx docs for projects. They have an interface for docs
requirements documented and defined that we can align. By aligning,
projects can use migrate between docs.o.o and readthedocs.org and still
have a consistent experience.

The PTI I'd like to propose for this is more complex, so I'd like to
describe it in terms of:

  • OpenStack organizational requirements
  • helper sugar for developers with per-language recommendations

I believe the result can be a single in-tree doc PTI that applies to
python, go and javascript - and a single Zuul job that applies to all of
our projects AND non-OpenStack projects as well.

Organizational Requirements


Following readthedocs.org logic we can actually support a wider range of
schemes technically, but there is human value in having consistency on
these topics across our OpenStack repos.

  • docs live in doc/source
  • Python requirements needed by Sphinx to build the docs live in
    doc/requirements.txt

If the project is python:

  • doc/requirements.txt can assume the project will have been installed
  • The following should be set in setup.cfg:

    [build_sphinx]
    source-dir = doc/source
    build-dir = doc/build

Doing the above allows the following commands to work cleanly in
automation no matter what the language is:

[ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
sphinx-build -b doc/source doc/build

I suspect you mean "sphinx-build -b html doc/source doc/build" (the
"html" arg was missing).

Yes! Please amend all code examples with the things to make them
actually work. :)

We should also include a PDF build, since it is useful for
users who do not have good access to docs.o.o (especially when
governments block the site).

YES

We should also formalize how we are going to handle translations
for both HTML and PDF. I think some of that is implemented, but I'm
not sure if it's actually documented anywhere.

We've got translation handling connected to sphinx builds integrated
into the releasenotes job. I was just saying to AJaeger that I would
love to see that extracted into a Sphinx plugin perhaps - but whatever
it is, I could not agree more.

It also occurs to me (just was poking at the v3 releasenotes job updates
mnaser did) - that we should perhaps:

a) add reno releasenotes to the PTI for all languages
b) similar to doc/requirements.txt add support for
releasenotes/requirements.txt - so that we could share much of the same
logic between releasenotes and docs jobs.

and ...

c) probably add 'use openstackdocstheme" to the docs PTI.

Those additions all make sense. Reno is most useful where we have stable
branches, but it does address merge conflict issues on a single branch,
too. And building the release notes with sphinx would work the same way
as building the docs, just using different paths as input arguments.

Ian started a separate thread about this [1] that didn't see much
in the way of response because everyone was involved in the zuul
migration work.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122461.html

Awesome. I'll respond over there too.

No additional commands should be required.

The setup.cfg stanza allows:

python setup.py build_sphinx

to continue to work. (also, everyone already has one)

Helper sugar for developers


The following are recommended ways to provide in-tree developer convenience:

  • tox -edocs # recommended for Python projects
  • make docs # recommended for golang projects
  • npm run document # recommended for javascript projects

These will not be used by Zuul - they are there purely for local
developer convenience.

Rolling it Out


First, the governance patch:

https://review.openstack.org/508694

I've also written a patch for pbr to add support for finding a
doc/requirements.txt file, if one exists, and adding the contents to a
"docs" extra:

https://review.openstack.org/#/c/492620

That will let people do this in their tox.ini files:

[testenv:docs]
extras = docs
commands=
  python setup.py build_sphinx

And a WIP job for Zuul v3 written to provide a non-tox based build:

https://review.openstack.org/#/c/492709/

(that job can be improved a little if we land the pbr docs extras patch)


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
responded Oct 9, 2017 by Doug_Hellmann (87,520 points)   3 4 7
...