Repository navigation
Full build with PDF is taking more than 24h #169
Description
Today we're building 6 versions for 13 languages (that's 78 builds).
I'm doing some tests on my machine to get an idea:
- An HTML build takes 55 s.
- A full build (html + text + PDF A4 + PDF letter + epub + texinfo) takes around 21 min, composed of:
- 1 min for
make htmlcomposed of:- 50 s for sphinx-build
- 10 s of html archiving
- 40 s for
make text - 9 min for the A4 PDF build composed of:
- 1 min for
make latex PAPER=a4 - 8 min for
make all-pdf(inbuild/latex/) (less if run again, like 1 s if not removing PDFs, or 1 min after removing PDFs). Can be cut down to 6 min with-j 4.
- 1 min for
- 8 min 30 for the letter PDF build, roughly similar to A4 unsurprisingly
- 1 min for the epub build
- 1 min 30 s for the texinfo build
- 1 min for
So a complete rebuild should take ~27h (on my machine, the server may have a different CPU).
Activity
willingc commentedon Dec 12, 2023
@hugovk @JulienPalard What are the next steps needed for this?
JulienPalard commentedon Dec 12, 2023
I see the last build on
Last updated on Dec 11, 2023 (04:33 UTC), so the build is running, still slow though.hugovk commentedon Dec 12, 2023
It now says
Last updated on Dec 12, 2023 (08:00 UTC)on https://docs.python.org/3/If it took 27.5 hours, what language/versions has it been building to delay it?
There have been changes to 3.11-3.13 in the past day, but a week or more for 3.10 and older.
JulienPalard commentedon Dec 13, 2023
Since we stopped rebuilding when it's not needed, we're maintaining a file with some infos:
see file
it's not that readable but may help a bit.
I can paste some logs here:
view file
so yes, there's still almost 24h between any builds.
In 14 days we've seen only 45 build being skipped vs 449 rebuilds being needed:
is is partly due to some translation repos having a cron commiting every day (like transifex pulls) even if there's no new translations in the files.
But the root cause of slow build time is probably more #169 (comment) (latexmk running xelatex in a loop until the result stop changing).
@ewdurbin what "CPU" do we have on the machine? I don't see usefull info on
/proc/cpuinfo.willingc commentedon Dec 14, 2023
How important are the latex builds?
We may want to look at additional ways of optimizing the latex builds. https://blog.martisak.se/2023/10/01/compiling/
JulienPalard commentedon Dec 14, 2023
PDF builds are based on LaTeX builds.
When PDFs are unavailable we have user complaints (typically on docs@).
willingc commentedon Dec 14, 2023
Yeah, we definitely need PDFs.
I wonder if it makes sense to try building PDF with an alternate tool,
such as pyppeteer which is used by Jupyter book. https://jupyterbook.org/en/stable/advanced/pdf.html#build-a-pdf-from-your-book-htmlplaywright-python.willingc commentedon Dec 14, 2023
Shotscraper could be another approach: https://shot-scraper.datasette.io/en/stable/pdf.html
methane commentedon Dec 14, 2023
PDF shoud provide table of contents and index.
So ePub is better than html for converting into PDF.
But I don't know good tool to convert ePub to PDF.
Maybe, Calibre?
m-aciek commentedon Dec 14, 2023
What are the versions of latexmk and xetex that are being used on the build server? The latest versions of those packages are 4.81 (2023-11-06) for latexmk and 0.9999.3 (September 2013) for xetex.
JulienPalard commentedon Dec 14, 2023
ned-deily commentedon Apr 29, 2024
(One comment: IIRC, at least one reason we do full rebuilds of the docs and not just incremental builds was that there were problems when a documentation page was deleted by a change and then would not automatically get deleted on-line. I'm not sure if that still is an issue.)
hugovk commentedon Jul 15, 2024
I set up a repo to monitor the build times: https://github.com/hugovk/last-updated/actions/runs/9934564679/job/27439140210
And it's now taking 40-50 hours between version builds:
https://github.com/hugovk/last-updated/actions/runs/9934564679/job/27439140210#step:6:14
picnixz commentedon Jul 15, 2024
Are the builds continuously built? if so, I'm wondering what I did to make it slow x2... What I remember is that I needed to remove some pickled doctree cache because of a bug and I know that it might have cause a slowdown but I wouldn't have expected that much of a slowdown since it should only affect the
make latexcommand (possibly) ormake htmlcommand but not themake all-pdfcommand...I should check if we can speed up some macro expansions. I'll also try to build the docs without producing a PDF just to see how long it actually takes. Unfortunately, we cannot switch to pdflatex since the English document requires special unicode characters (for the 're' module... =/)
AA-Turner commentedon Jul 18, 2024
@jfbu sorry to ping, but this issue/thread may be of interest -- unsure of anything we can do within Sphinx(/CPython's docs) to speed up the LaTeX builds, but you may have some insights?
A
jfbu commentedon Jul 18, 2024
I have not read the full details. There may be tricks allowing to make only 2 passes in place of 3 sometimes with latex triggered from latexmk. But recently I worked on a lualatex build of a 1800 pages document with lots of pdf inclusions and highlighted code-blocks, output was more than 40M (due to fonts I think of the included pdf's) and each latex pass cost more than 3 minutes on an old 2Ghz single core computer. When I first read the title I thought that a single file took 24h which was really quite spectacular but of course here it is about building many many PDFs are built. I think pdflatex is somewhat faster than xelatex and definitely is faster than lualatex and in some cases with only few Unicode characters to support, it could be option. But that will not be a 10x gain, the most we can hope 2x from this. Usually Latexmk is very good at not making extra builds. But with LaTeX, any change to a section title will cause next time at least 2 passes.
jayaddison commentedon Jul 18, 2024
Please correct me if I'm mistaken, but I think that Sphinx multi-page HTML build (and similarly
epub,text, ... andtexinfo(?)) pipelines can skip a lot of work during builds when only a small number of the source documents have been modified.Can a LaTeX-to-PDF build process do the same? (in other words: the reprocessing passes after a section title change: do those have to load/parse the entire Python LaTeX docs?)
jfbu commentedon Jul 18, 2024
@jayaddison alas, I don't think so. We could imagine Sphinx using
\includemechanisms so that at least producing a modified LaTeX file could be localized. But when comes the time of LaTeX-to-PDF, LaTeX will have to redo 100% of its job. We could imagine also producing separate small PDFs but I know of no feasible automated way to merge these small PDFs in such a way that all hyperlinks will work, and that all automated enumeration of figures for example work the same as when doing the single PDF from a single source (whether or not that source is a huge one or a small main file with lots of\include's, which from LaTeX point of view is almost same).jayaddison commentedon Jul 18, 2024
Thanks; that all makes sense. To avoid the problem of "new large paragraph on page five shifts all content from pages 5-300" -- which could happen if the intermediate files were TeX and did not have some kind of hacky force-page-break mechanism -- I think that PDF-per-docname would be the approach (mirroring the
htmlbuild).So that brings the PDF-merge problem, and yep, it makes sense that of the few PDF merge tools available, many probably don't retain internal hyperlinks. I'm optimistic that
pypdf(coincidental that it's Python) might be able to do this though, based on py-pdf/pypdf#471 (actually about HTML links, but it seems that they take hyperlinks seriously).Enumerated figures could be more challenging - and perhaps now we're drifting off-topic (although I do think that if we could find a solution here, it might be applicable to this thread's problem statement). Should I start a discussion thread over at
sphinx-doc/sphinx?jfbu commentedon Jul 18, 2024
@jayaddison I read a bit more closely some of this thread and indeed it seems we started drifting a bit off-topic and
sphinx-doc/sphinxwould be a better place for a general issue about faster PDF builds with or without LaTeX. For example I have nil experience with the rinohtype builder not for lack of interest but purely for lack of time. For that very same reason I will not in immediate future be able to contribute much to any such general discussion.Coming back to this thread I remember a few years back I could set-up the build on RTD of our (Sphinx) own PDF docs in a way which let it make only 2 not 3 passes. Perhaps something like this would be possible here for the Python docs. Some years back I did build them locally from time to time. I must update my hardware first... I would love to discover the joys of multi-threaded builds 😃
Crazy ideas like using Python to build all LaTeX auxiliary files from a Python tool are too crazy: it is impossible to know page numbers in advance and page numbers are the things that change most in the aux files and trigger (via Latexmk usually quasi-optimal decisions) if a re-build is needed. Of course if you do
make cleanyou are in for the full show of many passes.jfbu commentedon Jul 21, 2024
Hello all, I suddenly realize I never pointed out that info in Sphinx docs:
If you launch via
make latexpdf LATEXMKOPTS="-xelatex"(or similar withall-pdftarget inside the latex build repertory aftermake latex) thexelatexbuilds will get faster. To what extent, I am not sure.Probably we at Sphinx should have made this default some time ago, perhaps about two years ago or so. The problem is that if you use that
-xelatexwith an older Latexmk probably it will refuse to build (although it may have gracefully ignored it but I can't test now; the test for all those years was on whether people read our docs 😃 ). I will make a PR at Sphinx.hugovk commentedon Aug 16, 2024
The new Ubuntu 24.04 server (python/psf-salt#421) has:
And it has 4 CPUs with 8 GB RAM:
hugovk commentedon Aug 16, 2024
Here's some observations and a suggestion.
Observations
Whilst checking the fix for #186, for the past couple of days I've been keeping an eye on the server build log and top processes (
tail -f /var/log/docsbuild/docsbuild.logandtop -i -u docsbuild).Most of the time is spent on the
make -C /srv/docsbuild/cpython/Doc ...commands. Here's logs for a single language/version:logs
When this runs Sphinx, we sometimes get four
sphinx-buildprocesses using up to 100% CPU each. This is good, we're making good use of the resources available. (Sometimes there's only 1-2 processes, depending on what part of the build Sphinx is at.)But most of the time it's running a latex command. These only use CPU, but at least generally use 100%:
Here's the time between the "Build start"/"Build done" pairs:
table of build times
We're spending a LOT of time churning out those PDFs: an hour or two per language/version.
Stepping back a moment, the cron entry is:
This script builds all languages and versions, and currently takes about 30 hours for a full batch.
The cron triggers at 7 minutes past the hour. If there's a build currently running, the new one just ends.
Suggestion: split HTML and PDF builds
I suggest we have two cron jobs. One that builds only HTML, the other that does everything else except HTML.
We'd allow these to run at the same time. A full HTML batch should be very quick. For example, the Ukrainian build is HTML only and takes 3 minutes. 13 languages x 3 minutes = 39 minutes.
We'd get docs HTML updates out the door quickly: ~30 minutes instead of ~30 hours! I also expect many of the next hourly triggers would not to need to rebuild anything, because there's less likely to be new docs changes after one hour compared to 30 hours.
I expect the other cron job that does non-HTML would still take about as long as now, as it's mostly churning out latex files on a single process. But we're more efficiently using the other CPUs for HTML.
I don't have download numbers for the PDFs etc, but I expect they're used much less than the HTML. And HTML is read live on the web right now, others are downloaded for reference and will (maybe) be re-downloaded at some later date.
In fact, I just spotted that #127 / python/psf-salt#236 said we used to build PDFs daily and the docs hourly. That PR was merged two years ago, shall we go back to this?
AA-Turner commentedon Aug 17, 2024
I would even suggest building PDFs every 36/48 hours. The main problem with building PDFs less often seems to have been new releases (especially .0) don't have PDF versions at release date, but this seems surmountable.
It seems that we could run multiple
latexmkprocesses for each build (ref: https://tex.stackexchange.com/a/699204) which might improve things -- I'll try and have a look if no one beats me to it.A
jfbu commentedon Aug 17, 2024
Sounds promising. I can not really test, but wouldn't
-joption ofmake(in the builddir) help here?I tried running directly latexmk as in https://tex.stackexchange.com/a/699204 but on my system it does not exit once job completes one has to hit RET; besides one should not run it directly so I tried adding the
&inside the Makefile rule, same issue, then I tried&withmake foo.pdf &,make foo2.pdf &again same issue, and finally-jappears to work:make -j all-pdfin the latex builddir.But here it should be I guess the with the largest
.texfiles first somake -j library.pdf c-api.pdf reference.pdf <here the others>(in the latex builddir or with-Coption) and not thismake -j all-pdfI tried.(I already admitted being on a single core system, so I can not vouch this will use multiple cores in real life).
As per my earlier comment about
-xelatexoption, see sphinx-doc/sphinx#12636. Best sure way to gain some time is to run the whole build in a TeXLive2019 environment not a TeXLive2024 one. Of course if that is compatible with extras from conf.py (fonts configuration?) which I have not checked here. Core Sphinx (even latest current version) produces exactly same result in a TL2019 based environment as in a TL2024 one.AA-Turner commentedon Aug 17, 2024
It turns out we have inadvertently been running the full
latexmkprocess twice for every build, since around 2010. Luckily, this is fixable.The LaTex build in Sphinx generates a makefile which we use to run latexmk and generate the actual PDFs from the
.texsources that Sphinx creates. In CPython'sDoc/Makefile, we call that makefile in a submake:(cd build/latex; make clean && make all-pdf && make FMT=pdf zip bz2).Considering the generated Makefile at
Doc/build/latex/Makefile, theall-pdftarget depends on$(ALLPDF), a list of every desired PDF file. Thezipandbz2targets also depend on$(ALLPDF). All three ofall-pdf,zip, andbz2are.PHONYtargets, which are evaluated every timemakeruns. Normally, this wouldn't present an issue, as thezipandbz2targets depend on$(ALLPDF)rather thanall-pdf, so make would know not to rebuild the PDF files. The issue becomes clear with the%.pdftarget. By depending on the.PHONYtargetFORCE_MAKE, PDF files are forcibly regenerated on both invocations ofmake, bypassing the inbuilt artefact caching.Resolving this is straightforwards, and I have a branch prepared to open a PR to CPython. EDIT: I have opened the PR at python/cpython#123113.
Sample timings (8 core CPU, via WSL):
-xelatexmake -j LATEXMKOPTS="-xelatex" all-pdf && make FMT=pdf zip bz2make -j LATEXMKOPTS="-xelatex" all-pdf && make FMT=pdf zip bz2-xelatexmake -j all-pdf && make FMT=pdf zip bz2make -j12 all-pdf && make FMT=pdf zip bz2I am proposing to use the final option, as whilst not the absolute fastest, it helps to mitigate scheduling problems, and the docs server only has 4 CPU cores, per Hugo's comment. It seems that
-xelatexdoesn't make a massive difference here.A
jfbu commentedon Aug 17, 2024
Good catch! The blame is on me at sphinx-doc/sphinx@409605d, but to my defence former Makefile did 5 (pdf)latex builds each time. Actually re-reading your post I am wondering if
latexmkreally triggers againxelatex. It should say something like this (this is an example on a dummy project)jfbu commentedon Aug 17, 2024
I think one has to make sure none of the 4 or 5 biggest files end up being handled by same core which may require a more complex layer to first list files as per decreasing filesizes of the
.texand feed them in that order tomakeas pdf targets.Yes Sphinx docs is too optimistic about this. It makes a difference when build documents does a ton of graphics inclusions.Sphinx docs does say it makes a difference in case of tons of graphics inclusions.hugovk commentedon Aug 25, 2024
With @AA-Turner's latexmk improvements (#169 (comment)) we're saving about a third of the time!
Here's a difference in the build times (not publish and so on) for a full set of 3.14 builds:
PS Here's the separate before and after times, and the script I used to get the times from the server build times.
before
after
calc-times.py
jfbu commentedon Aug 25, 2024
If you can build in an environment with a LaTeX distribution based upon TeXLive 2019 you will probably observe 30% or more additional gain. See this comment. I can not guarantee it completely because the comment compared builds on TL2019 and TL2024 and you use a Debian TL2023, so the gain might not be as much (I currently can not test on a TL2023). The PDFs will be exactly identical.
edit: I mean will look and print identical. Perhaps things are different regarding copy-paste from PDF and some other matters.
(I can not quantify at this time if the change from TL2023 to TL2024 induced the major part in slow-down of LaTeX kernel, or if it occurred earlier).
hugovk commentedon Aug 25, 2024
Another 30% or more would be very welcome! But I'm not too sure if we should downgrade to an unsupported version? (Or how to do it.)
jfbu commentedon Aug 25, 2024
A docker image is available for all TeXLive versions since 2014. The problem is that it contains the full thing which is much much more than what is strictly needed for Sphinx
make latexpdfeven leaving aside all sources and all documentation. Unfortunately I have personally no experience with this (having multiple TeXLive's on my computer already there), and to start with I have no idea if you can use such a Docker image and if its large size is an impediment (and I have other tasks now).There is a sphinx-latexpdf Docker image used for our CI, if I knew the recipe I could modify it to use TeXLive2019. And in the process try to use a much smaller part of full TeXLive hence construct a much smaller Docker image. But this is time-consuming task for which I don't have the immediate qualifications.
hugovk commentedon Sep 18, 2024
Update, we've cut the build times by around another third, by cutting about 15h per full build loop through dropping the letter PDF build (and keeping A4 PDF):
And by about an extra half an hour by re-using an HTTPS connection for purging CDN URLs:
AA-Turner commentedon Sep 23, 2024
It seems we're currently running around 10h 40m for each version -- Python 3.13 hasn't done a full rebuild in a while as the last commit was a fortnight ago (save around a dozen in the last half-hour!). This would be roughly 32 hours for a full rebuild of 3.12--3.14.
Timings:
(reason is 'translation' if an updated translation caused the rebuild rather than updated docs, and 'FULL BUILD' captures the end of a full rebuild loop. The loops that are 'empty' but around 30m long attempt a build of
fr/3.13and fail, see #187)A
hugovk commentedon Sep 23, 2024
This initially caught me by surprise but is expected; the 3.13 branch is locked for RC2, so there's not been much merged on the branch recently:
merwok commentedon Oct 1, 2024
This is the recipe: https://github.com/csotomon/sphinx-docker/blob/master/latexpdf/Dockerfile
AA-Turner commentedon Oct 1, 2024
That link is to a fork, though it's unclear -- the up-to-date dockerfile is at https://github.com/sphinx-doc/sphinx-docker-images/blob/master/latexpdf/Dockerfile
AA-Turner commentedon Oct 2, 2024
I suggest that we close this issue in favour of #209, now that python/docs-community#131 has been implemented. The non-HTML archive builds still take a while, but it's a manageable duration and we now have the luxury of adjusting the frequency of those builds to e.g. once every two days rather than daily.
Thank you to everyone involved for the help in reducing build times, we have made significant strides here in a challenging problem straddling multiple teams, projects, and repos.
A
hugovk commentedon Oct 2, 2024
Here's a summary of the current times, much improved:
python/docs-community#131 (comment)
Yes, let's close and continue in other issues as needed.
Thanks all!