Postgkeyll Documentation Integration Plan

Codex·GPT-6-astra·Maxwell-Rosen·yesterday·53min·1 Checkpoint·36 file changes·+1426/-65·457.2K tokens

We have a documentation website. https://github.com/gkeyllorg/gkyl-doc also found at /home/maxwell-rosen/Documents/gkyl-doc. Add a section to the website for postgkeyll. This⠂section should include documentation for this codebase. The ⠈ documentation should be generated from the postgkeyll⠈repo directly⡀so that there is one source of truth. The documentation to postgkyl repository, but it's hosted on that website. Use good⢀examples from the data inside⠐postgkeyll.⢀If you need ⢀to generate more interesting data for figures,⠄add it to the testing data folder, preffering to use the data generator.⡀Construct a plan for integrating the ⠠ documentation with postgkeyll. The examples on the⢀website should be clear and realistic use cases.

yesterday·2.9s

I’ll inspect both repositories and the website build, then develop an integration plan that keeps the documentation in postgkyl and uses its example data.

The website uses Sphinx, and its old Postgkyl pages are in the archive rather than the active navigation. Postgkyl already has executable tutorials and tests that check API/CLI documentation consistency, so the plan can build on those. I also found a build mismatch to address: the website’s Read the Docs configuration selects Python 3.8.

I recommend keeping the guides, reference generation, and figure scripts in postgkyl, with the website building from a pinned postgkyl revision. The initial tutorials can use the existing TCV distribution and R–Z data, plus generated profile comparisons. The plan will distinguish simulation output from synthetic examples and specify how examples and figures are checked before publication.

Created the integration plan.

It covers:

  • Postgkyl-owned documentation, published through a pinned website dependency.
  • API/CLI reference generated from existing code.
  • Executable tutorials and figures using repository data.
  • Website navigation, build changes, CI, and publication milestones.

Validated the existing examples and documentation checks: 30 tests passed. Website implementation remains the next step.

Implement Docs should be generated from postgkeyll on main, not a fixed revision. Upgrade the python version of the docs to python 3.12 Employ CI and github actions to ensure testing of the documentation

yesterday·5.6s

I’ll implement the documentation section using Postgkyl’s main branch, upgrade the website build to Python 3.12, and add GitHub Actions checks for the examples and documentation builds in both repositories.

The build will fetch Postgkyl main each time and generate the guides, command reference, figures, and example downloads from that checkout. I’m keeping explicit local-checkout support so pull requests can test documentation changes before they reach main.

The tutorials now run from repository data, including two generated fixtures for pressure recovery and energy-growth fitting. I also corrected the gyrokinetic example: it no longer assigns an arbitrary species mass or describes M1 as velocity. Next I’m checking the Sphinx output and wiring the Python 3.12 CI builds.

The standalone build exposed an API coverage gap, which I’ve fixed so the reference includes the fluid, kinetic, PKPM, and gyrokinetic diagnostic families. The full website also has existing Sphinx warnings from duplicate links, stale references, and unused theme options. I’m fixing those so CI can reject warnings across the whole site.

The host tests now verify that a second build picks up a new commit on main, while preserving a dirty local checkout. Figure review also caught an unlabeled distribution plot and a clipped temperature peak; both are fixed. I’m running the final documentation tests and full-site build on Python 3.12.

The checks are passing: 34 documentation/example tests, 36 architecture tests, and 3 host integration tests. The complete website also builds on Python 3.12 with warnings treated as errors. The local preview is ready; hosted publication will require merging Postgkyl’s changes first, then the website integration.

Implemented in both repositories.

  • Website builds from Postgkyl main using Python 3.12.
  • Added generated API/CLI references, tested tutorials, figures, and downloadable examples.
  • Added GitHub Actions for PR checks and daily upstream testing.
  • 73 tests passed; full website builds without warnings.

Open the local preview.

Not published yet. Merge Postgkyl first, then gkyl-doc. Configure READTHEDOCS_TOKEN in gkyl-doc for automatic scheduled publication.

Add some animations to the generated data and the examples using the animate function to show how to load multiple data files and make figures using collect and animate. Show some examples using plotly for 3D and 2D surface plotting, as well as pyvista. Provide some examples of how to get the grid and values of the data directly out of the loaded objects so that users can manipulate the data manually if they would like.

All the examples on the website should have both a script version to generate the figures as well as a CLI command that generates the same figure. Show that both version generate the same figures

Break the python API section of the docs into different pages for each of the different commands, similar to how the command reference is laid out.

yesterday·5.2s

I’ll add time-dependent fixtures and examples for collect/animate, Plotly, PyVista, and direct grid/value access. I’ll also pair each website figure with a tested Python script and CLI command, and split the Python API reference into individual pages.

I’ll compare the outputs from the two interfaces in the same build: pixels and timing for images/GIFs, and trace data plus layout for Plotly. A few existing figures use custom Python-only plotting steps, so I’ll adapt those to equivalent pgkyl pipelines while keeping the scientific examples clear.

The comparison caught a real CLI mistake in the growth example: fit appends its result, so f1 selects the fitted curve while f0 still selects the input. That’s corrected. The new pages will show both generated outputs and their comparison results, including interactive Plotly views.

All 19 figure pairs now match, including both GIF animations, the Plotly specifications, and both PyVista screenshots. I’m checking that the interactive embeds work in both website layouts and that every Python command has its own reference page with a CLI link.

The examples and API pages pass the documentation tests. I’m refining the PyVista figures for readability, then rebuilding the full website to verify the animations and interactive embeds in their hosted layout.

Both animations match frame for frame between Python and the CLI, including playback timing. The docs now also show matching Plotly data and layouts, PyVista screenshots, and a NumPy example that copies and modifies grid/value arrays. The full website build is finishing.

Implemented in both repositories:

  • Generated time-series data, collect examples, and two animations.
  • Plotly 2D surfaces/3D isosurfaces and PyVista examples.
  • Direct grid/value access and manual NumPy manipulation.
  • Paired Python scripts and CLI commands for all 19 published figures, with automated equivalence checks.
  • 171 individual Python API reference pages linked to CLI commands.
  • Python 3.12 documentation CI, with website builds sourcing Postgkyl main.

Validation: 41 documentation/example tests and 3 website integration tests passed. The complete website builds with warnings treated as errors.

Open the local preview. Changes are local; nothing has been published.

a46f0ff# Postgkyl documentation integration plan Keep Postgkyl documentation in this repository and publish it as a section of the existing Gkeyll Sphinx website. The website selects a Postgkyl revision; that revision supplies the prose, API and CLI reference, executable examples, input data, and figure generation. This is a proposed implementation plan; the website integration has not yet been implemented. Reviewed against Postgkyl `9355e5614914d0255b97c9729588cab2f500b8e4` and gkyl-doc `3c339bb46df318025a141d88fcf86fcad9133677`. ## Existing foundations - gkyl-doc builds `source/` with Sphinx and Furo. Its active toctree has no Postgkyl entry; the previous documentation lives in `source_archive/postgkyl/`. `source/conf.py` still references `postgkyl/_static`. - The website's `.readthedocs.yaml` selects Python 3.8, while Postgkyl declares Python >=3.10. Use Python 3.12 for the combined build, matching Postgkyl's principal CI environment, and validate the resolved dependencies together. - Public function signatures and Google-style docstrings already supply both Python help and the generated CLI. `tests/test_documentation.py` checks this contract. Extend that machinery rather than writing another command registry. - `examples/scripts/` and `examples/cli_tutorial.md` are already executable documentation, exercised by `tests/test_examples.py`. - `tests/test_data/` contains real gyrokinetic output. Synthetic fixtures are produced by `tests/generate_test_data.py` into the gitignored `tests/test_data/generated/` directory. ## Ownership and build contract | Material | Authoritative home | Website treatment | | --- | --- | --- | | Concepts and task guides | Postgkyl `docs/` | Stage during build | | Parameters, defaults, return values | Implementing function signatures and docstrings | Generate reference | | CLI names and options | Existing compiled CLI and command metadata | Generate reference | | Tutorial code | `examples/scripts/` and existing CLI tutorial | Include the tested source | | Synthetic data | `tests/generate_test_data.py` | Generate before running examples | | Simulation fixtures | `tests/test_data/` | Read from selected revision | | Figures | Outputs of example scripts | Generate into ignored build directories | | Theme, navigation, hosting | gkyl-doc | Add the section and build integration | | Published Postgkyl revision | gkyl-doc Git submodule entry | Use for code, docs, examples, and data together | Add Postgkyl as `external/postgkyl` in gkyl-doc, pinned by the Git submodule commit. Do not maintain a second revision file or install an unrelated PyPI release. Normal website builds use that pin; local integration checks may explicitly supply a different checkout. Validate that the imported package comes from the selected checkout. Add one build entry point in Postgkyl, proposed as `scripts/build_docs.py`, with explicit source and output paths. It prepares a self-contained Sphinx source subtree: guides, generated reference pages, literal example sources, figures, and downloadable inputs. gkyl-doc invokes it to populate the ignored `source/postgkyl/` directory before Sphinx reads the toctree. Clean only this owned output directory so removed pages cannot survive a rebuild. No generated pages or figures are committed to gkyl-doc. The same preparation step supports a standalone Postgkyl preview. A small Postgkyl-owned Sphinx extension supplies any shared directives and setup to both builds; the host retains its own theme. Keep documentation machinery outside `src/postgkyl/` so the library's import DAG does not change. Build sequence: 1. Check out the selected revision, including its test fixtures. 2. Install NumPy and build tools, then install that checkout with `--no-build-isolation` and a new `docs` extra. Declare Postgkyl documentation dependencies once in `pyproject.toml`. Build the native bridge through the existing installation flow and require `gpython.available()`. 3. Generate fixtures explicitly; a documentation build must not rely on pytest having run first. Execute the gallery headlessly with `MPLBACKEND=Agg` and `PGKYL_EXAMPLE_OUTPUT` pointing into the build directory. 4. Generate reference pages and stage sources/assets with relative links that work both standalone and under `postgkyl/`. 5. Build the combined website with its existing Sphinx invocation. Use Read the Docs submodule configuration and build hooks for these steps. Measure a cold native build on the hosted service during the integration milestone; local success alone does not establish its time or memory budget. Do not silently skip examples or substitute mock imports when the bridge fails. Caching can be added if needed, keyed by the source revision, Gkeyll revision, Python, NumPy ABI, and platform. ## Reader-facing section Add `postgkyl/index` to `source/index.rst`, with these destinations: - **Start here:** installation, verifying the bridge, obtaining example data, and one complete load → inspect → interpolate → lineout → plot → save task. - **Working with data:** grids and components; modal, nodal, and quadrature representations; explicit conversions; DG versus pointwise arithmetic; integration and averaging; groups and CLI pipelines. - **Tutorials:** scientific questions answered with executable examples and the corresponding CLI spelling where available. - **Reference:** loading, fluent API, operations, diagnostics grouped by model family, rendering, and the complete generated command inventory. - **Troubleshooting and contributing:** missing metadata, bridge builds, representation errors, headless rendering, and building these docs locally. Include existing installation instructions from the README using a Markdown include mechanism, or move them once and replace the README section with a link. Do the same for the CLI tutorial; do not hand-copy either. Use Sphinx autodoc plus Napoleon for Google-style docstrings. Use literal includes of named script regions so displayed code is the code that runs. Generate CLI pages from the actual Click commands produced by `postgkyl.cli.app`, preserving underscore names, defaults, boolean behavior, and command groups. Derive API inventory from public exports and public class members, including Python-only members that CLI discovery intentionally hides. Document canonical callables once and link their fluent/functional aliases. Derive diagnostic quantity listings from their existing registries. Describe capability rules using `value_form`, not the oversimplification that all native data reject NumPy or plotting. Explain that basis metadata is fixed at loading and that modal arithmetic has different semantics from arithmetic after interpolation. Fix inaccurate source docstrings at their implementation. ## Initial tutorial gallery | Reader's task | Existing source/data | Figure and teaching outcome | | --- | --- | --- | | Inspect a field and extract a lineout | `01_quickstart.py`; generated `2d_c2p_rot45_ms_p1.gkyl` | Field map plus lineout; identify the input as an analytic coordinate map, explain axes and components, and demonstrate save/reload | | Compare two profile calculations | `mirror_comparison.py`; generated `mirror_comparison_*_1d_ms_p1.gkyl` | Four labeled panels with linear/log scales; explain that these are synthetic profiles and that the script rescales temperatures | | Recover fluid primitives | `03_diagnostics_five_moment.py` | Density and pressure across a shock-tube initial discontinuity; show conserved-to-primitive conversion | | Inspect gyrokinetic density and velocity-space structure | `04_gyrokinetics.py`; `rt_gk_tcv_iwl*` moment, distribution, and geometry files | Density profile and fixed-mu distribution slice; explain Jacobian correction and velocity coordinates | | View density in physical R–Z coordinates | `05_gk_rz.py`; `rt_gk_tcv_nt_iwl_3x2v_p1-elc_M0_5.gkyl` and matching map | R–Z density map; show which geometry files are needed and when projection reuse helps | | Measure a growth interval | Existing CLI tutorial and generated `energy_dynvec.gkyl` | Energy history with fitted interval and residuals; distinguish early growth from saturation | | Choose DG or pointwise arithmetic | `02_arithmetic_and_numpy.py` | Computed comparison with an analytic check; explain why interpolating first changes the question | Each page states the scientific question, input provenance, prerequisites, copyable execution command, expected interpretation, and a small numerical check. Include the complete script and the minimum required input bundle for download, preserving relative filenames and geometry companions. Provide clear axis labels, units only when verified, captions, and alt text. Generate PNG or SVG for browser display; keep PDF downloads where useful. Before publishing, audit the existing tutorials' scientific explanations: the shock example is an initial condition, not an evolved shock solution; the mirror profiles are not a simulation convergence study; the dynvector uses logistic curves rather than measured instability data. Check the `M1` description and its illustrative `mass=2.0` against the quantity implementation and available provenance before presenting physical units or conclusions. No new simulation output is needed for the first gallery. Move the shock-tube initial-condition construction into the existing test-data generator so the tutorial loads a reproducible fixture. Add deterministic analytic fixtures there only if a tutorial needs a known growth rate or a clearer field example. Store generated files under `tests/test_data/generated/`, with correct basis, value-form, component, and time metadata. Test numerical invariants rather than pixel equality. Do not add random coefficient fields as physical examples. ## Implementation milestones and acceptance 1. **Documentation foundation in Postgkyl.** Add `docs/`, the docs extra, reference generation, and the standalone preparation/build command. Acceptance: a clean checkout builds navigable reference pages with actual signatures and links; changing a source default changes the reference. 2. **Executable gallery.** Reuse and improve existing examples, add the shock fixture, and generate figures and downloads. Extend existing example tests for scientific invariants and required outputs. Acceptance: each displayed command runs, inputs are downloadable, and every figure is inspected for readable labels, scaling, clipping, and truthful captions. 3. **Website integration.** Add the pinned submodule, navigation entry, ignored staging directory, shared extension setup, and RTD hooks. Update Python and remove or reconcile obsolete static-path configuration. Acceptance: the complete website builds from a clean clone and the new section appears in navigation, search, and a hosted preview. 4. **Continuous checks and maintenance.** Add a Postgkyl docs CI lane and a gkyl-doc integration lane. Require the bridge and run the existing documentation/example tests; build Postgkyl docs with warnings as errors. Check the host's existing warning baseline separately and fix integration issues without blanket warning suppression. Check internal links and downloads on every build; external link checks can run separately. Acceptance: a broken example, missing figure, invalid reference, or wrong imported revision fails the build. 5. **Publication and revision updates.** Merge the Postgkyl implementation, then select its tested commit in gkyl-doc. A subsequent code release updates that submodule pin and rebuilds the site; it does not require prose edits in gkyl-doc. Include the Postgkyl version/commit and source-edit links on its pages, pointing to Postgkyl rather than the generated host files. Retain archived material as historical and map old entry URLs to the new landing page where hosting supports redirects. Updating the pin is the initial maintenance workflow; cross-repository update automation can follow if needed. The current local check passed: `POSTGKYL_REQUIRE_GKEYLL=1 MPLBACKEND=Agg python -m pytest tests/test_examples.py tests/test_documentation.py -q` — 30 tests passed. This verifies the reusable baseline, not the proposed website build or the scientific accuracy of every existing caption. ## Integration references - [Sphinx autodoc and Napoleon](https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html) provide signature/docstring extraction and Google-style parsing. - [Sphinx literal includes](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-literalinclude) allow pages to display the executable source. - [Read the Docs configuration](https://docs.readthedocs.com/platform/stable/config-file/v2.html) supports selecting Python and checking out submodules. - [Read the Docs build hooks](https://docs.readthedocs.com/platform/stable/build-customization.html) provide the preparation steps before the Sphinx build. Implemented in both repositories: - Generated time-series data, collect examples, and two animations. - Plotly 2D surfaces/3D isosurfaces and PyVista examples. - Direct grid/value access and manual NumPy manipulation. - Paired Python scripts and CLI commands for all 19 published figures, with automated equivalence checks. - 171 individual Python API reference pages linked to CLI commands. - Python 3.12 documentation CI, with website builds sourcing Postgkyl main. Validation: 41 documentation/example tests and 3 website integration tests passed. The complete website builds with warnings treated as errors. Open the local preview (/home/maxwell-rosen/Documents/gkyl- doc/build/html/postgkyl/index.html). Changes are local; nothing has been published. Entire-Checkpoint: 01M262P5WD0PNFXEACD976METC+1426/-65