From ba79342ceb3a3701e2e0495f55c696d82a2f20b3 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Fri, 7 Mar 2025 12:13:39 +0000 Subject: [PATCH 01/12] wip --- docs/source/install.rst | 153 +++++++++++++++++++++------------------- 1 file changed, 80 insertions(+), 73 deletions(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index 8c2e944228..6f3615e87b 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -15,8 +15,9 @@ Supported systems ================= A :ref:`native installation` of Firedrake is officially -supported on Ubuntu and macOS though it should be installable on any Linux -distribution. Windows users are encouraged to use WSL_ or one of Firedrake's +supported on Ubuntu and ARM Macs (Intel Macs are no longer supported) though +it should be installable on any Linux distribution. Windows users are encouraged +to use WSL_ or one of Firedrake's :ref:`alternative installation mechanisms`. @@ -52,25 +53,16 @@ To simplify the installation process, Firedrake provides a utility script called $ curl -O https://raw.githubusercontent.com/firedrakeproject/firedrake/master/scripts/firedrake-configure +Unlike the now deprecated ``firedrake-install`` script, ``firedrake-configure`` +**does not install Firedrake for you**. It is simply a helper script that emits +the configuration options that Firedrake needs for the various steps needed +during installation. -.. _firedrake_archs: - -Prepared configurations -~~~~~~~~~~~~~~~~~~~~~~~ - -``firedrake-configure`` provides a number of different possible configurations -(termed 'ARCH's) that specify how PETSc is configured and which external -packages are built. The currently supported ARCHs are: - -* ``default``: the default installation, suitable for most users -* ``complex``: an installation where PETSc is configured using complex numbers - -The different configurations can be selected by passing the flag ``--arch`` to -``firedrake-configure``. For example:: - - $ python3 firedrake-configure --show-system-packages --arch complex - -If ``--arch`` is not specified then ``default`` is used. +To improve robustness, ``firedrake-configure`` is intentionally kept extremely +minimal and simple. This means that if you want to install Firedrake in a +non-standard way (for instance with a custom installation of PETSc, HDF5 or MPI) +then it is your responsibility to modify the output from ``firedrake-configure`` +as necessary. This is described in more detail in :ref:`customising`. .. _install_system_dependencies: @@ -100,12 +92,6 @@ which will install the following packages: If you do not have one of these systems then these dependencies will need to be installed manually. -.. note:: - Not all the system dependencies declared by ``firedrake-configure`` have to - be installed at this stage. Some (e.g. HDF5, hwloc) can also be installed - from source by PETSc during the ``configure`` :ref:`step` by - passing additional flags (e.g. ``--download-hdf5``, ``--download-hwloc``). - .. _install_petsc: @@ -154,24 +140,6 @@ For the ``default`` ARCH, running ``firedrake-configure`` with :language: text -Customising the PETSc installation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Since ``firedrake-configure`` only outputs a string of options it is straightforward -to make changes to the options passed to PETSc ``configure``. You can either: - -* Append additional options when ``configure`` is invoked. For example, to compile - PETSc with 64 bit indices you should run:: - - $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --with-64-bit-indices - -* Write the output of ``firedrake-configure`` to a file than can be modified:: - - $ python3 ../firedrake-configure --show-petsc-configure-options > my_configure_options.txt - - $ ./configure $(cat my_configure_options.txt) - - .. _install_firedrake: Installing Firedrake @@ -193,7 +161,7 @@ install Firedrake. To do this perform the following steps: $ export $(python3 firedrake-configure --show-env) - At a minimum this will set the following variables: + Which at a minimum will set the following variables: .. code-block:: text @@ -236,17 +204,6 @@ install Firedrake. To do this perform the following steps: pip cache remove petsc4py - Equivalent commands may also be necessary for mpi4py and h5py if you are - changing the MPI and/or HDF5 libraries in use. - -.. note:: - If you are using an MPI installed into a nonstandard location it may be - necessary to set some additional environment variables before installation including: - - * ``MPICC`` to the location of ``mpicc`` - * ``MPI_HOME`` to the base directory of the MPI installation (e.g. ``/usr`` - or ``/opt/mpich``) - .. _firedrake_check: @@ -268,10 +225,35 @@ Firedrake with its optional test dependencies by specifying the ``[test]`` dependency group as shown :ref:`above`. +Updating Firedrake +------------------ + +Updating Firedrake involves following the same steps as above when +:ref:`installing Firedrake`. First, use ``firedrake-configure`` +to set the right environment variables and then run:: + + $ pip install --upgrade git+https://github.com/firedrakeproject/firedrake.git + +Updating PETSc +~~~~~~~~~~~~~~ + +To update PETSc you simply need to run:: + + $ cd petsc + $ git pull + $ make + +This will only recompile PETSc's source code, not that of the external +packages, and so should be relatively quick. If your PETSc is sufficiently +out-of-date you may also need to rebuild the external packages by running:: + + $ make reconfigure + + .. _dev_install: Developer install -~~~~~~~~~~~~~~~~~ +----------------- By default Firedrake is installed just like any other Python package into your environment. If you want to be able to edit Firedrake itself then @@ -294,29 +276,54 @@ should run:: whatever packages you just installed. -Updating Firedrake ------------------- +.. _customising: -Updating Firedrake involves following the same steps as above when -:ref:`installing Firedrake`. First, use ``firedrake-configure`` -to set the right environment variables and then run:: +Customising Firedrake +===================== - $ pip install --upgrade git+https://github.com/firedrakeproject/firedrake.git +# key point here: can tweak configure! -Updating PETSc -~~~~~~~~~~~~~~ +Since ``firedrake-configure`` only outputs a string of options it is straightforward +to make changes to the options passed to PETSc ``configure``. You can either: -To update PETSc you simply need to run:: +* Append additional options when ``configure`` is invoked. For example, to compile + PETSc with 64 bit indices you should run:: - $ cd petsc - $ git pull - $ make + $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --with-64-bit-indices -This will only recompile PETSc's source code, not that of the external -packages, and so should be relatively quick. If your PETSc is sufficiently -out-of-date you may also need to rebuild the external packages by running:: +* Write the output of ``firedrake-configure`` to a file than can be modified:: - $ make reconfigure + $ python3 ../firedrake-configure --show-petsc-configure-options > my_configure_options.txt + + $ ./configure $(cat my_configure_options.txt) + +.. _firedrake_archs: + +Prepared configurations +----------------------- + +``firedrake-configure`` provides a number of different possible configurations +(termed 'ARCHs') that specify how PETSc is configured and which external +packages are built. The currently supported ARCHs are: + +* ``default``: the default installation, suitable for most users +* ``complex``: an installation where PETSc is configured using complex numbers + +The different configurations can be selected by passing the flag ``--arch`` to +``firedrake-configure``. For example:: + + $ python3 firedrake-configure --show-petsc-configure-options --arch complex + + +Install optional dependencies +----------------------------- + +slepc +vtk +torch +netgen + +from pyproject? .. _alternative_install: From c6ee1caa54db2d10c9add831156a95ce35df1e3f Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Mon, 17 Mar 2025 16:00:20 +0000 Subject: [PATCH 02/12] tweaks --- .../qgbasinmodes.py.rst | 6 +- demos/netgen/netgen_mesh.py.rst | 2 +- docs/source/checkpointing.rst | 6 +- docs/source/install.rst | 93 ++++++++++++------- pyproject.toml | 7 ++ 5 files changed, 73 insertions(+), 41 deletions(-) diff --git a/demos/eigenvalues_QG_basinmodes/qgbasinmodes.py.rst b/demos/eigenvalues_QG_basinmodes/qgbasinmodes.py.rst index 4e957affc9..33c5b02d4a 100644 --- a/demos/eigenvalues_QG_basinmodes/qgbasinmodes.py.rst +++ b/demos/eigenvalues_QG_basinmodes/qgbasinmodes.py.rst @@ -18,10 +18,8 @@ dissipative forces. Unlike the other demo that integrated the equations forward in time, in this problem it is necessary to compute the eigenvalues and eigenfunctions for a particular differential operator. -This demo requires SLEPc and slepc4py to be installed. This is most easily -achieved by providing the optional `--slepc` flag to either `firedrake-install` -(for a new installation), or `firedrake-update` (to add SLEPc to an existing -installation). +This demo requires SLEPc and slepc4py to be installed. For instructions on how +to install them please follow `these instructions `_. Governing PDE diff --git a/demos/netgen/netgen_mesh.py.rst b/demos/netgen/netgen_mesh.py.rst index 688f150aa1..3d3235aa96 100755 --- a/demos/netgen/netgen_mesh.py.rst +++ b/demos/netgen/netgen_mesh.py.rst @@ -11,7 +11,7 @@ Finally we will show how to use mesh refinement features included in Netgen to c Installing Netgen ------------------ -This demo requires the NGSolve/Netgen suite to be installed. This is most easily achieved by providing the optional `--netgen` flag to either `firedrake-install` (for a new installation), or `firedrake-update` (to add the NGSolve/Netgen suite to an existing installation). :: +This demo requires the NGSolve/Netgen suite to be installed. This can be done by running ``pip install ngsPETSc``. :: from firedrake import * try: diff --git a/docs/source/checkpointing.rst b/docs/source/checkpointing.rst index 3a315c69b4..b41790b8d1 100644 --- a/docs/source/checkpointing.rst +++ b/docs/source/checkpointing.rst @@ -504,9 +504,9 @@ Firedrake uses the PETSc_ HDF5 Viewer_ object to write and read state. As such, writing data is collective across processes. h5py_ is used for attribute manipulation. To this end, h5py_ *must* be linked against the same version of the HDF5 library that PETSc was built -with. The ``firedrake-install`` script automates this, however, if -you build PETSc manually, you will need to ensure that h5py_ is linked -correctly following the instructions for custom installation here_. +with. If you have a custom installation of Firedrake then it may be +necessary to manually ensure that h5py is linked correctly. +Instructions for how to do this can be found here_. .. warning:: diff --git a/docs/source/install.rst b/docs/source/install.rst index 6f3615e87b..f24b9e7950 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -177,6 +177,14 @@ install Firedrake. To do this perform the following steps: you have exactly followed the instructions up to this point this should already be the case. +#. Remove possibly cached versions of petsc4py:: + + $ pip cache remove petsc4py + + This is necessary because cached petsc4py wheels will be linked against the + wrong PETSc. + + #. Install Firedrake:: $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test]" @@ -195,15 +203,6 @@ install Firedrake. To do this perform the following steps: Firedrake and its components. To install Firedrake in editable mode you should follow the instructions :ref:`below`. -.. note:: - During the installation Firedrake will compile and install petsc4py_. If - you have previously installed petsc4py on your computer with a different - PETSc then ``pip`` will erroneously reuse the existing petsc4py which is - linked against the wrong library. To avoid this you need to run the - command:: - - pip cache remove petsc4py - .. _firedrake_check: @@ -239,11 +238,11 @@ Updating PETSc To update PETSc you simply need to run:: - $ cd petsc + $ cd /path/to/petsc $ git pull $ make -This will only recompile PETSc's source code, not that of the external +Note that this will only recompile PETSc's source code, not that of the external packages, and so should be relatively quick. If your PETSc is sufficiently out-of-date you may also need to rebuild the external packages by running:: @@ -281,21 +280,6 @@ should run:: Customising Firedrake ===================== -# key point here: can tweak configure! - -Since ``firedrake-configure`` only outputs a string of options it is straightforward -to make changes to the options passed to PETSc ``configure``. You can either: - -* Append additional options when ``configure`` is invoked. For example, to compile - PETSc with 64 bit indices you should run:: - - $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --with-64-bit-indices - -* Write the output of ``firedrake-configure`` to a file than can be modified:: - - $ python3 ../firedrake-configure --show-petsc-configure-options > my_configure_options.txt - - $ ./configure $(cat my_configure_options.txt) .. _firedrake_archs: @@ -315,15 +299,58 @@ The different configurations can be selected by passing the flag ``--arch`` to $ python3 firedrake-configure --show-petsc-configure-options --arch complex -Install optional dependencies ------------------------------ +Optional dependencies +--------------------- + +PETSc with 64-bit indices +~~~~~~~~~~~~~~~~~~~~~~~~~ + +To build PETSc with 64-bit indices you need to append ``--with-64-bit-indices`` +to the set of options when PETSc is configured. For example this could be:: + + $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --with-64-bit-indices + +SLEPc +~~~~~ + +To install Firedrake with SLEPc support you must: + +#. Clone and install SLEPc (note that ``PETSC_DIR`` and ``PETSC_ARCH`` must be set):: + + $ git clone --depth 1 https://github.com/firedrakeproject/slepc.git + $ cd slepc + $ ./configure + $ make SLEPC_DIR=$PWD + +#. Set ``SLEPC_DIR``:: + + $ export SLEPC_DIR=$PWD + +# Install slepc4py:: + + $ pip install slepc4py + -slepc -vtk -torch -netgen +vtk? +torch? +netgen? -from pyproject? + +Advanced customisation +---------------------- + +Since ``firedrake-configure`` only outputs a string of options it is straightforward +to customise all aspects of the installation process. You can either: + +* Append additional options when ``configure`` is invoked. For example:: + + $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --download-exotic-package + +* Write the output of ``firedrake-configure`` to a file than can be modified. For example:: + + $ python3 ../firedrake-configure --show-petsc-configure-options > my_configure_options.txt + + $ ./configure $(cat my_configure_options.txt) .. _alternative_install: diff --git a/pyproject.toml b/pyproject.toml index dfbc156e8c..c2871ce16a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -64,6 +64,13 @@ spydump = "pyop2.scripts.spydump:main" [project.optional-dependencies] +# TODO: Is is necessary to specify these? Or is it sufficient to complain +# when the code in run to prompt the installation? +netgen = ["ngsPETSc"] +slepc = ["slepc4py"] +torch = ["torch"] +vtk = ["vtk"] + # Dependencies needed to run firedrake-check test = [ "mpi-pytest", From adde4a7710cecfd9feed655ebf746f98b110633d Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Mon, 17 Mar 2025 16:17:31 +0000 Subject: [PATCH 03/12] tweaks --- docs/source/install.rst | 35 ++++++++++++++++++++++++++++++----- pyproject.toml | 4 +--- 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index f24b9e7950..07d1049af2 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -326,14 +326,39 @@ To install Firedrake with SLEPc support you must: $ export SLEPC_DIR=$PWD -# Install slepc4py:: +# Install Firedrake with SLEPc support:: - $ pip install slepc4py + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,slepc]" +VTK +~~~ -vtk? -torch? -netgen? +To install Firedrake with VTK, it should be installed using the ``vtk`` optional +dependency. For example:: + + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,vtk]" + + +PyTorch +~~~~~~~ + +To install Firedrake with `PyTorch `_, it should be installed +using the ``torch`` optional dependency. For example:: + + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,torch]" --extra-index-url https://download.pytorch.org/whl/cpu + +Observe that, in addition to specifying ``torch``, an additional +argument (``--extra-index-url``) is needed. More information on installing +PyTorch can be found `here `__. + + +Netgen +~~~~~~ + +To install Firedrake with `Netgen `_ support, it should be +installed with the ``netgen`` optional dependency. For example:: + + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,netgen]" Advanced customisation diff --git a/pyproject.toml b/pyproject.toml index c2871ce16a..4d081ca4a2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -64,11 +64,9 @@ spydump = "pyop2.scripts.spydump:main" [project.optional-dependencies] -# TODO: Is is necessary to specify these? Or is it sufficient to complain -# when the code in run to prompt the installation? netgen = ["ngsPETSc"] slepc = ["slepc4py"] -torch = ["torch"] +torch = ["torch"] # requires passing '--extra-index-url' to work vtk = ["vtk"] # Dependencies needed to run firedrake-check From f9534ed04f22a0ea87e49077f62505795b195166 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 19 Mar 2025 14:47:38 +0000 Subject: [PATCH 04/12] Expand Docker section --- docs/source/install.rst | 64 +++++++++++++++++++++++++++++++++++------ 1 file changed, 56 insertions(+), 8 deletions(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index 07d1049af2..efec93c429 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -167,9 +167,6 @@ install Firedrake. To do this perform the following steps: CC=mpicc CXX=mpicxx PETSC_DIR=/path/to/petsc PETSC_ARCH=arch-firedrake-{default,complex} HDF5_MPI=ON - .. note:: - If you are installing - .. note:: This command will only work if you have the right starting directory. Specifically it is assumed that PETSc was cloned into a *subdirectory @@ -257,7 +254,9 @@ Developer install By default Firedrake is installed just like any other Python package into your environment. If you want to be able to edit Firedrake itself then an *editable* installation is needed. To install Firedrake in editable -mode you should run:: +mode you should follow the same +:ref:`steps as for a non-editable install` but replace the +final ``pip install`` command with:: $ git clone https://github.com/firedrakeproject/firedrake.git $ pip install --no-binary h5py --editable './firedrake[dev]' @@ -326,7 +325,7 @@ To install Firedrake with SLEPc support you must: $ export SLEPC_DIR=$PWD -# Install Firedrake with SLEPc support:: +#. Install Firedrake with SLEPc support:: $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,slepc]" @@ -383,10 +382,59 @@ to customise all aspects of the installation process. You can either: Alternative installation methods ================================ -As well as being installable through ``pip``, Firedrake also provides -`Docker containers `_ and -Jupyter notebooks running on :doc:`Google Colab`. +If for some reason you are unable to install Firedrake natively using pip, +Firedrake has a number of alternative mechanisms that you can use to obtain +an environment to run your Firedrake code. + +Docker +------ + +Firedrake provides a number of different +`Docker `_ images that can be found +`here `__. The main images best +suited for users are: + +* `firedrake-vanilla-default `__: a complete Firedrake installation with ARCH ``default`` +* `firedrake-vanilla-complex `__: a complete Firedrake installation with ARCH ``complex`` +* `firedrake `__: the firedrake-vanilla-default image with extra downstream packages installed + +To use one of the containers you should run:: + + $ docker pull firedrakeproject/:latest + +to download the most recent image (replacing ```` with the desired +image). Then you can run:: + + $ docker run -it firedrakeproject/:latest + +to start and enter a container. + +.. note:: + + The 'full-fat' ``firedrakeproject/firedrake`` image only exists for x86 + architectures because some external packages do not provide ARM wheels. + If you are using an ARM Mac (i.e. M1, M2, etc) then you are encouraged to + use the ``firedrakeproject/firedrake-vanilla-default`` or + ``firedrakeproject/firedrake-vanilla-complex`` images instead. + +It is possible to use `Microsoft VSCode `__ +inside a running container. Instructions for how to do this may be found +`here `__. + +.. warning:: + + The Docker daemon runs with superuser privileges and so has the potential to + damage your system, in particular if volumes are mounted between the container + and host. We therefore strongly advise you to take care when using Docker. + More information can be found + `here `__. + +Google Colab +------------ +Firedrake can also be used inside the brower using Jupyter notebooks and +`Google Colab `_. For more information +please see :doc:`here`. .. _getting_help: From 603b541a01a84893f505d336efec7aac91cd9e25 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 9 Apr 2025 07:49:06 +0100 Subject: [PATCH 05/12] Various improvements --- .github/workflows/build.yml | 2 +- docker/Dockerfile.vanilla | 2 +- docs/source/install.rst | 116 +++++++++++++++++++++++------------- pyproject.toml | 76 ++++++++++++----------- scripts/firedrake-configure | 2 +- 5 files changed, 120 insertions(+), 78 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index bf22d12bfb..2870dc9142 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -65,7 +65,7 @@ jobs: - name: Install PETSc run: | - git clone --depth 1 --branch $(python3 ./firedrake-repo/scripts/firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git + git clone --depth 1 --branch v$(python3 ./firedrake-repo/scripts/firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git cd petsc python3 ../firedrake-repo/scripts/firedrake-configure \ --arch ${{ matrix.arch }} --show-petsc-configure-options | \ diff --git a/docker/Dockerfile.vanilla b/docker/Dockerfile.vanilla index 09d30b6820..9272cf0b96 100644 --- a/docker/Dockerfile.vanilla +++ b/docker/Dockerfile.vanilla @@ -40,7 +40,7 @@ ENV OMPI_ALLOW_RUN_AS_ROOT=1 OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 # Install PETSc. We set the compiler optimisation flags manually here to # remove the default of '-march=native' which is not suitable for Docker images. # We use 'sed' to make sure that the options are appended. -RUN git clone --depth 1 --branch $(python3 /opt/firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git /opt/petsc \ +RUN git clone --depth 1 --branch v$(python3 /opt/firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git /opt/petsc \ && cd /opt/petsc \ && python3 /opt/firedrake-configure --arch $ARCH --show-petsc-configure-options | \ sed "s/$/ --COPTFLAGS='-O3 -mtune=generic' --CXXOPTFLAGS='-O3 -mtune=generic' --FOPTFLAGS='-O3 -mtune=generic'/" | \ diff --git a/docs/source/install.rst b/docs/source/install.rst index 927ce42c1b..82d76c5e2f 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -104,7 +104,7 @@ do the following steps: #. Clone the PETSc repository, checking out the right version:: - $ git clone --depth 1 --branch $(python3 firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git + $ git clone --depth 1 --branch v$(python3 firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git $ cd petsc #. Run PETSc ``configure``, passing in the flags generated by ``firedrake-configure``:: @@ -133,7 +133,7 @@ to be downloaded and compiled from source:: $ python3 ../firedrake-configure --no-package-manager --show-petsc-configure-options | xargs -L1 ./configure -For the ``default`` ARCH, running ``firedrake-configure`` with +For the default build, running ``firedrake-configure`` with ``--no-package-manager`` will produce the flags: .. literalinclude:: petsc_configure_options.txt @@ -174,21 +174,32 @@ install Firedrake. To do this perform the following steps: you have exactly followed the instructions up to this point this should already be the case. -#. Remove possibly cached versions of petsc4py:: +#. Install petsc4py_:: $ pip cache remove petsc4py + $ pip install petsc4py==$(python3 firedrake-configure --show-petsc-version) - This is necessary because cached petsc4py wheels will be linked against the - wrong PETSc. + .. note:: + A common issue encountered with this step is the installation will fail with + an error message like: + + .. code-block:: text + + FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-build-env-9j45p2su/normal/lib/python3.11/site-packages/petsc/lib/petsc/conf/petscvariables' + This usually means that the environment variable ``PETSC_DIR`` is not set + correctly. You can check this by making sure that you can run the following + command without error:: + + $ ls $PETSC_DIR #. Install Firedrake:: - $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test]" + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check]" .. note:: - Though not strictly necessary to install Firedrake's optional test - dependencies with ``[test]`` it is recommended because it allows you + Though not strictly necessary to install Firedrake's optional + dependencies with ``[check]`` it is recommended because it allows you to check that the install was successful (see :ref:`below`). @@ -217,7 +228,7 @@ minute or less. If they fail to run for any reason, please see the section below on how to :ref:`get help`. Note that for you to be able to run the tests you need to have installed -Firedrake with its optional test dependencies by specifying the ``[test]`` +Firedrake with its optional test dependencies by specifying the ``[check]`` dependency group as shown :ref:`above`. @@ -258,20 +269,22 @@ mode you should follow the same :ref:`steps as for a non-editable install` but replace the final ``pip install`` command with:: - $ git clone https://github.com/firedrakeproject/firedrake.git + $ git clone $ pip install --no-binary h5py --editable './firedrake[dev]' +where ```` is ``https://github.com/firedrakeproject/firedrake.git`` +or ``git@github.com:firedrakeproject/firedrake.git`` as preferred. + The same process applies for Firedrake's dependencies. For example, to install `FIAT `_ in editable mode you should run:: - $ git clone https://github.com/firedrakeproject/fiat.git + $ git clone $ pip install --editable ./fiat -.. note:: - Editable versions of Firedrake's dependencies should be installed *after* - Firedrake is installed. Otherwise installing Firedrake will overwrite - whatever packages you just installed. +Note that editable versions of Firedrake's dependencies should be installed *after* +Firedrake is installed. Otherwise installing Firedrake will overwrite +whatever packages you just installed. .. _customising: @@ -301,33 +314,25 @@ The different configurations can be selected by passing the flag ``--arch`` to Optional dependencies --------------------- -PETSc with 64-bit indices -~~~~~~~~~~~~~~~~~~~~~~~~~ - -To build PETSc with 64-bit indices you need to append ``--with-64-bit-indices`` -to the set of options when PETSc is configured. For example this could be:: - - $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --with-64-bit-indices - SLEPc ~~~~~ -To install Firedrake with SLEPc support you must: +To install Firedrake with SLEPc support you should: -#. Clone and install SLEPc (note that ``PETSC_DIR`` and ``PETSC_ARCH`` must be set):: +#. Pass ``--download-slepc`` when running PETSc ``configure`` (see :ref:`Installing PETSc`):: - $ git clone --depth 1 https://github.com/firedrakeproject/slepc.git - $ cd slepc - $ ./configure - $ make SLEPC_DIR=$PWD + $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --download-slepc #. Set ``SLEPC_DIR``:: - $ export SLEPC_DIR=$PWD + $ export SLEPC_DIR=$PETSC_DIR/$PETSC_ARCH -#. Install Firedrake with SLEPc support:: +#. Install slepc4py:: - $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,slepc]" + $ pip cache remove slepc4py + $ pip install slepc4py==$(python3 firedrake-configure --show-petsc-version) + +#. Continue with the installation as normal VTK ~~~ @@ -335,7 +340,7 @@ VTK To install Firedrake with VTK, it should be installed using the ``vtk`` optional dependency. For example:: - $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,vtk]" + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check,vtk]" PyTorch @@ -344,37 +349,66 @@ PyTorch To install Firedrake with `PyTorch `_, it should be installed using the ``torch`` optional dependency. For example:: - $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,torch]" --extra-index-url https://download.pytorch.org/whl/cpu + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check,torch]" --extra-index-url https://download.pytorch.org/whl/cpu Observe that, in addition to specifying ``torch``, an additional argument (``--extra-index-url``) is needed. More information on installing PyTorch can be found `here `__. +JAX +~~~ + +To install Firedrake with JAX, it should be installed using the ``jax`` optional +dependency. For example:: + + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check,jax]" + + Netgen ~~~~~~ To install Firedrake with `Netgen `_ support, it should be installed with the ``netgen`` optional dependency. For example:: - $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[test,netgen]" + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check,netgen]" -Advanced customisation ----------------------- +Customising PETSc +----------------- Since ``firedrake-configure`` only outputs a string of options it is straightforward -to customise all aspects of the installation process. You can either: +to customise the options that are passed to PETSc ``configure``. You can either: -* Append additional options when ``configure`` is invoked. For example:: +* Append additional options when ``configure`` is invoked. For example, to + build PETSc with support for 64-bit indices you should run:: - $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --download-exotic-package + $ python3 ../firedrake-configure --show-petsc-configure-options | xargs -L1 ./configure --with-64-bit-indices * Write the output of ``firedrake-configure`` to a file than can be modified. For example:: $ python3 ../firedrake-configure --show-petsc-configure-options > my_configure_options.txt - $ ./configure $(cat my_configure_options.txt) + $ cat my_configure_options.txt | xargs -L1 ./configure + +.. note:: + If additional options are passed to ``configure`` then care must be taken when + using externally-installed system packages (i.e. ``--with-package=...`` or + ``--with-package-{include,lib}=...`` are in the ``configure`` options) as they + may no longer be suitable for the new configuration. It is your responsibility + to either ensure that the configuration is suitable, or replace the + ``configure`` option with ``--download-package`` so that PETSc will download + and install the right thing. + + +Reconfiguring an existing PETSc +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If rebuilding an existing PETSc installation, rather than removing everything and +starting from scratch, it can be useful to modify and run the ``reconfigure-ARCH.py`` +Python script that PETSc generates. This can be found in +``$PETSC_DIR/$PETSC_ARCH/lib/petsc/conf``. Other example scripts can be found in +``$PETSC_DIR/config/examples`` directory. .. _alternative_install: diff --git a/pyproject.toml b/pyproject.toml index 061993afb2..40e14a13e0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -22,7 +22,7 @@ dependencies = [ "libsupermesh", # NOTE: If changing the PETSc/SLEPc version then firedrake-configure also needs # changing (as well as other references to petsc4py and slepc4py here) - "petsc4py==3.22.2", + "petsc4py>=3.22.2,<3.23", "numpy", "packaging", "pkgconfig", @@ -66,16 +66,47 @@ spydump = "pyop2.scripts.spydump:main" [project.optional-dependencies] -netgen = ["ngsPETSc"] -slepc = ["slepc4py"] -torch = ["torch"] # requires passing '--extra-index-url' to work -vtk = ["vtk"] - -# Dependencies needed to run firedrake-check -test = [ +check = [ "mpi-pytest", "pytest", ] +dev = [ # build dependencies that are needed to run 'make' + "Cython", + "mpi-pytest", + "pybind11", + "pytest", + "setuptools", +] +docs = [ + "bibtexparser", + "matplotlib", # needed to resolve API + "numpydoc", + "pylit", + "sphinx<8.2.0", # https://github.com/firedrakeproject/firedrake/issues/4059 + "sphinx-autobuild", + "sphinx-reredirects", + "sphinxcontrib-bibtex", + "sphinxcontrib-jquery", + "sphinxcontrib-svg2pdfconverter", + "sphinxcontrib-youtube", + "vtk", # needed to resolve API +] +jax = [ + "jax", +] +netgen = [ + "ngsPETSc", +] +slepc = [ + "slepc4py>=3.22.2<3.23", +] +torch = [ # requires passing '--extra-index-url' to work + "torch", +] +vtk = [ + "vtk", +] + # Dependencies needed to run the full test suite ci = [ "ipympl", # needed for notebook testing @@ -89,7 +120,7 @@ ci = [ "pytest-split", # needed for firedrake-run-split-tests "pytest-timeout", "pytest-xdist", - "slepc4py==3.22.2", + "slepc4py>=3.22.2,<3.23", "torch", # requires passing '--extra-index-url' to work "vtk", ] @@ -103,30 +134,7 @@ docker = [ # Used in firedrake-vanilla container "pytest-split", # needed for firedrake-run-split-tests "pytest-timeout", "pytest-xdist", - "slepc4py==3.22.2", -] -# Dependencies needed to build the docs -docs = [ - "bibtexparser", - "matplotlib", # needed to resolve API - "numpydoc", - "pylit", - "sphinx<8.2.0", # https://github.com/firedrakeproject/firedrake/issues/4059 - "sphinx-autobuild", - "sphinx-reredirects", - "sphinxcontrib-bibtex", - "sphinxcontrib-jquery", - "sphinxcontrib-svg2pdfconverter", - "sphinxcontrib-youtube", - "vtk", # needed to resolve API -] -# Developer dependencies. In particular the build dependencies that are needed to run 'make'. -dev = [ - "Cython", - "mpi-pytest", - "pybind11", - "pytest", - "setuptools", + "slepc4py>=3.22.2,<3.23", ] [build-system] @@ -139,7 +147,7 @@ requires = [ "pkgconfig", "pybind11", "setuptools>61.2", - "petsc4py==3.22.2", + "petsc4py>=3.22.2,<3.23", "rtree>=1.2", ] build-backend = "setuptools.build_meta" diff --git a/scripts/firedrake-configure b/scripts/firedrake-configure index 6eecdaa6d4..05363ad77f 100755 --- a/scripts/firedrake-configure +++ b/scripts/firedrake-configure @@ -41,7 +41,7 @@ ARCH_COMPLEX = FiredrakeArch.COMPLEX # NOTE: When updating this variable corresponding changes must be made inside # pyproject.toml -SUPPORTED_PETSC_VERSION = "v3.22.2" +SUPPORTED_PETSC_VERSION = "3.22.2" def main(): From 05e84bcd309096cdb2ea1a5d2536094762f85f03 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 9 Apr 2025 08:05:30 +0100 Subject: [PATCH 06/12] Add section about linux ARM VTK --- docs/source/install.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/source/install.rst b/docs/source/install.rst index 82d76c5e2f..fc962be131 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -342,6 +342,11 @@ dependency. For example:: $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check,vtk]" +At present VTK wheels are not available for ARM Linux machines. Depending on your +Python version you may be able to work around this by downloading and pip installing +the appropriate ``.whl`` file from +`here `__. + PyTorch ~~~~~~~ From 67d93b1a90ac7d620e855adba43a7f827002bca6 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 9 Apr 2025 08:14:17 +0100 Subject: [PATCH 07/12] fixup --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 40e14a13e0..46bfd4b884 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -98,7 +98,7 @@ netgen = [ "ngsPETSc", ] slepc = [ - "slepc4py>=3.22.2<3.23", + "slepc4py>=3.22.2,<3.23", ] torch = [ # requires passing '--extra-index-url' to work "torch", From e9bbe7fe72ae9152f8525fd8c2ddebb7b6444c19 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 9 Apr 2025 09:07:58 +0100 Subject: [PATCH 08/12] fixup --- .github/workflows/pyop2.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/pyop2.yml b/.github/workflows/pyop2.yml index 1a821524a1..2bf25fcd0a 100644 --- a/.github/workflows/pyop2.yml +++ b/.github/workflows/pyop2.yml @@ -73,7 +73,7 @@ jobs: run: | export HDF5_DIR="$PETSC_DIR/$PETSC_ARCH" export HDF5_MPI=ON - python -m pip install --no-binary h5py -v ".[test]" + python -m pip install --no-binary h5py -v ".[check]" - name: Run TSFC tests shell: bash From b726afc591f1a535d8b34c1fd67f73f4be34be46 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 9 Apr 2025 17:20:16 +0100 Subject: [PATCH 09/12] Undo version tweaks in pyproject and dogfood in build.yml --- .github/workflows/build.yml | 2 ++ pyproject.toml | 10 +++++----- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 2870dc9142..a826cfa928 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -85,6 +85,8 @@ jobs: : # install of PETSc. pip cache remove petsc4py pip cache remove slepc4py + pip install petsc4py==$(python3 firedrake-configure --show-petsc-version) + pip install slepc4py==$(python3 firedrake-configure --show-petsc-version) pip install --verbose \ --no-binary h5py \ --extra-index-url https://download.pytorch.org/whl/cpu \ diff --git a/pyproject.toml b/pyproject.toml index 46bfd4b884..a25fce98d2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -22,7 +22,7 @@ dependencies = [ "libsupermesh", # NOTE: If changing the PETSc/SLEPc version then firedrake-configure also needs # changing (as well as other references to petsc4py and slepc4py here) - "petsc4py>=3.22.2,<3.23", + "petsc4py==3.22.2", "numpy", "packaging", "pkgconfig", @@ -98,7 +98,7 @@ netgen = [ "ngsPETSc", ] slepc = [ - "slepc4py>=3.22.2,<3.23", + "slepc4py==3.22.2", ] torch = [ # requires passing '--extra-index-url' to work "torch", @@ -120,7 +120,7 @@ ci = [ "pytest-split", # needed for firedrake-run-split-tests "pytest-timeout", "pytest-xdist", - "slepc4py>=3.22.2,<3.23", + "slepc4py==3.22.2", "torch", # requires passing '--extra-index-url' to work "vtk", ] @@ -134,7 +134,7 @@ docker = [ # Used in firedrake-vanilla container "pytest-split", # needed for firedrake-run-split-tests "pytest-timeout", "pytest-xdist", - "slepc4py>=3.22.2,<3.23", + "slepc4py==3.22.2", ] [build-system] @@ -147,7 +147,7 @@ requires = [ "pkgconfig", "pybind11", "setuptools>61.2", - "petsc4py>=3.22.2,<3.23", + "petsc4py==3.22.2", "rtree>=1.2", ] build-backend = "setuptools.build_meta" From d93c08a0492bd87bb788662412e78f4ce368f06d Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 9 Apr 2025 17:41:10 +0100 Subject: [PATCH 10/12] Build isolation is inelegant --- .github/workflows/build.yml | 5 ++--- docker/Dockerfile.vanilla | 2 +- docs/source/install.rst | 30 +++++++++++++----------------- scripts/firedrake-configure | 2 +- 4 files changed, 17 insertions(+), 22 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d9821d9f66..4dd69e95ae 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -65,7 +65,7 @@ jobs: - name: Install PETSc run: | - git clone --depth 1 --branch v$(python3 ./firedrake-repo/scripts/firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git + git clone --depth 1 --branch $(python3 ./firedrake-repo/scripts/firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git cd petsc python3 ../firedrake-repo/scripts/firedrake-configure \ --arch ${{ matrix.arch }} --show-petsc-configure-options | \ @@ -74,6 +74,7 @@ jobs: make check echo "PETSC_DIR=/__w/firedrake/firedrake/petsc" >> "$GITHUB_ENV" echo "PETSC_ARCH=arch-firedrake-${{ matrix.arch }}" >> "$GITHUB_ENV" + echo "SLEPC_DIR=/__w/firedrake/firedrake/petsc/arch-firedrake-${{ matrix.arch }}" >> "$GITHUB_ENV" - name: Install Firedrake id: install @@ -85,8 +86,6 @@ jobs: : # install of PETSc. pip cache remove petsc4py pip cache remove slepc4py - pip install petsc4py==$(python3 firedrake-configure --show-petsc-version) - pip install slepc4py==$(python3 firedrake-configure --show-petsc-version) pip install --verbose \ --no-binary h5py \ --extra-index-url https://download.pytorch.org/whl/cpu \ diff --git a/docker/Dockerfile.vanilla b/docker/Dockerfile.vanilla index 9272cf0b96..09d30b6820 100644 --- a/docker/Dockerfile.vanilla +++ b/docker/Dockerfile.vanilla @@ -40,7 +40,7 @@ ENV OMPI_ALLOW_RUN_AS_ROOT=1 OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 # Install PETSc. We set the compiler optimisation flags manually here to # remove the default of '-march=native' which is not suitable for Docker images. # We use 'sed' to make sure that the options are appended. -RUN git clone --depth 1 --branch v$(python3 /opt/firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git /opt/petsc \ +RUN git clone --depth 1 --branch $(python3 /opt/firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git /opt/petsc \ && cd /opt/petsc \ && python3 /opt/firedrake-configure --arch $ARCH --show-petsc-configure-options | \ sed "s/$/ --COPTFLAGS='-O3 -mtune=generic' --CXXOPTFLAGS='-O3 -mtune=generic' --FOPTFLAGS='-O3 -mtune=generic'/" | \ diff --git a/docs/source/install.rst b/docs/source/install.rst index fc962be131..884cf399dc 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -174,10 +174,16 @@ install Firedrake. To do this perform the following steps: you have exactly followed the instructions up to this point this should already be the case. -#. Install petsc4py_:: +#. Install Firedrake:: + + $ pip cache remove petsc4py + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check]" - $ pip cache remove petsc4py - $ pip install petsc4py==$(python3 firedrake-configure --show-petsc-version) + .. note:: + Though not strictly necessary to install Firedrake's optional + dependencies with ``[check]`` it is recommended because it allows you + to check that the install was successful (see + :ref:`below`). .. note:: A common issue encountered with this step is the installation will fail with @@ -193,15 +199,6 @@ install Firedrake. To do this perform the following steps: $ ls $PETSC_DIR -#. Install Firedrake:: - - $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check]" - - .. note:: - Though not strictly necessary to install Firedrake's optional - dependencies with ``[check]`` it is recommended because it allows you - to check that the install was successful (see - :ref:`below`). #. Firedrake is now installed and ready for use! @@ -327,12 +324,11 @@ To install Firedrake with SLEPc support you should: $ export SLEPC_DIR=$PETSC_DIR/$PETSC_ARCH -#. Install slepc4py:: - - $ pip cache remove slepc4py - $ pip install slepc4py==$(python3 firedrake-configure --show-petsc-version) +#. Continue with the installation as normal but remove slepc4py from the pip cache + and install Firedrake with the ``slepc`` optional dependency. For example:: -#. Continue with the installation as normal + $ pip cache remove slepc4py + $ pip install --no-binary h5py "firedrake @ git+https://github.com/firedrakeproject/firedrake.git#[check,slepc]" VTK ~~~ diff --git a/scripts/firedrake-configure b/scripts/firedrake-configure index 05363ad77f..6eecdaa6d4 100755 --- a/scripts/firedrake-configure +++ b/scripts/firedrake-configure @@ -41,7 +41,7 @@ ARCH_COMPLEX = FiredrakeArch.COMPLEX # NOTE: When updating this variable corresponding changes must be made inside # pyproject.toml -SUPPORTED_PETSC_VERSION = "3.22.2" +SUPPORTED_PETSC_VERSION = "v3.22.2" def main(): From 246bae6c10729804239ed4693f33f6608871b9c0 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 9 Apr 2025 17:45:26 +0100 Subject: [PATCH 11/12] linting --- .github/workflows/build.yml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 4dd69e95ae..5bd8da3c7a 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -72,9 +72,11 @@ jobs: xargs -L1 ./configure --with-make-np=8 --download-slepc make PETSC_DIR=/__w/firedrake/firedrake/petsc PETSC_ARCH=arch-firedrake-${{ matrix.arch }} make check - echo "PETSC_DIR=/__w/firedrake/firedrake/petsc" >> "$GITHUB_ENV" - echo "PETSC_ARCH=arch-firedrake-${{ matrix.arch }}" >> "$GITHUB_ENV" - echo "SLEPC_DIR=/__w/firedrake/firedrake/petsc/arch-firedrake-${{ matrix.arch }}" >> "$GITHUB_ENV" + { + echo "PETSC_DIR=/__w/firedrake/firedrake/petsc" + echo "PETSC_ARCH=arch-firedrake-${{ matrix.arch }}" + echo "SLEPC_DIR=/__w/firedrake/firedrake/petsc/arch-firedrake-${{ matrix.arch }}" + } >> "$GITHUB_ENV" - name: Install Firedrake id: install From 905453a8c40fd6d08b234e5c6971d959997004c5 Mon Sep 17 00:00:00 2001 From: Connor Ward Date: Wed, 9 Apr 2025 17:47:02 +0100 Subject: [PATCH 12/12] fixup --- docs/source/install.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/install.rst b/docs/source/install.rst index 884cf399dc..6f5e1bfc1a 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -104,7 +104,7 @@ do the following steps: #. Clone the PETSc repository, checking out the right version:: - $ git clone --depth 1 --branch v$(python3 firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git + $ git clone --depth 1 --branch $(python3 firedrake-configure --show-petsc-version) https://gitlab.com/petsc/petsc.git $ cd petsc #. Run PETSc ``configure``, passing in the flags generated by ``firedrake-configure``::