Save project settings.

So far, HydPy does not yet provide a way to save the project settings made in module pub. If you need to use the same settings later or, for example, in different scripts, it is currently the best practice to write one Python script that makes these settings, for example:

my_settings.py

from hydpy import HydPy, pub
hp = HydPy('my_project')
pub.timegrids = '2000-01-01', '2001-01-01', '1d'
pub.options.checkseries = False
pub.networkmanager.currentdir = 'my_folder'

Then, you can import these settings in each script (or in the Python Interpreter) as follows:

my_script.py

from my_settings import *
...

The idea is, to improve the string representation of the pub module (which is already the object of the HydPy-class Pub, derived from Module). As always, the string representation should be executable code as far as possible and be applicable to restore the current configuration of the object. To achieve this goal, we would also need to improve the string representations of some other classes (e.g. Options and ControlManager).

An additional method of class Pub to save the string representation directly into a file would be convenient.

We should discuss if we also want to support saving custom settings. For example, one could also use the module pub to save some constants relevant for different scripts. But this would be restricted to using objects already coming with suitable string representations. So, strings and numbers would be fine, but more complex objects probably not. (We could implement a check into method setattr of module Pub, which executes the string representation for each given attribute and compares the old with the new object.)

Would something like this probably be a solution to save and load the project settings (hydpy.filetools), probably not sophisticated enough looking at the discussions above:

def save_projectsettings(filepath: str = "settings.py", write_timegrids: bool = False) -> None:
    """Save the project settings to file 'filepath'. If the 'filepath is relative, it is interpreted as relative to
    the root directory of the |HydPy| project. |Options| attributes of module |pub|,
    base and current directories of the file managers as well as relevant attributes of the |SequenceManager| are
    written.

    If 'write_timegrids=True', the currently defined |TimeGrids| attribute of module |pub| is written, too.

    >>> from hydpy.core.testtools import prepare_full_example_2
    >>> _, pub, TestIO = prepare_full_example_2()

    We write the project settings to the settings file

    >>> with TestIO():
    ...     save_projectsettings()
    ...     with open("HydPy-H-Lahn/settings.py") as file:
    ...         print(file.read())

    Trying to save non defined timegrids lead to an error:
    >>> with TestIO():
    ...     save_projectsettings(write_timegrids=True)
    Traceback (most recent call last):
    ...
    hydpy.core.exceptiontools.AttributeNotReady: Attribute timegrids of module `pub` is not defined at the moment.

    We define the pub.timegrids and save them to the settings file in the subdirectory
    settings

    >>> pub.timegrids = "2010-01-01","2010-01-10","1d"
    >>> with TestIO():
    ...     save_projectsettings("settings/settings.py", write_timegrids = True)
    ...     with open("HydPy-H-Lahn/settings/settings.py") as file:
    ...         print(file.read())

    """

    if not os.path.isabs(filepath):
        filepath = os.path.abspath(os.path.join(hydpy.pub.networkmanager.projectdir, filepath))

    lines = ["# -*- coding: utf-8 -*-\nimport hydpy\n\n"]

    if write_timegrids:
        lines.append("hydpy.pub.timegrids=" + repr(hydpy.pub.timegrids).replace("Timegrid", "hydpy.Timegrid")+"\n\n")
        lines.append(f"hydpy.pub.options.simulationstep('{str(hydpy.pub.options.simulationstep)}')")

    # save options
    lines.append(f"hydpy.pub.options.parameterstep('{str(hydpy.pub.options.parameterstep)}')")
    lines.append(f"hydpy.pub.options.checkprojectstructure({bool(hydpy.pub.options.checkprojectstructure)})")
    lines.append(f"hydpy.pub.options.checkseries({bool(hydpy.pub.options.checkseries)})")
    lines.append(f"hydpy.pub.options.ellipsis({hydpy.pub.options.ellipsis})")
    lines.append(f"hydpy.pub.options.printprogress({bool(hydpy.pub.options.printprogress)})")
    lines.append(f"hydpy.pub.options.reprdigits({hydpy.pub.options.reprdigits})")
    lines.append(f"hydpy.pub.options.timestampleft({bool(hydpy.pub.options.timestampleft)})")
    lines.append(f"hydpy.pub.options.trimvariables({bool(hydpy.pub.options.trimvariables)})")
    lines.append(f"hydpy.pub.options.usecython({bool(hydpy.pub.options.usecython)})")
    lines.append(f"hydpy.pub.options.usedefaultvalues({bool(hydpy.pub.options.usedefaultvalues)})")
    lines.append(f"hydpy.pub.options.utclongitude({hydpy.pub.options.utclongitude})")
    lines.append(f"hydpy.pub.options.utcoffset({hydpy.pub.options.utcoffset})")
    lines.append(f"hydpy.pub.options.warnmissingcontrolfile({bool(hydpy.pub.options.warnmissingcontrolfile)})")
    lines.append(f"hydpy.pub.options.warnmissingobsfile({bool(hydpy.pub.options.warnmissingobsfile)})")
    lines.append(f"hydpy.pub.options.warnmissingsimfile({bool(hydpy.pub.options.warnmissingsimfile)})")
    lines.append(f"hydpy.pub.options.warnsimulationstep({bool(hydpy.pub.options.warnsimulationstep)})")
    lines.append(f"hydpy.pub.options.warntrim({bool(hydpy.pub.options.warntrim)})\n\n")

    # save filemanager directories
    lines.append(f"hydpy.pub.networkmanager.BASEDIR='{hydpy.pub.networkmanager.BASEDIR}'")
    lines.append(f"hydpy.pub.networkmanager.currentdir='{hydpy.pub.networkmanager.currentdir}'\n")

    lines.append(f"hydpy.pub.controlmanager.BASEDIR='{hydpy.pub.controlmanager.BASEDIR}'")
    lines.append(f"hydpy.pub.controlmanager.currentdir='{hydpy.pub.controlmanager.currentdir}'\n")

    lines.append(f"hydpy.pub.conditionmanager.prefix('{hydpy.pub.conditionmanager.prefix}')")
    lines.append(f"hydpy.pub.conditionmanager.BASEDIR='{hydpy.pub.conditionmanager.BASEDIR}'\n")
    currentdir = hydpy.pub.conditionmanager._currentdir
    if not currentdir:
        lines.append("hydpy.pub.conditionmanager.currentdir=None\n")
    else:
        lines.append(f"hydpy.pub.conditionmanager.currentdir='{hydpy.pub.conditionmanager.currentdir}'\n")

    lines.append(f"hydpy.pub.sequencemanager.filetype('{hydpy.pub.sequencemanager.filetype}')")
    lines.append(f"hydpy.pub.sequencemanager.reset({bool(hydpy.pub.sequencemanager.reset)})")
    lines.append(f"hydpy.pub.sequencemanager.overwrite({bool(hydpy.pub.sequencemanager.overwrite)})")
    lines.append(f"hydpy.pub.sequencemanager.aggregation('{hydpy.pub.sequencemanager.aggregation}')")
    lines.append(f"hydpy.pub.sequencemanager.convention('{hydpy.pub.sequencemanager.convention}')")

    lines.append(f"hydpy.pub.sequencemanager.BASEDIR='{hydpy.pub.sequencemanager.BASEDIR}'")
    lines.append(f"hydpy.pub.sequencemanager.currentdir='{hydpy.pub.sequencemanager.currentdir}'\n\n")

    text = "".join("\n".join(lines))
    dirpath = os.path.dirname(filepath)
    if not os.path.exists(dirpath):
        os.makedirs(dirpath)
        if hydpy.pub.options.printprogress:
            print(f"Directory {dirpath} has been created.")
    with open(filepath, mode="w", encoding="utf-8") as settingsfile:
        settingsfile.write(text)

    if hydpy.pub.options.printprogress:
        print(
            f"Saved project settings in file: {filepath}'."
        )


def load_projectsettings(filepath: str = "settings.py") -> None:
    """Load the project settings from file 'filepath'. If the 'filepath is relative, it is interpreted as relative to
    the root directory of the |HydPy| project.
    
    Trying to read project settings from a non-existent file leads to an error
    >>> load_projectsettings("file.py")
    Traceback (most recent call last):
    ...
    RuntimeError: The settings file `C:\Users\Klein\HydPy_Ressourcen\hydpy_tutorial\HydPy-H-Lahn\file.py` does not exist.

    >>> from hydpy.core.testtools import prepare_full_example_2
    >>> _, pub, TestIO = prepare_full_example_2()

    We define the pub.timegrids and save the project settings
    
    >>> pub.timegrids = "2010-01-01","2010-01-10","1d"
    >>> with TestIO():
    ...     save_projectsettings(write_timegrids = True)
    
    and change some project settings
    
    >>> with TestIO():
    ...     pub.sequencemanager.filetype="nc"
    ...     pub.sequencemanager.currentdir="input"
    ...     pub.sequencemanager.BASEDIR="sequences"
    ...     pub.options.checkseries=False
    ...     pub.timegrids = "2011-01-01","2011-01-10","1d"
    
    After reloading the project settings the values hold their old values
    >>> with TestIO():
    ...     load_projectsettings()
    >>> pub.sequencemanager.filetype
    'asc'
    >>> pub.sequencemanager.currentdir
    'default'
    >>> pub.sequencemanager.BASEDIR
    'series'
    >>> pub.options.checkseries
    TRUE
    >>> pub.timegrids
    ...     pub.options.checkseries=False
    ...     pub.timegrids = "2011-01-01","2011-01-10","1d"
    Timegrids("2010-01-01 00:00:00",
          "2010-01-10 00:00:00",
          "1d")
    """
    if not os.path.isabs(filepath):
        filepath = os.path.abspath(os.path.join(hydpy.pub.networkmanager.projectdir, filepath))
    if not os.path.exists(filepath):
        raise RuntimeError(
            f"The settings file `{filepath}` does not exist."
        )
    runpy.run_path(filepath)

(I edited your code example slightly to enable code highlighting.)

Good idea to make writing the Timegrids instance optional. Maybe we could also add an option like ignore_timegrids to load_projectsettings.

I think, your suggestion covers all the essential functionalities I mentioned above (except for the last paragraph about custom settings). But we should split everything into separate methods. One toy refactoring based on your example:

class Options:

    def __repr__(self) -> str:
        return self.assignrepr(prefix="")

    def assignrepr(self, prefix: str) -> str:
        """
        >>> from hydpy import pub
        >>> print(Options.assignrepr(pub.options, prefix="hydpy.pub.options."))
        hydpy.pub.options.checkprojectstructure(True)
        hydpy.pub.options.checkseries(True)
        hydpy.pub.options.ellipsis(-999)
        hydpy.pub.options.parameterstep(1d)
        hydpy.pub.options.printprogress(True)
        hydpy.pub.options.reprdigits(-1)
        hydpy.pub.options.simulationstep()
        hydpy.pub.options.timestampleft(True)
        hydpy.pub.options.trimvariables(True)
        hydpy.pub.options.usecython(True)
        hydpy.pub.options.usedefaultvalues(False)
        hydpy.pub.options.utclongitude(15)
        hydpy.pub.options.utcoffset(60)
        hydpy.pub.options.warnmissingcontrolfile(False)
        hydpy.pub.options.warnmissingobsfile(True)
        hydpy.pub.options.warnmissingsimfile(True)
        hydpy.pub.options.warnsimulationstep(True)
        hydpy.pub.options.warntrim(True)
        """
        lines = []
        for name in vars(type(self)):
            if not name.startswith("_"):
                try:
                    value = str(getattr(self, name))
                except AttributeError:  # ToDo: refactor based on AttributeNotReady
                    value = ""
                if value in ("TRUE", "FALSE"):  # separate method?
                    value = value.capitalize()
                lines.append(f"{prefix}{name}({value})")
        return "\n".join(lines)

Or:

class Options:

    def __repr__(self) -> str:
        return self.assignrepr(prefix="")

    def assignrepr(self, prefix: str) -> str:
        """
        >>> from hydpy import pub
        >>> print(Options.assignrepr(pub.options, prefix="hydpy.pub.options = "))
        hydpy.pub.options = Options(
            checkprojectstructure=True,
            checkseries=True,
            ellipsis=-999,
            parameterstep=1d,
            printprogress=True,
            reprdigits=-1,
            simulationstep=None,
            timestampleft=True,
            trimvariables=True,
            usecython=True,
            usedefaultvalues=False,
            utclongitude=15,
            utcoffset=60,
            warnmissingcontrolfile=False,
            warnmissingobsfile=True,
            warnmissingsimfile=True,
            warnsimulationstep=True,
            warntrim=True,
        )
        """
        lines = [f"{prefix}Options("]
        for name in vars(type(self)):
            if not name.startswith("_"):
                try:
                    value = str(getattr(self, name))
                except AttributeError:  # ToDo: refactor based on AttributeNotReady
                    value = "None"
                if value in ("TRUE", "FALSE"):  # separate method?
                    value = value.capitalize()
                lines.append(f"    {name}={value},")
        lines.append(")")
        return "\n".join(lines)

Including this in HydPy 6.2 should not require too much effort.

The extension of class Options looks great.

I also thought about adding ignore_timegrids to read_projectsettings but I was not sure about the standard value, False would be my prefered option but it is probably inconsistent with the prefered option write_timegrids = False when writing project settings.

Yes, I am also very uncertain about the "best" default values here (even for both functions). Is this maybe a hint at not providing any default values so that users are forced to think about the right choice in each situation?

I agree with you, providing no default values is probably less convinient for the user but we make sure that the user knows what he is doing.

Navigation Menu

Repository navigation

Save project settings. #47

my_settings.py

my_script.py

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Save project settings. #47

Description

my_settings.py

my_script.py

Activity

BGWKlein commented on Oct 22, 2024

tyralla commented on Oct 23, 2024

BGWKlein commented on Oct 23, 2024

tyralla commented on Oct 24, 2024

BGWKlein commented on Oct 24, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions