Skip to content

ditto.recorders

Recorder protocol, registry, and built-in recorders.

RECORDER_REGISTRY discovers names from installed entry-point metadata and loads each recorder on its first successful lookup. A recorder's name is also its persisted identifier. Membership checks, key iteration, len() and problems do not load plugins. Names retain discovery order, followed by names added with register() in the order they were added.

Every registration is checked against the plugin contract when the registry is built, so problems is complete without loading anything, and loading a recorder never changes it. A pytest run with any problem stops before collection.

The registry is read-only apart from register(), which adds a recorder under a new name and raises DittoRecorderConflictError if the name conflicts with one already registered. Registered recorders cannot be replaced or removed. Operations that read values, such as get() and iteration over items() or values(), can load plugins and raise DittoRecorderLoadError. The fallback argument to recorders.get() applies only to absent names; it does not suppress errors from an installed recorder that fails to load.

copy.copy(RECORDER_REGISTRY) creates an independent registry without loading plugins. Already loaded recorder objects are shared, while later registrations affect only the registry they are made in. For isolated tests of recorders.get() and recorders.register(), pass a RecorderRegistry([], []) through their registry argument.

Recorder(dumps, loads) dataclass

Bases: Generic[T]

Recorder: a pair of functions that serialise a value to bytes and back.

A Recorder specifies how a snapshot value becomes the bytes a backend stores, and how those bytes become a value again. The bytes are the snapshot file's contents: a JSON recorder produces ordinary JSON text, a parquet recorder an ordinary parquet file. The type parameter T constrains the data type this recorder operates on. Use Recorder[Any] for generic formats (yaml, json). Use a concrete type for format-specific recorders (e.g. Recorder[pd.DataFrame]).

A recorder's persisted identifier, which ends its snapshot filenames and is recorded in ditto.lock, is the name it is registered under (e.g. "yaml", "pandas.parquet"), not a property of the recorder.

Snapshots are handled whole: the value and its serialised bytes must fit in memory together. For a library that can only read and write files, wrap its functions with recorder_from_files.

Parameters:

Name Type Description Default
dumps Callable[[T], bytes]

Function that serialises a value of type T to bytes. It must not modify the value: the caller compares that value with what loads(dumps(value)) returns.

required
loads Callable[[bytes], T]

Function that deserialises bytes produced by dumps into a value of type T.

required

RecorderRegistry(entry_points=None, marks_entry_points=None)

Bases: Mapping[str, Recorder]

Recorders keyed by name, imported only when first looked up.

A recorder's name is also its persisted identifier: it ends the recorder's snapshot filenames and is recorded in ditto.lock.

Names come from ditto_recorders entry-point metadata, which needs no import, so membership tests, iteration and problems never import a plugin. Every registration is checked against the plugin contract when the registry is built, so problems is complete from the start and loading a recorder never changes it. Looking up a name a problem affects raises DittoRecorderConflictError, so the registry never picks one of the conflicting registrations.

register adds a recorder under a new name, and rejects a name that breaks the contract alone or together with the names already registered. A registration therefore never makes an earlier one unusable. Registrations are never replaced or removed. Iteration preserves discovery order, followed by registered names in the order they were added.

Looking up values (including through get, items or values) can load plugins and raise DittoRecorderLoadError or DittoRecorderConflictError.

Parameters:

Name Type Description Default
entry_points Iterable[EntryPoint]

Entry points to serve. Defaults to the installed ditto_recorders group.

None
marks_entry_points Iterable[EntryPoint]

Entry points in the removed ditto_marks group, which identify plugins still on the 1.x contract. Defaults to the installed ditto_marks group.

None
Source code in ditto/recorders/_plugins.py
def __init__(
    self,
    entry_points: Iterable[EntryPoint] | None = None,
    marks_entry_points: Iterable[EntryPoint] | None = None,
) -> None:
    if entry_points is None:
        entry_points = importlib.metadata.entry_points(group="ditto_recorders")
    if marks_entry_points is None:
        marks_entry_points = importlib.metadata.entry_points(group="ditto_marks")
    entry_points = list(entry_points)
    legacy = {_distribution(ep) for ep in marks_entry_points}

    self._registrations = [
        Registration(ep.name, _distribution(ep)) for ep in entry_points
    ]
    self._entries: dict[str, Recorder | EntryPoint] = {}
    for ep in entry_points:
        self._entries.setdefault(ep.name, ep)
    self._legacy_distributions = frozenset(d.name for d in legacy)
    self._problems = (
        *find_name_problems(self._registrations),
        *find_legacy_problems(legacy),
    )
    self._conflicts = {
        name: problem for problem in self._problems for name in problem.names
    }
    # Recorders loaded from entry points so far.
    self._loaded: dict[str, Recorder] = {}

problems property

Every contract problem among the registrations, found without imports.

register(name, recorder)

Add recorder under name, which is also its persisted identifier.

Raises:

Type Description
TypeError

If recorder is not a Recorder.

DittoRecorderConflictError

If name is already registered, or breaks another naming rule together with the names already registered.

Source code in ditto/recorders/_plugins.py
def register(self, name: str, recorder: Recorder) -> None:
    """Add `recorder` under `name`, which is also its persisted identifier.

    Raises
    ------
    TypeError
        If `recorder` is not a `Recorder`.
    DittoRecorderConflictError
        If `name` is already registered, or breaks another naming rule
        together with the names already registered.
    """
    if not isinstance(recorder, Recorder):
        raise TypeError(
            f"{type(recorder).__name__} is not a ditto.recorders.Recorder"
        )
    registration = Registration(name, LOCAL_REGISTRATION)
    problems = [
        problem
        for problem in find_name_problems([*self._registrations, registration])
        if name in problem.names
    ]
    if problems:
        raise DittoRecorderConflictError(
            " ".join(problem.message for problem in problems)
        )
    self._registrations.append(registration)
    self._entries[name] = recorder

__copy__()

Copy registrations and loaded recorders without loading entry points.

Source code in ditto/recorders/_plugins.py
def __copy__(self) -> "RecorderRegistry":
    """Copy registrations and loaded recorders without loading entry points."""
    registry = RecorderRegistry([], [])
    registry._registrations = self._registrations.copy()
    registry._entries = self._entries.copy()
    registry._legacy_distributions = self._legacy_distributions
    registry._problems = self._problems
    registry._conflicts = self._conflicts
    registry._loaded = self._loaded.copy()
    return registry

recorder_from_files(*, save, load, suffix)

Adapt a library that reads and writes files to the byte-based Recorder.

Each dumps and loads call uses a fresh temporary directory holding one file, snapshot<suffix>, which is removed afterwards whether the call succeeds or fails. Libraries that can serialise to bytes directly should build a Recorder from their in-memory functions instead, avoiding the file I/O.

Only single-file formats are supported. A value returned by load must not depend on the temporary file after load returns, so lazy or memory-mapped readers are unsuitable.

Parameters:

Name Type Description Default
save Callable[[T, Path], None]

Function that writes a value to the given path.

required
load Callable[[Path], T]

Function that reads a value from the given path.

required
suffix str

Filename suffix the library expects, e.g. ".parquet". It names only the temporary file; the snapshot's filename comes from the name the recorder is registered under.

required

Returns:

Type Description
Recorder[T]

A recorder whose dumps and loads go through a temporary file.

Raises:

Type Description
ValueError

If suffix does not start with ., is only ., or contains a path separator.

Source code in ditto/recorders/_files.py
def recorder_from_files(
    *,
    save: Callable[[T, Path], None],
    load: Callable[[Path], T],
    suffix: str,
) -> Recorder[T]:
    """
    Adapt a library that reads and writes files to the byte-based `Recorder`.

    Each `dumps` and `loads` call uses a fresh temporary directory holding one
    file, `snapshot<suffix>`, which is removed afterwards whether the call
    succeeds or fails. Libraries that can serialise to bytes directly should
    build a `Recorder` from their in-memory functions instead, avoiding the
    file I/O.

    Only single-file formats are supported. A value returned by `load` must not
    depend on the temporary file after `load` returns, so lazy or memory-mapped
    readers are unsuitable.

    Parameters
    ----------
    save : Callable[[T, Path], None]
        Function that writes a value to the given path.
    load : Callable[[Path], T]
        Function that reads a value from the given path.
    suffix : str
        Filename suffix the library expects, e.g. `".parquet"`. It names only
        the temporary file; the snapshot's filename comes from the name the
        recorder is registered under.

    Returns
    -------
    Recorder[T]
        A recorder whose `dumps` and `loads` go through a temporary file.

    Raises
    ------
    ValueError
        If `suffix` does not start with `.`, is only `.`, or contains a path
        separator.
    """
    if not suffix.startswith(".") or suffix == "." or "/" in suffix or "\\" in suffix:
        raise ValueError(
            f"suffix must be a filename suffix such as '.parquet', got {suffix!r}"
        )

    def dumps(value: T) -> bytes:
        with TemporaryDirectory() as directory:
            path = Path(directory) / f"snapshot{suffix}"
            save(value, path)
            return path.read_bytes()

    def loads(raw: bytes) -> T:
        with TemporaryDirectory() as directory:
            path = Path(directory) / f"snapshot{suffix}"
            path.write_bytes(raw)
            return load(path)

    return Recorder(dumps=dumps, loads=loads)

register(name, recorder, registry=RECORDER_REGISTRY)

Add a recorder to the given registry under a new name.

The name is also the recorder's persisted identifier: it ends the recorder's snapshot filenames and is recorded in ditto.lock.

Parameters:

Name Type Description Default
name str

Name to register the recorder under, following the recorder name grammar (<format> or <namespace>.<format>).

required
recorder Recorder

The recorder instance to register.

required
registry RecorderRegistry

Registry to add to. Defaults to the shared RECORDER_REGISTRY. Pass an isolated RecorderRegistry([], []) in tests to avoid mutating shared state.

RECORDER_REGISTRY

Raises:

Type Description
DittoRecorderConflictError

If name is already registered, or breaks another naming rule together with the names already registered. Registered recorders cannot be replaced.

Source code in ditto/recorders/__init__.py
def register(
    name: str,
    recorder: Recorder,
    registry: RecorderRegistry = RECORDER_REGISTRY,
) -> None:
    """
    Add a recorder to the given registry under a new name.

    The name is also the recorder's persisted identifier: it ends the
    recorder's snapshot filenames and is recorded in `ditto.lock`.

    Parameters
    ----------
    name : str
        Name to register the recorder under, following the recorder name
        grammar (`<format>` or `<namespace>.<format>`).
    recorder : Recorder
        The recorder instance to register.
    registry : RecorderRegistry, optional
        Registry to add to. Defaults to the shared `RECORDER_REGISTRY`.
        Pass an isolated `RecorderRegistry([], [])` in tests to avoid mutating
        shared state.

    Raises
    ------
    DittoRecorderConflictError
        If `name` is already registered, or breaks another naming rule together
        with the names already registered. Registered recorders cannot be
        replaced.
    """
    registry.register(name, recorder)

get(name, registry=RECORDER_REGISTRY, fallback=_MISSING)

Look up a recorder by name.

Parameters:

Name Type Description Default
name str

Key to look up in the registry.

required
registry Mapping[str, Recorder]

Registry to query. Defaults to the shared RECORDER_REGISTRY. Pass an isolated dict in tests to avoid depending on shared state.

RECORDER_REGISTRY
fallback Recorder

Recorder to return when name is not found. If omitted, an unknown recorder raises DittoUnknownRecorderError. A registered recorder that fails to load raises DittoRecorderLoadError, even if a fallback is given.

_MISSING

Returns:

Type Description
Recorder

The registered recorder, or an explicitly supplied fallback.

Source code in ditto/recorders/__init__.py
def get(
    name: str,
    registry: Mapping[str, Recorder] = RECORDER_REGISTRY,
    fallback: Recorder | object = _MISSING,
) -> Recorder:
    """
    Look up a recorder by name.

    Parameters
    ----------
    name : str
        Key to look up in the registry.
    registry : Mapping[str, Recorder], optional
        Registry to query. Defaults to the shared `RECORDER_REGISTRY`.
        Pass an isolated dict in tests to avoid depending on shared state.
    fallback : Recorder, optional
        Recorder to return when `name` is not found. If omitted, an unknown
        recorder raises `DittoUnknownRecorderError`. A registered recorder that
        fails to load raises `DittoRecorderLoadError`, even if a fallback is given.

    Returns
    -------
    Recorder
        The registered recorder, or an explicitly supplied `fallback`.
    """
    if name in registry:
        return registry[name]
    if fallback is not _MISSING:
        return cast(Recorder, fallback)
    raise DittoUnknownRecorderError(name, list(registry))

default()

Return the default recorder.

Returns:

Type Description
Recorder

The strict JSON recorder.

Source code in ditto/recorders/__init__.py
def default() -> Recorder:
    """
    Return the default recorder.

    Returns
    -------
    Recorder
        The strict JSON recorder.
    """
    return _default