pelutils.serialization package

JSON persistence for Pydantic models containing non-JSON values.

JSON is useful when the structure of a file should remain inspectable, but it does not represent values such as NumPy arrays, tensors, or arbitrary Python objects. UniversalJsonModel keeps JSON-native values in the document and stores unsupported field values as base64-encoded pickle payloads. This preserves a readable JSON envelope without requiring callers to write converters for every field type.

The usual interface is UniversalJsonModel.save() and UniversalJsonModel.load(). save creates missing parent directories and writes the model to a JSON file; load reconstructs the model from that file. Use UniversalJsonModel.to_json_dict() and UniversalJsonModel.from_json_dict() when the serialized representation should be nested inside another structure or handled by another storage layer.

For standalone dictionaries and lists, pretty_json converts Python objects to pretty-formatted JSON, filling the same role as json.dumps, but often making the resulting JSON more readable. JSONL helpers handle files containing multiple JSON records separated by newlines.

Quick start

import numpy as np
from pelutils.serialization import UniversalJsonModel
from pelutils.types import FloatArray

class Result(UniversalJsonModel):
    accuracy: float
    predictions: FloatArray   # numpy arrays are handled automatically

result = Result(accuracy=0.97, predictions=np.arange(5, dtype=np.float16))
result.save("results/run-1.json")
result = Result.load("results/run-1.json")

This format is intended for trusted application data such as experiment results and cached state, not as a language-neutral interchange format. Pickle payloads are Python-specific, and loading them can execute arbitrary code. Use a format with an explicit schema and safe decoder when data comes from outside the application.

Warning

UniversalJsonModel.load and the pickle fallback execute code while loading. Never load files from an untrusted source.

class pelutils.serialization.UniversalJsonModel

Bases: BaseModel

Pydantic BaseModel with JSON persistence for arbitrary Python values.

Values unsupported by JSON are pickle-encoded and stored as base64 strings. Do not load data from an untrusted source. Custom model_config values must retain arbitrary_types_allowed=True if any fields on subclasses are not JSON-serialisable by default.

classmethod from_json_dict(json_dict: dict[str, Any], **model_validate_kwargs: Any) → Self

Build a model from to_json_dict() output. Do not load untrusted data.

Keyword arguments are forwarded to pydantic.BaseModel.model_validate().

classmethod load(path: str | Path, *, encoding: str = 'utf-8') → Self

Load a model from path. Do not load untrusted data.

model_config = {'arbitrary_types_allowed': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

save(path: str | Path, *, max_line_length: int = 140, indent: int = 2, encoding: str = 'utf-8')

Save the model to path as human-readable JSON, creating parent directories as needed.

to_json_dict(**model_dump_kwargs: Any) → dict[str, Any]

Return a JSON-compatible dictionary, pickle-encoding unsupported values.

Keyword arguments are forwarded to pydantic.BaseModel.model_dump(), except mode and fallback, which are fixed to preserve this method’s JSON format.

pelutils.serialization.jsonl_dump(objects: Iterable[Any], f: TextIO, single_block: bool = True)

Save an iterable to a .jsonl file.

If single_block is True, objects will be joined to a single block before being written. It can be useful to set this to False if there is a large amount of lazily generated data.

pelutils.serialization.jsonl_dumps(objects: Iterable[Any]) → str

Return the objects as a single .jsonl-formatted string, one JSON object per line.

pelutils.serialization.jsonl_load(f: TextIO) → Iterator[Any]

Return a generator of parsed lines in a .jsonl file. Empty lines are ignored.

pelutils.serialization.jsonl_loads(string: str) → Iterator[Any]

Return a generator of parsed lines. Empty lines are ignored.

pelutils.serialization.pretty_json(obj: dict[str, Any] | list[Any], *, max_line_length: int = 140, indent: int = 2) → str

Recursively serialise dicts or lists to JSON, similar to json.dumps, but with more readable formatting.

• The root container is always expanded (one element per line).

• Nested containers stay on one line when they fit within max_line_length; otherwise they are expanded recursively.

• Primitive-only lists are bin-packed: items fill each line up to max_line_length before wrapping to the next.

Parameters:

• obj – A dict or list (may contain arbitrary Python objects).

• max_line_length – Soft limit for line width.

• indent – Number of spaces per indentation level.

Returns:

A pretty-formatted, valid JSON string.

Return type:

str