Skip to content

Transformation JSON schema

Each cleaned station is described by a single JSON file that records every operation applied to the raw signal: the valid data window, dropped ranges and timestamps, breakpoints, and tsunami events. Because the cleaning is fully driven by these configs, the whole dataset is reproducible from the raw IOC data plus the transformations.

Getting the transformations

  • Automatic - C.load_transformation(station, sensor) resolves the configs for you: it uses the local transformations/ directory if present, otherwise it downloads and caches transformations.tar.gz from Zenodo. See Data Access.
  • Zenodo - the complete dataset includes transformations.tar.gz on the Zenodo record, if you prefer to fetch and extract them by hand.
  • GitHub - the transformations and metadata are also mirrored on the GitHub releases, and live in the transformations/ directory of the repository.

Build your own

You are not limited to the shipped configs. To clean a station yourself, write a JSON file following the schema below and load it with C.load_transformation_from_path(...). This lets you adjust the valid window, drop additional bad ranges, or flag events for stations and periods not yet covered.

import searvey
import ioc_cleanup as C

station = "abed"
df_raw = searvey.fetch_ioc_station(station, "2020-01-01", "2026-01-01")

trans = C.load_transformation_from_path(
    "my_configs/abed_bub.json"
)

df_clean = C.transform(df_raw, trans)

Example

{
  "ioc_code": "abed",
  "sensor": "bub",
  "notes": "",
  "skip": false,
  "wip": false,
  "start": "2020-01-01T00:00:00",
  "end": "2026-01-01T00:00:00",
  "high": null,
  "low": null,
  "dropped_date_ranges": [
    ["2022-03-27 03:00:00", "2022-03-27 03:45:00"],
    ["2023-03-26 03:00:00", "2023-03-26 03:45:00"]
  ],
  "dropped_timestamps": [
    "2022-09-30T14:45:00",
    "2022-09-30T15:30:00",
    "2022-10-02T06:45:00",
    "2022-10-02T07:00:00",
    "2023-06-21T00:15:00",
    "2024-04-24T11:00:00",
    "2024-09-07 12:00:00"
  ],
  "breakpoints": [],
  "tsunami": [],
  "scale": 1.0,
  "offset": 0.0
}

Field descriptions

Field Description
ioc_code IOC station code
sensor Sensor identifier
start, end Valid data window
dropped_date_ranges Continuous ranges to remove
dropped_timestamps Individual timestamps
breakpoints Sensor regime changes
tsunami Eventual tsunami(s) date ranges
skip Ignore station
wip Work in progress
scale Multiplicative factor applied to the sensor signal (default 1.0; use -1.0 for inverted sensors)
offset Constant added to the sensor signal, in meters (default 0.0)

Scale and offset

After the bad data have been dropped, the sensor column is corrected linearly:

value_clean = value_raw * scale + offset

Both fields are optional and default to a no-op (scale = 1.0, offset = 0.0). The typical use case is a sensor wired or reported with the opposite sign: set "scale": -1.0 to flip it. offset is meant for a constant datum shift; note that it has no effect on demeaned output.