Automations - first roundtrip for forecasts

[Flix6x]

Description

An automation is a recurring task defined on an asset. This first roundtrip covers forecasts: the
automation decides when work is due and queues it; the existing forecasting worker still computes
the forecast and stores it as timed beliefs. Nothing about how a forecast is computed changes.

Data model. A new automation table, with per-asset ownership (cascade delete), a type
(forecasts for now), a name, a cronstr recurrence, an IANA timezone in which that recurrence
is interpreted, a cursor, an active flag, a generator_id pointing at the data source
that holds the forecaster configuration, and JSONB parameters for the per-run forecast parameters.

CLI.

• flexmeasures add automation — create one, defaulting to daily at midnight in
  FLEXMEASURES_TIMEZONE. The forecaster configuration is stored on a data source; the forecast
  parameters are validated and stored on the automation. --source reuses an existing forecaster's
  data source, in which case the options that source already determines are refused.
• flexmeasures edit automation — rename, re-schedule (--cron), change the --timezone, activate
  or deactivate.
• flexmeasures delete automation
• flexmeasures jobs run-automations — queue jobs for whatever is due. Run once per minute from
  cron or another host scheduler.

Recurrence and timezones. Each automation's cron expression is interpreted in its own IANA
timezone, so moving the server or changing FLEXMEASURES_TIMEZONE does not silently reschedule
existing automations. Daylight saving is handled explicitly: a local time skipped in spring runs once
at the transition boundary, and a local time that occurs twice in autumn is not queued twice.

Catching up. Each automation stores a UTC cursor, so restarting the runner does not
lose runs that fell during downtime. Several missed runs are coalesced into the latest
useful forecast rather than replayed one by one, and timing parameters that default to the run time
resolve when the caught-up job is queued, so the result is a current forecast. A new automation
starts from its own creation and does not replay history.

At most once. A run is claimed before it is queued, with a compare-and-swap that also
requires the recurrence, the timezone and the active flag to be unchanged since the run was
read. Concurrent runners therefore cannot queue the same run twice, and an edit made between
reading and claiming invalidates the claim rather than clobbering the rebased cursor. A failed
attempt is not retried automatically, because it may already have queued some jobs — durable run
records and safe retries are tracked in #2393.

Validation. A recurrence must be a five-field cron expression that matches at least one real
date, so 0 0 30 2 * is refused rather than accepted as an automation that can never fire. A
timezone must be a real IANA name.

API. [GET] /assets/(id)/automations and [GET] /assets/(id)/automations/(automation_id) list
and inspect an asset's automations, including the sensors each reads from and writes to, its
timezone and cursor, and counts of recently created jobs per status. A new
[GET] /sources/(id) returns the full record of one data source, including the attributes where
data generators keep their configuration. Asset job entries carry created_via provenance.

UI. The asset gains an Automations page: a sortable listing, and a details modal linking to the
sensors an automation reads from and writes to. A sensor's own page lists the automations that feed
it, and its data source can be inspected as a full record.

Provenance. Every queued job records how it came about — via the CLI, the API, or an automation
(with the automation's id) — so the status page can tell them apart.

• Added changelog item in documentation/changelog.rst

Look & Feel

Automations defined on an asset get their own page. The listing is sortable, describes each
recurrence in natural language, and shows the timezone that recurrence is interpreted in. The
Schedules and Reports tabs are placeholders until #2293 and #2297:

An automation's details show the timezone its cron expression is interpreted in, its cursor, the
data generator holding the forecaster configuration, and the sensors it reads from and writes to,
each linking to its own page. Recently created jobs are counted per status:

How to test

See the manual test walkthrough in the PR comments. In short: create an automation with
flexmeasures add automation, activate it, run flexmeasures jobs run-automations, and confirm a
forecasting job was queued and appears on the asset's status page.

Automated coverage:

pytest \
  flexmeasures/cli/tests/test_automations.py \
  flexmeasures/data/tests/test_automations_fresh_db.py \
  flexmeasures/data/tests/test_automation_scheduling_fresh_db.py \
  flexmeasures/data/schemas/tests/test_automations.py \
  flexmeasures/api/v3_0/tests/test_automations_api.py

Further improvements

• Durable automation-run records and safe retry semantics — Add durable automation-run records and safe retry semantics #2393. Until those exist the runner keeps
  the at-most-once guard rather than risking duplicate jobs.
• Letting operators bound or disable catch-up — Let operators bound or disable automation catch-up #2423.
• The runner is driven by an external per-minute cron. Using rq-scheduler for that tick, so
  FlexMeasures schedules the runner inside the RQ ecosystem, is a natural follow-up.
• Automations are created through the CLI here; administering them through the API and UI, and
  restricting them to sensors their creator can access, follows in CRUD for automations in the API and UI #2294.

Related items

Closes #2288. Closes #2392 (timezone-aware recurrence and catch-up, merged in from #2396).
Part of the automations story #2334. Followed by #2293 (schedules as automations).

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or another incompatible license.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cron-descriptor@​2.1.0

View full report

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #34333441 | 📁 Comparing 06bf14b against latest (60cfda8)

  🔍 Preview build  

138 files changed · + 24 added · ± 114 modified

+ Added

• _autodoc/COUPLING.html
• _autodoc/COUPLING_COEFFICIENT.html
• _autodoc/GROUP.html
• _autodoc/INFLEXIBLE_CONSUMPTION.html
• _autodoc/INFLEXIBLE_PRODUCTION.html
• _autodoc/OPERATION_MODES.html
• _autosummary/flexmeasures.api.common.rate_limiting.html
• _autosummary/flexmeasures.api.v3_0.utils.html
• _autosummary/flexmeasures.data.models.automations.html
• _autosummary/flexmeasures.data.models.planning.devices.html
• and 14 more...

± Modified

• changelog.html
• configuration.html
• genindex.html
• getting-started.html
• http-routingtable.html
• index.html
• py-modindex.html
• _autodoc/COMMITMENTS.html
• _autodoc/CONSUMPTION_CAPACITY.html
• _autodoc/INFLEXIBLE_DEVICE_SENSORS.html
• and 104 more...

[BelhsanHmida]

Manual test walkthrough

A reviewer can exercise the whole roundtrip with this. It is written against this branch only:
schedules and reports are still "coming soon", and automations are created from the CLI
(administering them through the API and UI arrives in #2294).

Start a worker in a second terminal, and pick a sensor that already holds history so a forecast has
something to train on. The examples use asset 242 and its power sensor 913.

flexmeasures jobs run-worker --name automations-demo --queue forecasting

1. The command explains itself

flexmeasures add automation --help

The help stays focused on the automation: the asset, the name, the recurrence and its timezone, the
type, and how the forecaster is configured. Every option flexmeasures add forecast accepts is
still accepted, but is kept out of the help so it does not bury the options that describe the
automation itself.

Options:
  --asset GENERICASSETIDFIELD  ID of the asset to automate a recurring task
                               for.  [required]
  --name TEXT                  Name of the automation.  [required]
  --cron CRONFIELD             Recurrence as a standard five-field cron
                               expression, e.g. "0 6 * * *" for daily at
                               06:00. The expression is interpreted in the
                               automation timezone. Defaults to daily at
                               midnight.  [default: 0 0 * * *]
  --timezone TIMEZONEFIELD     IANA timezone in which to interpret --cron,
                               e.g. "UTC" or "Europe/Amsterdam". Defaults to
                               FLEXMEASURES_TIMEZONE.  [default:
                               (FLEXMEASURES_TIMEZONE)]
  --type [forecasts]           Type of task to automate.  [default: forecasts]
  --inactive                   Add this flag to create the automation in
                               deactivated state.
  --forecaster TEXT            Forecaster class registered in
                               flexmeasures.data.models.forecasting or in an
                               available flexmeasures plugin. Defaults to
                               TrainPredictPipeline. Use the command
                               `flexmeasures show forecasters` to list all the
                               available forecasters. Cannot be combined with
                               --source, which already determines the
                               forecaster.
  --source DATASOURCEIDFIELD   DataSource ID of the `Forecaster`. The
                               forecaster class and its configuration are read
                               from the data source's data generator
                               attributes, so --forecaster and --config are
                               not needed (or allowed) with it.
  --config FILENAME            Path to the JSON or YAML file with the
                               configuration of the forecaster. Cannot be
                               combined with --source, which already
                               determines the configuration.
  --parameters FILENAME        Path to the JSON or YAML file with the forecast
                               parameters (passed to the compute step on each
                               run of the automation).
  --help                       Show this message and exit.

2. Validation refuses what could never work

# a recurrence that matches no real date
flexmeasures add automation --asset 242 --name "Impossible" --cron "0 0 30 2 *" --sensor 913
# → Error: Invalid value for '--cron': '0 0 30 2 *' does not match any possible date.

# six fields instead of five
flexmeasures add automation --asset 242 --name "Six fields" --cron "0 0 * * * *" --sensor 913
# → Error: ... must contain exactly five fields (minute, hour, day of month, month, and day of week).

# a timezone that does not exist
flexmeasures add automation --asset 242 --name "Bad tz" --cron "0 6 * * *" \
  --timezone Europe/NotAmsterdam --sensor 913
# → Error: Invalid value for '--timezone': Timezone 'Europe/NotAmsterdam' does not exist.

Nothing is created by any of these. The first is the interesting one: 0 0 30 2 * is a perfectly
well-formed cron expression, and without this check it would be accepted as an automation that can
never fire.

3. Create one, inactive, and read it back

flexmeasures add automation \
  --asset 242 \
  --name "Campus power forecast" \
  --cron "0 6 * * *" \
  --timezone Europe/Amsterdam \
  --inactive \
  --sensor 913
# → Successfully created inactive automation '...' (ID: N) to compute forecasts for asset 242,
#   recurring per cron string '0 6 * * *' in timezone 'Europe/Amsterdam'.

/assets/242/automations

The listing is sortable and shows the recurrence in natural language alongside the timezone it is
interpreted in. On this branch the Schedules and Reports tabs are present but disabled.

Details shows what the automation would read and write. Both are sensor 913 here, because a
forecast is saved to the sensor it forecasts unless --sensor-to-save says otherwise, and each
sensor links to its own page.

The same information is available over the API:

/api/v3_0/assets/242/automations              # the listing

/api/v3_0/assets/242/automations/<N>          # one automation, in full

The detail response carries input_sensors, output_sensors, timezone, scheduling_cursor,
recurrence_description, the generator holding the forecaster configuration, the stored
parameters, and job_stats counting recently created jobs per status.

4. The recurrence belongs to the automation, not the server

flexmeasures edit automation --id N --timezone Asia/Seoul
# → Successfully updated automation '...' (ID: N): timezone: 'Europe/Amsterdam' → 'Asia/Seoul'.

0 6 * * * in Asia/Seoul is 21:00 UTC the day before, so moving the server or changing
FLEXMEASURES_TIMEZONE does not silently reschedule anything. Changing the timezone also rebases
the scheduling cursor, so the automation does not immediately catch up on occurrences that only
exist in the new timezone's past. Put it back:

flexmeasures edit automation --id N --timezone Europe/Amsterdam

5. Run it

flexmeasures edit automation --id N --cron "* * * * *" --activate
flexmeasures jobs run-automations
# → Automation N ('Campus power forecast') queued 1 forecasting job(s) for asset 242.

Run it again within the same minute:

flexmeasures jobs run-automations
# → the occurrence was already claimed; nothing is queued

That is the at-most-once guard. An occurrence is claimed before it is queued, and the claim also
requires the recurrence, the timezone and the active flag to be unchanged since it was read — so two
runners cannot queue the same occurrence, and an edit made in between invalidates the claim instead
of being overwritten.

6. See where the job came from

/assets/242/status

Jobs record how they were created — via the CLI, the API, or an automation, with the automation's
id — so a job queued here is distinguishable from one triggered by hand.

Once the worker has run it, the forecast is ordinary belief data:

/sensors/913

The sensor's page also lists the automations that write to it, and its data source can be inspected
as a full record.

7. Catching up after downtime

Leave the automation active and due, but stop running the runner for a few minutes. Then run it
once. It queues a single job, not one per missed minute: several missed occurrences are coalesced
into the latest useful forecast, and timing parameters that default to the run time resolve when the
job is queued, so the result is a current forecast rather than a replay of an old window.

A newly created automation starts from its own creation and does not replay history, and the cursor
lives in the database, so restarting the runner does not lose due occurrences.

8. Clean up

flexmeasures edit automation --id N --deactivate
flexmeasures delete automation --id N --force

Deleting an automation removes only its definition. Forecasts it already produced are ordinary
beliefs and stay.

What this deliberately does not cover

• Retrying a run that failed partway. The runner is at-most-once until durable run records exist
  (Add durable automation-run records and safe retry semantics #2393), because a failure may have happened after some jobs were already queued.
• Bounding or disabling catch-up (Let operators bound or disable automation catch-up #2423).
• Creating and editing automations through the API and the UI, and restricting them to sensors their
  creator can access — that is CRUD for automations in the API and UI #2294.
• Schedules and reports as automations — Schedules as automations #2293 and Reports as jobs and as automations #2297.

[BelhsanHmida]

@Flix6x can you review this

[Flix6x]

Note, I'm just posting these comments, and will pick up a first round of changes myself.

[Flix6x]

Note, I'm just posting these comments, and will pick up a first round of changes myself.

[Flix6x]

Note, I'm just posting these comments, and will pick up a first round of changes myself.

[Flix6x]

There is nothing in the CLI command that tells it that it should be forecasting, which feels off. I imagine this example will be changed in ongoing follow-up PRs when flexmeasures add automation can be used for scheduling and reporting, too. Perhaps those PRs offer a clue as to how this example could be made more explicit within this PR already.

[Flix6x]

Made explicit: the example now passes --type forecasts. That option already exists on this branch (default forecasts, click.Choice(Automation.SUPPORTED_TYPES)), and it is exactly what #2293 and #2297 use to select schedules and reports — so spelling it out here is the clue you expected, rather than something new. 4051dc2

[Flix6x]

Up until this point, it wasn't clear that --forecaster and --config were needed, so this comes as a bit of a surprise.

[Flix6x]

Introduced them before they are waved away: the paragraph now says a forecast automation accepts everything flexmeasures add forecast accepts, naming --forecaster and --config, and only then that --source makes them unnecessary. 4051dc2

[Flix6x]

Most of this feels like a developer's note more worthy of being in an appendix than in the main body explaining forecasting automations. It isn't even specific to forecasting.

[Flix6x]

Agreed, and you are right that it is not forecasting-specific. The cursor mechanics and the daylight-saving-time rules are now an appendix on the new page, prefaced with a note that it is bookkeeping you do not need in order to use automations. The main body keeps only the user-visible consequence: after downtime, just the latest missed run is queued. 4051dc2

[Flix6x]

This reads like a todo. Should it really be in the docs?

[Flix6x]

Dropped — it was a todo, and belongs on the issue rather than in the docs. I kept the preceding sentence, that a failed attempt is not retried automatically, since that is behaviour users have to plan around. 4051dc2

[Flix6x]

I think this whole section is better suited in features/automations.rst, since most of it will apply to scheduling and reporting automations, too, in the future.

[Flix6x]

Moved to documentation/features/automations.rst, and rewritten to talk about automations in general with forecasts as today's only type. features/forecasting.rst keeps a short pointer under the same automating_forecasts label, and the new page is in the Features toctree. 4051dc2