Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/concepts/plans.md
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,24 @@ These examples demonstrate how to select which models to restate based on model
sqlmesh plan --restate-model "db.model_a" --start "2024-01-01" --end "2024-01-10"
```

### Restatement after moving a model's start earlier

When you move an incremental model's `start` earlier, SQLMesh backfills that model for the newly added date range, as usual. Downstream models may already hold intervals for that same range, though, and those were computed while the upstream model was empty there.

To keep that stale data from lingering, SQLMesh also restates the downstream models' existing intervals for the added range. The range is widened to each downstream model's interval unit. The plan output lists these models under the heading "Models restated because the start of an upstream model moved earlier".

No restatement happens when the new `start` is later than before or when the change is [breaking](#breaking-change), because downstream models are rebuilt anyway.

SQLMesh skips a model and prints a single warning in these cases. Models downstream of a skipped model are not restated through it, since its data doesn't change, but they can still be restated through another upstream model:

- The model has `disable_restatement` set to `true` (this is the default for SCD Type 2 models).
- The model's kind only supports restating its full history (`INCREMENTAL_BY_UNIQUE_KEY`, `INCREMENTAL_UNMANAGED`, `INCREMENTAL_BY_PARTITION`, and the SCD Type 2 kinds). The warning prints the `sqlmesh plan --restate-model ...` command to run manually.
- The plan targets a development environment and the downstream model's table is one that `prod` uses or will use. These models are restated when the change is applied to `prod`.

If you use `--select-model`, `--skip-backfill` or `--empty-backfill`, stale models that the plan does not backfill are not restated either. The same applies to the part of the added range before the plan's `--start`, when it is set later than the new model `start` outside a development environment. In both cases the warning prints the `--restate-model` command to run manually.

Unlike a plan with `--restate-model`, these automatic restatements do not make the plan a restatement plan. For example, [signals](../guides/signals.md) are evaluated as in a regular run.

### Restating production vs development

Restatement plans behave differently depending on if you're targeting the `prod` environment or a [development environment](./environments.md#how-to-use-environments).
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/model_configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Configuration options for SQLMesh model properties. Supported by all model kinds
| `tags` | Arbitrary strings used to organize or classify a model | array[str] | N |
| `cron` | The cron expression specifying how often the model should be refreshed. (Default: `@daily`) | str | N |
| `interval_unit` | The temporal granularity of the model's data intervals. Supported values: `year`, `month`, `day`, `hour`, `half_hour`, `quarter_hour`, `five_minute`. (Default: inferred from `cron`) | str | N |
| `start` | The date/time that determines the earliest date interval that should be processed by a model. Can be a datetime string, epoch time in milliseconds, or a relative datetime such as `1 year ago`. (Default: `yesterday`) | str \| int | N |
| `start` | The date/time that determines the earliest date interval that should be processed by a model. Can be a datetime string, epoch time in milliseconds, or a relative datetime such as `1 year ago`. (Default: `yesterday`). Moving `start` earlier also restates the existing intervals of downstream models for the added range; see [restatement after moving a model's start earlier](../concepts/plans.md#restatement-after-moving-a-models-start-earlier). | str \| int | N |
| `end` | The date/time that determines the latest date interval that should be processed by a model. Can be a datetime string, epoch time in milliseconds, or a relative datetime such as `1 year ago`. | str \| int | N |
| `description` | Description of the model. Automatically registered in the SQL engine's table COMMENT field or equivalent (if supported by the engine). | str | N |
| `column_descriptions` | A key-value mapping of column names to column comments that will be registered in the SQL engine's table COMMENT field (if supported by the engine). Specified as key-value pairs (`column_name = 'column comment'`). If present, [inline column comments](../concepts/models/overview.md#inline-column-comments) will not be registered in the SQL engine. | dict | N |
Expand Down
2 changes: 1 addition & 1 deletion sqlmesh/core/analytics/collector.py
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ def on_plan_apply_start(
"no_gaps": plan.no_gaps,
"forward_only": plan.forward_only,
"ensure_finalized_snapshots": plan.ensure_finalized_snapshots,
"has_restatements": bool(plan.restatements),
"has_restatements": plan.is_restatement,
"directly_modified_count": len(plan.directly_modified_snapshots),
"indirectly_modified_count": len(
{
Expand Down
21 changes: 20 additions & 1 deletion sqlmesh/core/console.py
Original file line number Diff line number Diff line change
Expand Up @@ -2064,12 +2064,31 @@ def _prompt_categorize(
# We are computing dev previews, do not confuse the user by printing out something to do
# with restatements. Dev previews are already highlighted in the backfill step
pass
else:

if not plan.restatements or (
plan.start_extension_restatements and not plan.selected_models_to_restate
):
self.show_environment_difference_summary(
plan.context_diff,
no_diff=no_diff,
)

if plan.start_extension_restatements:
model_fqn_to_snapshot = {s.name: s for s in plan.snapshots.values()}
tree = Tree(
"[bold]Models restated because the start of an upstream model moved earlier:[/bold]\n"
"Their existing intervals for the added date range were computed without the new upstream data"
)
for s_id in sorted(plan.start_extension_restatements):
tree.add(
model_fqn_to_snapshot[s_id.name].display_name(
plan.environment_naming_info,
default_catalog if self.verbosity < Verbosity.VERY_VERBOSE else None,
dialect=self.dialect,
)
)
self._print(tree)

if plan.context_diff.has_changes:
self.show_model_difference_summary(
plan.context_diff,
Expand Down
Loading