Multi-dimensional batch calculation: documentation

docs/examples/Power Flow Example.ipynb

@@ -1146,6 +1146,64 @@
     "print(output_data[ComponentType.line][\"i_from\"])"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "344efbe8",
+   "metadata": {},
+   "source": [
+    "#### Cartesian product of batch datasets\n",
+    "\n",
+    "It is possible to conduct a batch calculation of multiple datasets in form of a cartesian product of their scenarios.\n",
+    "Assume certain batch datasets with N1, N2, N3, ... scenarios. \n",
+    "This would give us $N1 * N2 * N3 * ...$ possible combinations via the cartesian product.\n",
+    "The resultant output data is in flat form and it has dimension of N1 * N2 * N3 with first dataset being the highest\n",
+    "dimension. \n",
+    "This can be beneficial in reducing complexity of implementation of such batch calculation \n",
+    "along with keeping the size of such resultant update_data to a minimum.\n",
+    "\n",
+    "```note\n",
+    "The data validation module mentioned in #validation-(optional) does not support cartesian product of datasets. \n",
+    "It is advised to combine the datasets them manually via numpy to use the validator.\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5de1a2f7",
+   "metadata": {},
+   "source": [
+    "We can combine the time series mutation for all n-1 contingencies together in following way.\n",
+    "Both the batch datasets are passed together in `update_data` in a list."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "id": "d690eeb7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Output data has shape (30, 3)\n",
+      "[0 1 1]\n",
+      "[0 1 1]\n",
+      "[1 0 1]\n",
+      "[1 0 1]\n"
+     ]
+    }
+   ],
+   "source": [
+    "output_data = model.calculate_power_flow(update_data=[n_min_1_mutation_update_specific, time_series_mutation])\n",
+    "print(\"Output data has shape\", output_data[ComponentType.line].shape)\n",
+    "line_output = output_data[ComponentType.line][\"energized\"]\n",
+    "print(line_output[0, :])\n",
+    "print(line_output[1, :])\n",
+    "print(line_output[10, :])\n",
+    "print(line_output[11, :])"
+   ]
+  },
   {
    "attachments": {},
    "cell_type": "markdown",
@@ -1210,7 +1268,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 29,
+   "execution_count": 30,
    "id": "b5f10bae",
    "metadata": {},
    "outputs": [
@@ -1270,7 +1328,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 30,
+   "execution_count": 31,
    "id": "1a221507",
    "metadata": {},
    "outputs": [
@@ -1312,15 +1370,15 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 31,
+   "execution_count": 32,
    "id": "541af620",
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Iteration failed to converge after 20 iterations! Max deviation: 3.54512e-16, error tolerance: 1e-20.\n",
+      "Iteration failed to converge after 20 iterations! Max deviation: 3.54512293063893e-16, error tolerance: 1e-20.\n",
       "\n",
       "Try validate_input_data() or validate_batch_data() to validate your data.\n",
       "\n"
@@ -1360,7 +1418,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 32,
+   "execution_count": 33,
    "id": "20d8285c",
    "metadata": {},
    "outputs": [],
@@ -1371,7 +1429,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 33,
+   "execution_count": 34,
    "id": "b702eb15",
    "metadata": {},
    "outputs": [
@@ -1384,7 +1442,7 @@
       "\n",
       "Failed scenarios: [3 7]\n",
       "Succeeded scenarios: [0 1 2 4 5 6 8 9]\n",
-      "Error messages: ['The id cannot be found: 1000\\n', 'Sparse matrix error, possibly singular matrix!\\nIf you get this error from state estimation, it might mean the system is not fully observable, i.e. not enough measurements.\\nIt might also mean that you are running into a corner case where PGM cannot resolve yet.See https://github.com/PowerGridModel/power-grid-model/issues/864.']\n"
+      "Error messages: ['The id cannot be found: 1000\\n', 'Sparse matrix error, possibly singular matrix!\\nIf you get this error from state estimation, it might mean the system is not fully observable, i.e. not enough measurements.\\nIt might also mean that you are running into a corner case where PGM cannot resolve yet.\\nSee https://github.com/PowerGridModel/power-grid-model/issues/864.']\n"
      ]
     }
    ],
@@ -1406,7 +1464,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 34,
+   "execution_count": 35,
    "id": "1ba71901",
    "metadata": {},
    "outputs": [
@@ -1415,16 +1473,16 @@
      "output_type": "stream",
      "text": [
       "Node data with invalid results\n",
-      "[[ 9.99401170e-001  9.92685785e-001  9.94521366e-001]\n",
-      " [ 9.99347687e-001  9.86226389e-001  9.89352855e-001]\n",
-      " [ 9.99288384e-001  9.79654011e-001  9.84095542e-001]\n",
-      " [-2.66881060e+116  2.33997016e-302  6.70346672e-198]\n",
-      " [ 9.99151380e-001  9.66149483e-001  9.73298790e-001]\n",
-      " [ 9.99073166e-001  9.59205860e-001  9.67750710e-001]\n",
-      " [ 9.98988099e-001  9.52126208e-001  9.62096474e-001]\n",
-      " [-2.44756775e+092  5.35663612e-256  1.91838796e-203]\n",
-      " [ 9.98796126e-001  9.37530046e-001  9.50447962e-001]\n",
-      " [ 9.98688504e-001  9.29997471e-001  9.44441670e-001]]\n",
+      "[[9.99401170e-001 9.92685785e-001 9.94521366e-001]\n",
+      " [9.99347687e-001 9.86226389e-001 9.89352855e-001]\n",
+      " [9.99288384e-001 9.79654011e-001 9.84095542e-001]\n",
+      " [2.18565566e-312 4.89761332e-310 2.97079411e-313]\n",
+      " [9.99151380e-001 9.66149483e-001 9.73298790e-001]\n",
+      " [9.99073166e-001 9.59205860e-001 9.67750710e-001]\n",
+      " [9.98988099e-001 9.52126208e-001 9.62096474e-001]\n",
+      " [4.89761332e-310 4.89761332e-310 4.89761332e-310]\n",
+      " [9.98796126e-001 9.37530046e-001 9.50447962e-001]\n",
+      " [9.98688504e-001 9.29997471e-001 9.44441670e-001]]\n",
       "Node data with only valid results\n",
       "[[0.99940117 0.99268579 0.99452137]\n",
       " [0.99934769 0.98622639 0.98935286]\n",
@@ -1467,7 +1525,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "venv",
+   "display_name": "power-grid-model",
    "language": "python",
    "name": "python3"
   },
@@ -1481,7 +1539,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.13.3"
+   "version": "3.14.2"
   }
  },
  "nbformat": 4,

docs/user_manual/calculations.md

@@ -1092,6 +1092,28 @@ for component_idx, scenario in enumerate(line_update):
 independent_update_data = {'line': line_update}
 ```
 
+### Cartesian product of Batch Datasets
+
+Consider an example of running a contingency analysis with a timeseries data.
+Or maybe probablistic data along with timeseries data.
+In such simulations, it is required to perform a loadflow on a cartesian product of situations.
+This is possible to do via providing the `update_data` with a list of multiple batch datasets.
+ie. a list[{py:class}`BatchDataset <power_grid_model.data_types.BatchDataset>`]
+The datasets can be of row based or columnar format.
+The output of such calculation would be flattened with dimension $scenarios * components$.
+
+#### Example: Cartesian product of datasets
+
+```py
+# 5 scenarios of timeseries
+load_update = initialize_array('update', 'sym_load', (5, 1))
+# (Fill load_update)
+line_update = initialize_array('update', 'line', (3, 1))
+# (Fill line_update)
+
+product_update_data = [{'line': load_update}, {'sym_load': line_udpate }]
+```
+
 ### Parallel Computing
 
 The batch calculation supports shared memory multi-threading parallel computing.

docs/user_manual/data-validator.md

@@ -13,14 +13,6 @@ An alternative approach would be to validate only when an exception is raised, b
 will raise exceptions, most of them wil just yield invalid results without warning.
 The main validation functions and classes can be included from `power_grid_model.validation`.
 
-Two helper type definitions are used throughout the validation functions, `InputData` and `UpdateData`.
-They are not special types or classes, but merely type hinting aliases:
-
-```python
-InputData = dict[str, np.ndarray]
-UpdateData = dict[str, np.ndarray | dict[str, np.ndarray]]
-```
-
 ```{seealso}
 Check the [example](examples/Validation%20Examples.ipynb) for an example of function applications.
 ```
@@ -49,11 +41,13 @@ class ValidationError:
 ### Manual validation
 
 The validation functions below can be used to validate input/batch data manually.
-The functions require `input_data: InputData`, which is power-grid-model input data, and `symmetric: bool`, stating if
+The functions require `input_data` of {py:class}`SingleDataset <power_grid_model.data_types.SingleDataset>` type,
+which is power-grid-model input data, and `symmetric: bool`, stating if
 the data will be used for symmetric or asymmetric calculations.
 `calculation_type: CalculationType` is optional and can be supplied to allow missing values for unused fields; see the
 [API reference](../api_reference/python-api-reference.md#enum) for more information.
-To validate update/batch data `update_data: UpdateData`, power-grid-model update data, should also be supplied.
+To validate update/batch data `update_data` of {py:class}`BatchDataset <power_grid_model.data_types.BatchDataset>` type,
+power-grid-model update data, should also be supplied.
 
 - `validate_input_data(input_data, calculation_type, symmetric) -> list[ValidationError]` validates input_data.
 - `validate_batch_data(input_data, update_data, calculation_type, symmetric) -> dict[int, list[ValidationError]]`
@@ -68,6 +62,10 @@ In such cases, the latter is leading when only running batch calculations.
 Running single calculations on an incomplete input data set is, of course, unsupported.
 ```
 
+Validating a cartesian product of datasets used in PGM's `update_data` via providing it with `list[BatchDataset]`is
+not straightforward.
+User should convert such multiple dataset into a single flat batch dataset.
+
 ### Assertions
 
 Instead of manual validation it is possible to use the assertion functions below to assert valid data.

docs/user_manual/performance-guide.md

@@ -53,6 +53,15 @@ If you are running on a system where memory is the bottle-neck, using a columnar
 footprint.
 This may or may not induce a slight computational overhead during calculations.
 
+Some simulations might require a cartesian product of scenarios of two batch datasets.
+This can be done by passing them to `update_data` as a list
+ie. a list[{py:class}`BatchDataset <power_grid_model.data_types.BatchDataset>`]
+(Check [Power Flow Example](../examples/Power%20Flow%20Example.ipynb)).
+This gets treated as a cartesian product of the provided datasets and the combination of scenarios gets handled
+internally.
+Hence there is no need to allocate memory for full `N1 * N2 * ...` scenarios for a cartesian product of data sets with
+scenario size `N1, N2, ...`.
+
 ### Output data volume
 
 For most use cases, only certain output values are relevant.