Multi-dimensional batch calculation: list of update batch dataset as a cartesian product

docs/examples/Power Flow Example.ipynb

@@ -1146,6 +1146,64 @@
     "print(output_data[ComponentType.line][\"i_from\"])"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "344efbe8",
+   "metadata": {},
+   "source": [
+    "#### Cartesian product of batch datasets\n",
+    "\n",
+    "It is possible to conduct a batch calculation of multiple datasets in form of a cartesian product of their scenarios.\n",
+    "Assume certain batch datasets with N1, N2, N3, ... scenarios. \n",
+    "This would give us $N1 * N2 * N3 * ...$ possible combinations via the cartesian product.\n",
+    "The resultant output data is in flat form and it has dimension of N1 * N2 * N3 with first dataset being the highest\n",
+    "dimension. \n",
+    "This can be beneficial in reducing complexity of implementation of such batch calculation \n",
+    "along with keeping the size of such resultant update_data to a minimum.\n",
+    "\n",
+    "```note\n",
+    "The data validation module mentioned in #validation-(optional) does not support cartesian product of datasets. \n",
+    "It is advised to combine the datasets them manually via numpy to use the validator.\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5de1a2f7",
+   "metadata": {},
+   "source": [
+    "We can combine the time series mutation for all n-1 contingencies together in following way.\n",
+    "Both the batch datasets are passed together in `update_data` in a list."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "id": "d690eeb7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Output data has shape (30, 3)\n",
+      "[0 1 1]\n",
+      "[0 1 1]\n",
+      "[1 0 1]\n",
+      "[1 0 1]\n"
+     ]
+    }
+   ],
+   "source": [
+    "output_data = model.calculate_power_flow(update_data=[n_min_1_mutation_update_specific, time_series_mutation])\n",
+    "print(\"Output data has shape\", output_data[ComponentType.line].shape)\n",
+    "line_output = output_data[ComponentType.line][\"energized\"]\n",
+    "print(line_output[0, :])\n",
+    "print(line_output[1, :])\n",
+    "print(line_output[10, :])\n",
+    "print(line_output[11, :])"
+   ]
+  },
   {
    "attachments": {},
    "cell_type": "markdown",
@@ -1210,7 +1268,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 29,
+   "execution_count": 30,
    "id": "b5f10bae",
    "metadata": {},
    "outputs": [
@@ -1270,7 +1328,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 30,
+   "execution_count": 31,
    "id": "1a221507",
    "metadata": {},
    "outputs": [
@@ -1312,15 +1370,15 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 31,
+   "execution_count": 32,
    "id": "541af620",
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Iteration failed to converge after 20 iterations! Max deviation: 3.54512e-16, error tolerance: 1e-20.\n",
+      "Iteration failed to converge after 20 iterations! Max deviation: 3.54512293063893e-16, error tolerance: 1e-20.\n",
       "\n",
       "Try validate_input_data() or validate_batch_data() to validate your data.\n",
       "\n"
@@ -1360,7 +1418,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 32,
+   "execution_count": 33,
    "id": "20d8285c",
    "metadata": {},
    "outputs": [],
@@ -1371,7 +1429,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 33,
+   "execution_count": 34,
    "id": "b702eb15",
    "metadata": {},
    "outputs": [
@@ -1384,7 +1442,7 @@
       "\n",
       "Failed scenarios: [3 7]\n",
       "Succeeded scenarios: [0 1 2 4 5 6 8 9]\n",
-      "Error messages: ['The id cannot be found: 1000\\n', 'Sparse matrix error, possibly singular matrix!\\nIf you get this error from state estimation, it might mean the system is not fully observable, i.e. not enough measurements.\\nIt might also mean that you are running into a corner case where PGM cannot resolve yet.See https://github.com/PowerGridModel/power-grid-model/issues/864.']\n"
+      "Error messages: ['The id cannot be found: 1000\\n', 'Sparse matrix error, possibly singular matrix!\\nIf you get this error from state estimation, it might mean the system is not fully observable, i.e. not enough measurements.\\nIt might also mean that you are running into a corner case where PGM cannot resolve yet.\\nSee https://github.com/PowerGridModel/power-grid-model/issues/864.']\n"
      ]
     }
    ],
@@ -1406,7 +1464,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 34,
+   "execution_count": 35,
    "id": "1ba71901",
    "metadata": {},
    "outputs": [
@@ -1415,16 +1473,16 @@
      "output_type": "stream",
      "text": [
       "Node data with invalid results\n",
-      "[[ 9.99401170e-001  9.92685785e-001  9.94521366e-001]\n",
-      " [ 9.99347687e-001  9.86226389e-001  9.89352855e-001]\n",
-      " [ 9.99288384e-001  9.79654011e-001  9.84095542e-001]\n",
-      " [-2.66881060e+116  2.33997016e-302  6.70346672e-198]\n",
-      " [ 9.99151380e-001  9.66149483e-001  9.73298790e-001]\n",
-      " [ 9.99073166e-001  9.59205860e-001  9.67750710e-001]\n",
-      " [ 9.98988099e-001  9.52126208e-001  9.62096474e-001]\n",
-      " [-2.44756775e+092  5.35663612e-256  1.91838796e-203]\n",
-      " [ 9.98796126e-001  9.37530046e-001  9.50447962e-001]\n",
-      " [ 9.98688504e-001  9.29997471e-001  9.44441670e-001]]\n",
+      "[[9.99401170e-001 9.92685785e-001 9.94521366e-001]\n",
+      " [9.99347687e-001 9.86226389e-001 9.89352855e-001]\n",
+      " [9.99288384e-001 9.79654011e-001 9.84095542e-001]\n",
+      " [2.18565566e-312 4.89761332e-310 2.97079411e-313]\n",
+      " [9.99151380e-001 9.66149483e-001 9.73298790e-001]\n",
+      " [9.99073166e-001 9.59205860e-001 9.67750710e-001]\n",
+      " [9.98988099e-001 9.52126208e-001 9.62096474e-001]\n",
+      " [4.89761332e-310 4.89761332e-310 4.89761332e-310]\n",
+      " [9.98796126e-001 9.37530046e-001 9.50447962e-001]\n",
+      " [9.98688504e-001 9.29997471e-001 9.44441670e-001]]\n",
       "Node data with only valid results\n",
       "[[0.99940117 0.99268579 0.99452137]\n",
       " [0.99934769 0.98622639 0.98935286]\n",
@@ -1467,7 +1525,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "venv",
+   "display_name": "power-grid-model",
    "language": "python",
    "name": "python3"
   },
@@ -1481,7 +1539,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.13.3"
+   "version": "3.14.2"
   }
  },
  "nbformat": 4,

docs/user_manual/calculations.md

@@ -1092,6 +1092,28 @@ for component_idx, scenario in enumerate(line_update):
 independent_update_data = {'line': line_update}
 ```
 
+### Cartesian product of Batch Datasets
+
+Consider an example of running a contingency analysis with a timeseries data.
+Or maybe probablistic data along with timeseries data.
+In such simulations, it is required to perform a loadflow on a cartesian product of situations.
+This is possible to do via providing the `update_data` with a list of multiple batch datasets.
+ie. a list[{py:class}`BatchDataset <power_grid_model.data_types.BatchDataset>`]
+The datasets can be of row based or columnar format.
+The output of such calculation would be flattened with dimension $scenarios * components$.
+
+#### Example: Cartesian product of datasets
+
+```py
+# 5 scenarios of timeseries
+load_update = initialize_array('update', 'sym_load', (5, 1))
+# (Fill load_update)
+line_update = initialize_array('update', 'line', (3, 1))
+# (Fill line_update)
+
+product_update_data = [{'line': load_update}, {'sym_load': line_udpate }]
+```
+
 ### Parallel Computing
 
 The batch calculation supports shared memory multi-threading parallel computing.

docs/user_manual/data-validator.md

@@ -13,14 +13,6 @@ An alternative approach would be to validate only when an exception is raised, b
 will raise exceptions, most of them wil just yield invalid results without warning.
 The main validation functions and classes can be included from `power_grid_model.validation`.
 
-Two helper type definitions are used throughout the validation functions, `InputData` and `UpdateData`.
-They are not special types or classes, but merely type hinting aliases:
-
-```python
-InputData = dict[str, np.ndarray]
-UpdateData = dict[str, np.ndarray | dict[str, np.ndarray]]
-```
-
 ```{seealso}
 Check the [example](examples/Validation%20Examples.ipynb) for an example of function applications.
 ```
@@ -49,11 +41,13 @@ class ValidationError:
 ### Manual validation
 
 The validation functions below can be used to validate input/batch data manually.
-The functions require `input_data: InputData`, which is power-grid-model input data, and `symmetric: bool`, stating if
+The functions require `input_data` of {py:class}`SingleDataset <power_grid_model.data_types.SingleDataset>` type,
+which is power-grid-model input data, and `symmetric: bool`, stating if
 the data will be used for symmetric or asymmetric calculations.
 `calculation_type: CalculationType` is optional and can be supplied to allow missing values for unused fields; see the
 [API reference](../api_reference/python-api-reference.md#enum) for more information.
-To validate update/batch data `update_data: UpdateData`, power-grid-model update data, should also be supplied.
+To validate update/batch data `update_data` of {py:class}`BatchDataset <power_grid_model.data_types.BatchDataset>` type,
+power-grid-model update data, should also be supplied.
 
 - `validate_input_data(input_data, calculation_type, symmetric) -> list[ValidationError]` validates input_data.
 - `validate_batch_data(input_data, update_data, calculation_type, symmetric) -> dict[int, list[ValidationError]]`
@@ -68,6 +62,10 @@ In such cases, the latter is leading when only running batch calculations.
 Running single calculations on an incomplete input data set is, of course, unsupported.
 ```
 
+Validating a cartesian product of datasets used in PGM's `update_data` via providing it with `list[BatchDataset]`is
+not straightforward.
+User should convert such multiple dataset into a single flat batch dataset.
+
 ### Assertions
 
 Instead of manual validation it is possible to use the assertion functions below to assert valid data.

docs/user_manual/performance-guide.md

@@ -53,6 +53,15 @@ If you are running on a system where memory is the bottle-neck, using a columnar
 footprint.
 This may or may not induce a slight computational overhead during calculations.
 
+Some simulations might require a cartesian product of scenarios of two batch datasets.
+This can be done by passing them to `update_data` as a list
+ie. a list[{py:class}`BatchDataset <power_grid_model.data_types.BatchDataset>`]
+(Check [Power Flow Example](../examples/Power%20Flow%20Example.ipynb)).
+This gets treated as a cartesian product of the provided datasets and the combination of scenarios gets handled
+internally.
+Hence there is no need to allocate memory for full `N1 * N2 * ...` scenarios for a cartesian product of data sets with
+scenario size `N1, N2, ...`.
+
 ### Output data volume
 
 For most use cases, only certain output values are relevant.

power_grid_model_c/power_grid_model/include/power_grid_model/auxiliary/dataset.hpp

@@ -287,6 +287,9 @@ template <dataset_type_tag dataset_type_> class Dataset {
     }
     constexpr bool is_dense(Idx const i) const { return is_dense(buffers_[i]); }
     constexpr bool is_dense(Buffer const& buffer) const { return buffer.indptr.empty(); }
+    constexpr bool is_dense() const {
+        return std::ranges::all_of(buffers_, [this](Buffer const& buffer) { return is_dense(buffer); });
+    }
     constexpr bool is_sparse(std::string_view component, bool with_attribute_buffers = false) const {
         Idx const idx = find_component(component, false);
         if (idx == invalid_index) {
@@ -510,10 +513,52 @@ template <dataset_type_tag dataset_type_> class Dataset {
         return result;
     }
 
+    // get slice dataset from batch
+    Dataset get_slice_scenario(Idx begin, Idx end) const
+        requires(!is_indptr_mutable_v<dataset_type>)
+    {
+        assert(begin <= end);
+        assert(0 <= begin);
+        assert(end <= batch_size());
+        assert(is_batch());
+        assert(is_dense());
+
+        // empty slice
+        if (begin == end) {
+            Dataset result{true, 0, dataset_info_.dataset->name, *meta_data_};
+            result.add_buffer("node", 0, 0, nullptr, nullptr);
+            return result;
+        }
+
+        // start with begin
+        Dataset result = get_individual_scenario(begin);
+        Idx const batch_size = end - begin;
+        result.dataset_info_.is_batch = true;
+        result.dataset_info_.batch_size = batch_size;
+        for (auto& component_info : result.dataset_info_.component_info) {
+            Idx const size = component_info.elements_per_scenario * batch_size;
+            component_info.total_elements = size;
+        }
+        return result;
+    }
+
+    void set_next_cartesian_product_dimension(Dataset const* next) {
+        Dataset const* current = next;
+        while (current != nullptr) {
+            if (this == current) {
+                throw DatasetError{"Cannot create cyclic cartesian product dimension linked list!\n"};
+            }
+            current = current->get_next_cartesian_product_dimension();
+        }
+        next_ = next;
+    }
+    Dataset const* get_next_cartesian_product_dimension() const { return next_; }
+
   private:
     MetaData const* meta_data_;
     DatasetInfo dataset_info_;
     std::vector<Buffer> buffers_;
+    Dataset const* next_{};
 
     std::span<Indptr> get_indptr_span(Indptr* indptr) const {
         return std::span{indptr, static_cast<size_t>(batch_size() + 1)};

power_grid_model_c/power_grid_model_c/include/power_grid_model_c/dataset.h

@@ -212,6 +212,22 @@ PGM_API void PGM_dataset_const_add_attribute_buffer(PGM_Handle* handle, PGM_Cons
  */
 PGM_API PGM_DatasetInfo const* PGM_dataset_const_get_info(PGM_Handle* handle, PGM_ConstDataset const* dataset);
 
+/**
+ * @brief Set the next const dataset as cartesian product dimension.
+ *
+ * This function allows users to run a batch calculation with multiple dimensions of scenarios.
+ * The way users can archive this is to combine multiple batch datasets
+ * to create a multi-dimension batch calculation using a linked list pattern. The calculation core will
+ * interpret the combined dataset as a cartesian product on a linked list of all the scenarios.
+ * Each batch dataset in the linked list represents one dimension of the cartesian product.
+ *
+ * @param handle
+ * @param dataset
+ * @param next_dataset The next dataset in the linked list.
+ */
+PGM_API void PGM_dataset_const_set_next_cartesian_product_dimension(PGM_Handle* handle, PGM_ConstDataset* dataset,
+                                                                    PGM_ConstDataset const* next_dataset);
+
 /**
  * @brief Get the dataset info of the instance PGM_WritableDataset.
  * @param handle

power_grid_model_c/power_grid_model_c/include/power_grid_model_c/model.h

@@ -91,6 +91,11 @@ PGM_API void PGM_get_indexer(PGM_Handle* handle, PGM_PowerGridModel const* model
  * If batch_dataset == NULL, it is a one-time calculation.
  * If batch_dataset != NULL, it is a batch calculation with batch update in the batch_dataset.
  *
+ * The user can use the function set_next_cartesian_product_dimension() to combine multiple batch datasets
+ * to create a multi-dimension batch calculation using a linked list pattern. The calculation core will
+ * interpret the combined dataset as a cartesian product on a linked list of all the scenarios.
+ * Each batch dataset in the linked list represents one dimension of the cartesian product.
+ *
  * You need to pre-allocate all output buffer.
  *
  * Use PGM_error_code() and PGM_error_message() to check the error.

power_grid_model_c/power_grid_model_c/src/dataset.cpp

@@ -101,6 +101,13 @@ void PGM_dataset_const_add_attribute_buffer(PGM_Handle* handle, PGM_ConstDataset
         PGM_regular_error);
 }
 
+void PGM_dataset_const_set_next_cartesian_product_dimension(PGM_Handle* handle, PGM_ConstDataset* dataset,
+                                                            PGM_ConstDataset const* next_dataset) {
+    call_with_catch(
+        handle, [dataset, next_dataset]() { dataset->set_next_cartesian_product_dimension(next_dataset); },
+        PGM_regular_error);
+}
+
 PGM_DatasetInfo const* PGM_dataset_const_get_info(PGM_Handle* /*unused*/, PGM_ConstDataset const* dataset) {
     return &dataset->get_description();
 }