Skip to content

add load sql to data import in 13x #933

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leto-b wants to merge 1 commit into
base: main
Choose a base branch
from
+188 −8
Open

add load sql to data import in 13x #933

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 47 additions & 2 deletions src/UserGuide/V1.3.x/Tools-System/Data-Import-Tool-1-3-4.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Data Import

## 1. Overview
IoTDB supports two methods for data import:
IoTDB supports three methods for data import:
* Data Import Tool: Use the import-data.sh (Unix/OS X) or import-data.bat (Windows) script in the tools directory to manually import CSV, SQL, or TsFile (open-source time-series file format) data into IoTDB.
* TsFile Auto-Loading Feature
* Load `TsFile` SQL

<table style="text-align: left;">
<tbody>
Expand All @@ -21,13 +22,17 @@ IoTDB supports two methods for data import:
<td>Can be used for single or batch import of SQL files into IoTDB</td>
</tr>
<tr>
<td rowspan="2">TsFile</td>
<td rowspan="3">TsFile</td>
<td>Can be used for single or batch import of TsFile files into IoTDB</td>
</tr>
<tr>
<td>TsFile Auto-Loading Feature</td>
<td>Can automatically monitor a specified directory for newly generated TsFiles and load them into IoTDB.</td>
</tr>
<tr>
<td>Load SQL</td>
<td>Can be used for single or batch import of TsFile files into IoTDB</td>
</tr>
</tbody>
</table>

Expand Down Expand Up @@ -242,3 +247,43 @@ Add the following parameters to `iotdb-system.properties` (template: `iotdb-syst
2. ​​**Restricted Directories**​: Do NOT set Pipe receiver directories, data directories, or other system paths as monitored directories.
3. ​​**Directory Conflicts**​: Ensure `load_active_listening_fail_dir` does not overlap with `load_active_listening_dirs` or its subdirectories.
4. ​​**Permissions**​: The monitored directory must have write permissions. Files are deleted after successful loading; insufficient permissions may cause duplicate loading.

## 4. Load SQL

IoTDB supports importing one or multiple TsFile files containing time series into another running IoTDB instance directly via SQL execution through the CLI.

### 4.1 Command

```SQL
load '<path/dir>' with (
'attribute-key1'='attribute-value1',
'attribute-key2'='attribute-value2',
)
```

* `<path/dir>` : The path to a TsFile or a folder containing multiple TsFiles.
* `<attributes>`: Optional parameters, as described below.

| Key | Key Description | Value Type | Value Range | Value is Required | Default Value |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|--------------------------------|-------------------|----------------------------|
| `database-level` | When the database corresponding to the TsFile does not exist, the database hierarchy level can be specified via the ` database-level` parameter. The default is the level set in `iotdb-common.properties`. For example, setting level=1 means the prefix path of level 1 in all time series in the TsFile will be used as the database. | Integer | `[1: Integer.MAX_VALUE]` | No | 1 |
| `on-success` | Action for successfully loaded TsFiles: `delete` (delete the TsFile after successful import) or `none` (retain the TsFile in the source folder). | String | `delete / none` | No | delete |
| `convert-on-type-mismatch` | Whether to perform type conversion during loading if data types in the TsFile mismatch the target schema. | Boolean | `true / false` | No | true |
| `verify` | Whether to validate the schema before loading the TsFile. | Boolean | `true / false` | No | true |
| `tablet-conversion-threshold` | Size threshold (in bytes) for converting TsFiles into tablet format during loading. Default: `-1` (no conversion for any TsFile).(Available since v1.3.6) | Integer | `[-1,0 :`​`Integer.MAX_VALUE]` | No | -1 |

### 4.2 Example

```SQL
-- Import tsfile by excuting load sql
IoTDB> load '/home/dump1.tsfile' with ( 'on-success'='none')
Msg: The statement is executed successfully.

-- Verify whether the import was successful
IoTDB> select * from root.testdb.**
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
| Time|root.testdb.device.model.temperature|root.testdb.device.model.humidity|root.testdb.device.model.status|
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
|2025-04-17T10:35:47.218+08:00| 22.3| 19.4| true|
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
```
49 changes: 47 additions & 2 deletions src/UserGuide/dev-1.3/Tools-System/Data-Import-Tool-1-3-4.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Data Import

## 1. Overview
IoTDB supports two methods for data import:
IoTDB supports three methods for data import:
* Data Import Tool: Use the import-data.sh (Unix/OS X) or import-data.bat (Windows) script in the tools directory to manually import CSV, SQL, or TsFile (open-source time-series file format) data into IoTDB.
* TsFile Auto-Loading Feature
* Load `TsFile` SQL

<table style="text-align: left;">
<tbody>
Expand All @@ -21,13 +22,17 @@ IoTDB supports two methods for data import:
<td>Can be used for single or batch import of SQL files into IoTDB</td>
</tr>
<tr>
<td rowspan="2">TsFile</td>
<td rowspan="3">TsFile</td>
<td>Can be used for single or batch import of TsFile files into IoTDB</td>
</tr>
<tr>
<td>TsFile Auto-Loading Feature</td>
<td>Can automatically monitor a specified directory for newly generated TsFiles and load them into IoTDB.</td>
</tr>
<tr>
<td>Load SQL</td>
<td>Can be used for single or batch import of TsFile files into IoTDB</td>
</tr>
</tbody>
</table>

Expand Down Expand Up @@ -242,3 +247,43 @@ Add the following parameters to `iotdb-system.properties` (template: `iotdb-syst
2. ​​**Restricted Directories**​: Do NOT set Pipe receiver directories, data directories, or other system paths as monitored directories.
3. ​​**Directory Conflicts**​: Ensure `load_active_listening_fail_dir` does not overlap with `load_active_listening_dirs` or its subdirectories.
4. ​​**Permissions**​: The monitored directory must have write permissions. Files are deleted after successful loading; insufficient permissions may cause duplicate loading.

## 4. Load SQL

IoTDB supports importing one or multiple TsFile files containing time series into another running IoTDB instance directly via SQL execution through the CLI.

### 4.1 Command

```SQL
load '<path/dir>' with (
'attribute-key1'='attribute-value1',
'attribute-key2'='attribute-value2',
)
```

* `<path/dir>` : The path to a TsFile or a folder containing multiple TsFiles.
* `<attributes>`: Optional parameters, as described below.

| Key | Key Description | Value Type | Value Range | Value is Required | Default Value |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|--------------------------------|-------------------|----------------------------|
| `database-level` | When the database corresponding to the TsFile does not exist, the database hierarchy level can be specified via the ` database-level` parameter. The default is the level set in `iotdb-common.properties`. For example, setting level=1 means the prefix path of level 1 in all time series in the TsFile will be used as the database. | Integer | `[1: Integer.MAX_VALUE]` | No | 1 |
| `on-success` | Action for successfully loaded TsFiles: `delete` (delete the TsFile after successful import) or `none` (retain the TsFile in the source folder). | String | `delete / none` | No | delete |
| `convert-on-type-mismatch` | Whether to perform type conversion during loading if data types in the TsFile mismatch the target schema. | Boolean | `true / false` | No | true |
| `verify` | Whether to validate the schema before loading the TsFile. | Boolean | `true / false` | No | true |
| `tablet-conversion-threshold` | Size threshold (in bytes) for converting TsFiles into tablet format during loading. Default: `-1` (no conversion for any TsFile).(Available since v1.3.6) | Integer | `[-1,0 :`​`Integer.MAX_VALUE]` | No | -1 |

### 4.2 Example

```SQL
-- Import tsfile by excuting load sql
IoTDB> load '/home/dump1.tsfile' with ( 'on-success'='none')
Msg: The statement is executed successfully.

-- Verify whether the import was successful
IoTDB> select * from root.testdb.**
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
| Time|root.testdb.device.model.temperature|root.testdb.device.model.humidity|root.testdb.device.model.status|
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
|2025-04-17T10:35:47.218+08:00| 22.3| 19.4| true|
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
```
49 changes: 47 additions & 2 deletions src/zh/UserGuide/V1.3.x/Tools-System/Data-Import-Tool-1-3-4.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@

## 1. 功能概述

IoTDB 支持两种方式进行数据导入
IoTDB 支持三种方式进行数据导入

* 数据导入工具：tools 目录下的手动数据导入工具 `import-data.sh/bat`，可以将 CSV、SQL、及TsFile（开源时序文件格式）的数据导入 IoTDB。
* TsFile 自动加载功能
* `Load SQL` 导入 TsFile

<table style="text-align: left;">
<tbody>
Expand All @@ -23,13 +24,17 @@ IoTDB 支持两种方式进行数据导入
<td>可用于单个或一个目录的 SQL 文件批量导入 IoTDB</td>
</tr>
<tr>
<td rowspan="2">TsFile</td>
<td rowspan="3">TsFile</td>
<td>可用于单个或一个目录的 TsFile 文件批量导入 IoTDB</td>
</tr>
<tr>
<td>TsFile 自动加载功能</td>
<td> 可以监听指定路径下新产生的TsFile文件，并将其加载进IoTDB</td>
</tr>
<tr>
<td>Load SQL</td>
<td>可用于单个或一个目录的 TsFile 文件批量导入 IoTDB</td>
</tr>
</tbody>
</table>

Expand Down Expand Up @@ -249,3 +254,43 @@ error: Invalid thread number '0'. Please set a positive integer.
2. 禁止设置 Pipe 的 receiver 目录、存放数据的 data 目录等作为监听目录
3. 禁止 `load_active_listening_fail_dir` 与 `load_active_listening_dirs` 存在相同的目录，或者互相嵌套
4. 保证 `load_active_listening_dirs` 目录有足够的权限，在加载成功之后，文件将会被删除，如果没有删除权限，则会重复加载

## 4. Load SQL

IoTDB 支持通过 CLI 执行 SQL 直接将存有时间序列的一个或多个 TsFile 文件导入到另外一个正在运行的 IoTDB 实例中。

### 4.1 运行命令

```SQL
load '<path/dir>' with (
'attribute-key1'='attribute-value1',
'attribute-key2'='attribute-value2',
)
```

* `<path/dir>` ：文件本身，或是包含若干文件的文件夹路径
* `<attributes>`：可选参数，具体如下表所示

| Key | Key 描述 | Value 类型 | Value 取值范围 | Value 是否必填 | Value 默认值 |
| --------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------------ | ----------------------------------------- | ---------------- | -------------------------- |
| `database-level` | 当 tsfile 对应的 database 不存在时，可以通过` database-level`参数的值来制定 database 的级别，默认为`iotdb-common.properties`中设置的级别。<br>例如当设置 level 参数为 1 时表明此 tsfile 中所有时间序列中层级为1的前缀路径是 database。 | Integer | `[1: Integer.MAX_VALUE]` | 否 | 1 |
| `on-success` | 表示对于成功载入的 tsfile 的处置方式:默认为`delete`，即tsfile 成功加载后将被删除;`none `表明 tsfile 成功加载之后依然被保留在源文件夹, | String | `delete / none` | 否 | delete |
| `convert-on-type-mismatch` | 加载 tsfile 时，如果数据类型不一致，是否进行转换 | Boolean | `true / false` | 否 | true |
| `verify` | 加载 tsfile 前是否校验 schema | Boolean | `true / false` | 否 | true |
| `tablet-conversion-threshold` | 转换为 tablet 形式的 tsfile 大小阈值，针对小文件 tsfile 加载，采用将其转换为 tablet 形式进行写入:默认值为 -1，即任意大小 tsfile 都不进行转换 （V1.3.6及以后的V1.x版本支持） | Integer | `[-1,0 :`​`Integer.MAX_VALUE]` | 否 | -1 |

### 4.2 运行示例

```SQL
-- 通过load sql 导入 tsfile
IoTDB> load '/home/dump1.tsfile' with ( 'on-success'='none')
Msg: The statement is executed successfully.

-- 验证数据导入成功
IoTDB> select * from root.testdb.**
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
| Time|root.testdb.device.model.temperature|root.testdb.device.model.humidity|root.testdb.device.model.status|
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
|2025-04-17T10:35:47.218+08:00| 22.3| 19.4| true|
+-----------------------------+------------------------------------+---------------------------------+-------------------------------+
```
Loading