Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 23 additions & 3 deletions docs/34.0.0/ingestion/data-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -964,12 +964,11 @@ Each line can be further parsed using [`parseSpec`](#parsespec).

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::


:::info
You need to include [`druid-avro-extensions`](../development/extensions-core/avro.md) as an extension to use the Avro Hadoop Parser.

Expand Down Expand Up @@ -1031,6 +1030,13 @@ For example, using Avro Hadoop parser with custom reader's schema file:

### ORC Hadoop Parser

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::


:::info
You need to include the [`druid-orc-extensions`](../development/extensions-core/orc.md) as an extension to use the ORC Hadoop Parser.
:::
Expand Down Expand Up @@ -1276,6 +1282,13 @@ setting `"mapreduce.job.user.classpath.first": "true"`, then this will not be an

### Parquet Hadoop Parser

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::


:::info
You need to include the [`druid-parquet-extensions`](../development/extensions-core/parquet.md) as an extension to use the Parquet Hadoop Parser.
:::
Expand Down Expand Up @@ -1420,6 +1433,13 @@ However, the Parquet Avro Hadoop Parser was the original basis for supporting th

### Parquet Avro Hadoop Parser

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::


:::info
Consider using the [Parquet Hadoop Parser](#parquet-hadoop-parser) over this parser to ingest
Parquet files. See [Parquet Hadoop Parser vs Parquet Avro Hadoop Parser](#parquet-hadoop-parser-vs-parquet-avro-hadoop-parser)
Expand Down
4 changes: 1 addition & 3 deletions docs/34.0.0/ingestion/hadoop.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@ sidebar_label: "Hadoop-based"

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::

Expand Down
5 changes: 2 additions & 3 deletions docs/34.0.0/operations/other-hadoop.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,11 @@ title: "Working with different versions of Apache Hadoop"

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion)

:::


Apache Druid can interact with Hadoop in two ways:

1. [Use HDFS for deep storage](../development/extensions-core/hdfs.md) using the druid-hdfs-storage extension.
Expand Down
13 changes: 13 additions & 0 deletions docs/34.0.0/release-info/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,14 @@ This section contains important information about new and existing features.

### Java 11 support

**Update**: Support for Java 11 is scheduled to end with Druid 37.0.0.

Java 11 support has been deprecated since Druid 32.0, and official support will be removed as early as Druid 35.0.0

### Hadoop-based ingestion

**Update**: Support for Hadoop-based ingestion is scheduled to end with Druid 37.0.0.

Hadoop-based ingestion has been deprecated since Druid 32.0 and will be removed as early as Druid 35.0.0.
We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md).

Expand Down Expand Up @@ -372,12 +376,21 @@ Additionally, MSQ task engine metrics now include the following dimensions:

- The task runner log now includes the task id for a job and includes a log before a job is created [#18105](https://github.com/apache/druid/pull/18105)


## Upgrade notes and incompatible changes

### Upgrade notes

#### Java 11 support

**Update**: Support for Java 11 is scheduled to end with Druid 37.0.0.

Java 11 support has been deprecated since Druid 32.0, and official support will be removed as early as Druid 35.0.0

#### Hadoop-based ingestion

**Update**: Support for Hadoop-based ingestion is scheduled to end with Druid 37.0.0.

Hadoop-based ingestion has been deprecated since Druid 32.0 and will be removed as early as Druid 35.0.0.

We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md).
Expand Down
8 changes: 8 additions & 0 deletions docs/34.0.0/release-info/upgrade-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,16 @@ If you're already using this feature, you don't need to take any action.

### Upgrade notes

#### Java 11 support

**Update**: Support for Java 11 is scheduled to end with Druid 37.0.0.

Java 11 support has been deprecated since Druid 32.0, and official support will be removed as early as Druid 35.0.0

#### Hadoop-based ingestion

**Update**: Support for Hadoop-based ingestion is scheduled to end with Druid 37.0.0.

Hadoop-based ingestion has been deprecated since Druid 32.0 and will be removed as early as Druid 35.0.0.

We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md).
Expand Down
5 changes: 2 additions & 3 deletions docs/34.0.0/tutorials/tutorial-batch-hadoop.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,11 @@ sidebar_label: Load from Apache Hadoop

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion)

:::


This tutorial shows you how to load data files into Apache Druid using a remote Hadoop cluster.

For this tutorial, we'll assume that you've already completed the previous
Expand Down
4 changes: 1 addition & 3 deletions docs/34.0.0/tutorials/tutorial-kerberos-hadoop.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@ sidebar_label: Kerberized HDFS deep storage

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion)

:::

Expand Down
26 changes: 23 additions & 3 deletions docs/latest/ingestion/data-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -964,12 +964,11 @@ Each line can be further parsed using [`parseSpec`](#parsespec).

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::


:::info
You need to include [`druid-avro-extensions`](../development/extensions-core/avro.md) as an extension to use the Avro Hadoop Parser.

Expand Down Expand Up @@ -1031,6 +1030,13 @@ For example, using Avro Hadoop parser with custom reader's schema file:

### ORC Hadoop Parser

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::


:::info
You need to include the [`druid-orc-extensions`](../development/extensions-core/orc.md) as an extension to use the ORC Hadoop Parser.
:::
Expand Down Expand Up @@ -1276,6 +1282,13 @@ setting `"mapreduce.job.user.classpath.first": "true"`, then this will not be an

### Parquet Hadoop Parser

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::


:::info
You need to include the [`druid-parquet-extensions`](../development/extensions-core/parquet.md) as an extension to use the Parquet Hadoop Parser.
:::
Expand Down Expand Up @@ -1420,6 +1433,13 @@ However, the Parquet Avro Hadoop Parser was the original basis for supporting th

### Parquet Avro Hadoop Parser

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::


:::info
Consider using the [Parquet Hadoop Parser](#parquet-hadoop-parser) over this parser to ingest
Parquet files. See [Parquet Hadoop Parser vs Parquet Avro Hadoop Parser](#parquet-hadoop-parser-vs-parquet-avro-hadoop-parser)
Expand Down
4 changes: 1 addition & 3 deletions docs/latest/ingestion/hadoop.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@ sidebar_label: "Hadoop-based"

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion).

:::

Expand Down
5 changes: 2 additions & 3 deletions docs/latest/operations/other-hadoop.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,11 @@ title: "Working with different versions of Apache Hadoop"

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion)

:::


Apache Druid can interact with Hadoop in two ways:

1. [Use HDFS for deep storage](../development/extensions-core/hdfs.md) using the druid-hdfs-storage extension.
Expand Down
13 changes: 13 additions & 0 deletions docs/latest/release-info/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,14 @@ This section contains important information about new and existing features.

### Java 11 support

**Update**: Support for Java 11 is scheduled to end with Druid 37.0.0.

Java 11 support has been deprecated since Druid 32.0, and official support will be removed as early as Druid 35.0.0

### Hadoop-based ingestion

**Update**: Support for Hadoop-based ingestion is scheduled to end with Druid 37.0.0.

Hadoop-based ingestion has been deprecated since Druid 32.0 and will be removed as early as Druid 35.0.0.
We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md).

Expand Down Expand Up @@ -372,12 +376,21 @@ Additionally, MSQ task engine metrics now include the following dimensions:

- The task runner log now includes the task id for a job and includes a log before a job is created [#18105](https://github.com/apache/druid/pull/18105)


## Upgrade notes and incompatible changes

### Upgrade notes

#### Java 11 support

**Update**: Support for Java 11 is scheduled to end with Druid 37.0.0.

Java 11 support has been deprecated since Druid 32.0, and official support will be removed as early as Druid 35.0.0

#### Hadoop-based ingestion

**Update**: Support for Hadoop-based ingestion is scheduled to end with Druid 37.0.0.

Hadoop-based ingestion has been deprecated since Druid 32.0 and will be removed as early as Druid 35.0.0.

We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md).
Expand Down
8 changes: 8 additions & 0 deletions docs/latest/release-info/upgrade-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,16 @@ If you're already using this feature, you don't need to take any action.

### Upgrade notes

#### Java 11 support

**Update**: Support for Java 11 is scheduled to end with Druid 37.0.0.

Java 11 support has been deprecated since Druid 32.0, and official support will be removed as early as Druid 35.0.0

#### Hadoop-based ingestion

**Update**: Support for Hadoop-based ingestion is scheduled to end with Druid 37.0.0.

Hadoop-based ingestion has been deprecated since Druid 32.0 and will be removed as early as Druid 35.0.0.

We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md).
Expand Down
5 changes: 2 additions & 3 deletions docs/latest/tutorials/tutorial-batch-hadoop.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,11 @@ sidebar_label: Load from Apache Hadoop

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion)

:::


This tutorial shows you how to load data files into Apache Druid using a remote Hadoop cluster.

For this tutorial, we'll assume that you've already completed the previous
Expand Down
4 changes: 1 addition & 3 deletions docs/latest/tutorials/tutorial-kerberos-hadoop.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@ sidebar_label: Kerberized HDFS deep storage

:::caution[Deprecated]

Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md)

You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239)
Hadoop-based ingestion is deprecated. For more information, see the [upgrade notes](../release-info/upgrade-notes.md#hadoop-based-ingestion)

:::

Expand Down
5 changes: 2 additions & 3 deletions src/pages/community/join-slack.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,10 @@ import BrowserOnly from '@docusaurus/BrowserOnly';

<BrowserOnly>
{() => {
window.location.replace("https://join.slack.com/t/apachedruidworkspace/shared_invite/zt-265fczhyf-xFq8OyzzQ6TykfHDpfUzDA");
window.location.replace("https://join.slack.com/t/apachedruidworkspace/shared_invite/zt-3eulauy9i-liykRPrufmV~OOncRNQbzQ");
return null;
}}
</BrowserOnly>


<a href ="https://join.slack.com/t/apachedruidworkspace/shared_invite/zt-265fczhyf-xFq8OyzzQ6TykfHDpfUzDA">Click here if you're not redirected.</a>

<a href ="https://join.slack.com/t/apachedruidworkspace/shared_invite/zt-3eulauy9i-liykRPrufmV~OOncRNQbzQ">Click here if you're not redirected.</a>