Flag to disable storage when stateful processing is not needed

[sgaragan]

For a number of transformations, stateful transformation is not needed but for the current setup of Candyfloss, it still requires the creation of the internal changelog topics needed by KStream for stateful processing. Having a flag such as the following in the application config would disable this and remove the requirement of needing this topic

kstream.changelog.disabled = true

Thanks,
Sean

[cuihengch]

Thanks for the input, Sean. Just a few comments for my understanding:

The example config seems to be a "general config" for overall Candyfloss behavior, in practice, if we have multiple pipelines and at least one pipeline needs a Normalization, then, changeLog topic will be needed; meanwhile if this config is in "disabled", would these be "logically conflicted"? Or I may misunderstand anything?

Thanks
Heng

[sgaragan]

Thanks for the input, Sean. Just a few comments for my understanding:

The example config seems to be a "general config" for overall Candyfloss behavior, in practice, if we have multiple pipelines and at least one pipeline needs a Normalization, then, changeLog topic will be needed; meanwhile if this config is in "disabled", would these be "logically conflicted"? Or I may misunderstand anything?

Thanks Heng

Yes, the config is a general one as the setting happens on the StreamBuilder when creating the topology. If you have at least one pipeline that needs Normalization (hence a stateful transformation) then setting this would break it. However, if you have a set of pipelines where none of them need normalization, then the changelog topic is not needed. In our case, we use Candyfloss as a general JSON to JSON transformation engine without normalization for some of the pipelines so the changelog topic is not needed. However, because it is required, we have to set one up which uses up partition quota in Kafka (a very limited resource in this case). Removing this when not needed means one less item to configure and fewer resources used

Thanks,
Sean