Ability to have multiple output topics per transformation

[sgaragan]

As we work through our use cases we found that it would be very helpful to have a way to define multiple outputs per transformation. The reason for this is so that we can push messages with the same record structure but different values to separate topics.

For example, if we have two messages like

{
   "network_domain": "network_A",
   "message_type": "openconfig-interfaces:interfaces",
   "payload": {
     ....
   }
}

and

{
   "network_domain": "network_B",
   "message_type": "openconfig-interfaces:interfaces",
   "payload": {
     ....
   }
}

where we want the transformed messages to route to a topic based on the value of "network_domain" but otherwise the transformation is the same. Currently, this would require copying the transformation, having a match that looks at the "network_domain" value, then defining another pipeline with the new output topic. This will end up with a lot of cut/paste code and introduce a new way for errors to creep in.

The thought was to either have a field in the pipeline config that acts as a value match for the output (so it can use the same transformation file in two pipelines) or, add a new section in the transformation spec like

{
   "match": {
      ... # Top level match shared by all messages
   }
   "output: {
      [ 
         {
            "topic": "network_A_topic",
            "match" : {
              ... # Match for Network_A
            },
         },
         {
            "topic": "network_B_topic",
            "match" : {
              ... # Match for Network_B
            },
         }
   },

The syntax is just an example of course, but the idea is there

Thanks,
Sean