[FEATURE]: add transforms to the logs table

[shahrokni]

Relates to perses/perses#3786

Description 🖊️

This PR will add the current transform feature to the logs table.

Checklist

• Pull request has a descriptive title and context useful to a reviewer.
• Pull request title follows the [<catalog_entry>] <commit message> naming convention using one of the
  following catalog_entry values: FEATURE, ENHANCEMENT, BUGFIX, BREAKINGCHANGE, DOC,IGNORE.
• All commits have DCO signoffs.

UI Changes

• Changes that impact the UI include screenshots and/or screencasts of the relevant changes.
• Code follows the UI guidelines.

[jgbernalp]

Suggested change

console.log(transforms);

[shahrokni]

done

[jgbernalp]

There seems to be an issue with the transforms editor. The columns value which is marked as required is not saved.

[shahrokni]

There seems to be an issue with the transforms editor. The columns value which is marked as required is not saved.

Shouldn't this be fixed with a dedicated issue for the transform editor?
I just embedded the current existing Transform Editor from the Shared repo into the Logs Table plugin
I have not even added the Extract transformer yet
I myself also found an issue

[jgbernalp]

sure, but why don't do the fix first so we can actually test this is working?

[shahrokni]

sure, but why don't do the fix first so we can actually test this is working?

Hmm, yes why not. Let's do the fix first.
I am just wondering, if the transform feature has been used seriously so far. It seems there are some serious issues with the feature. Have you ever tried this feature yourself from your side?

[jgbernalp]

I just shared with you my testing of the feature, that's why I found the issue... There's probably a lack of testing for this, as this used to be working before.

[shahrokni]

There seems to be an issue with the transforms editor. The columns value which is marked as required is not saved.

@jgbernalp I would to mention that from what I understood the Transform has not been broken with a recent change.

Why doesn't it work?

We can not see the effect of the transform for a Log Table because it has never been used and designed for log table!

The data structure doesn't fit

The data transform utility does not recognize log entry structure. And what is really a column in a log entry table?
Please take into account that maybe some of these transform does not have any meaning for logs. If this is the case, the list of proper transforms should be configurable. It seems the transform was initially designed for a simple table

/* This is the utility */
export function transformData(
  data: Array<Record<string, unknown>>,
  transforms: Transform[]
): Array<Record<string, unknown>> => {/* ... */}

/* This is the log structure */ 
/* There is column here */
export interface LogEntry {
    timestamp: number;
    line: string;
    labels: Labels;
}

/* If you try the wrong usage, it throws an exception. (Type mismatch) */
  const rawLogs = queryResults
    .flatMap((result) => result?.data.logs?.entries ?? [])
    .sort((a, b) => b.timestamp - a.timestamp);

  const logs = useMemo(() => transformData(rawLogs, spec.transforms ?? []), [rawLogs, spec.transforms]);

Likely steps

1. Make the Transform Kind dynamic (You should be able to include exclude items to the transform Kind)
2. Implement logs table specific transforms
3. Get back to this PR and force it use Logs specific transforms

[Gladorme]

Transformation principe has been thought to be re-used in any query type, but yeah currently there are only TimeSeriesQuery transformations, that will probably not work/make sense with logs or other query types.

I wonder if we should not create a new type of plugin (Transformation), so datasources can add their own transformations and people with specific needs can add their own too 🤔

[jgbernalp]

IMO transforms should apply to a table data, regardless of the QueryType behind. When the data is on the table it has already been transformed from query type to Record of things. Maybe we should re check if this should be the case, but I think it does not make sense to force currently the logs to match the timeseries transforms...

[shahrokni]

IMO transforms should apply to a table data, regardless of the QueryType behind. When the data is on the table it has already been transformed from query type to Record of things. Maybe we should re check if this should be the case, but I think it does not make sense to force currently the logs to match the timeseries transforms...

I think the name Logs Table is misleading. What we see is not the table/table view.
We need to have an option to switch between this view and table view. (A simple table comprises rows and columns) You can find the same feature in Grafana (Logs Table View)

https://grafana.com/whats-new/2023-12-13-logs-table-ui/

Second 27

What is a Log Table View

A table view turn the logs into a structure suitable for rendering in a table. The transforms can be applied to the table view.

Steps

1. Table view should process logs and extract set of possible columns
2. Then it generates the according rows
3. Then it should create the table
4. Transforms can be applied to tables, as it has already been mentioned

[Gladorme]

IMO transforms should apply to a table data, regardless of the QueryType behind. When the data is on the table it has already been transformed from query type to Record of things. Maybe we should re check if this should be the case, but I think it does not make sense to force currently the logs to match the timeseries transforms...

I agree with this.

... We need to have an option to switch between this view and table view ...

I did not really use or check Log Table panel, so I can't tell what is the best solution

[jgbernalp]

@shahrokni Not sure If understand correctly. What is the scope of the work you are planning to do? Have a button to switch from logs table to a generic table?