[Proposal] Enhanced Lease Statuses for Hooks

[kirkbrauer]

Problem

Jumpstarter PR #606 adds new before/after lease hooks, but the client experience is poor due to lack of visibility into hook execution.

Current Issues

1. Poor User Experience: When a client connects to an exporter running beforeLease hooks, the exporter appears to "freeze" - clients can connect but all driver calls are blocked by the _before_lease_ready Event until the before lease hook is complete.

2. No Progress Indication: Users have no way to know that hooks are running, how long they might take, or what's happening during execution.

3. Confusing Availability: During after lease hook execution, the exporter appears unavailable for new leases, which can confuse users.

4. No Monitoring: Users cannot monitor after lease hook execution after releasing a lease from the CLI.

Current Implementation

• Exporters use an async Event to block all driver calls until the hook is complete
• Hook execution happens silently with only internal logging to the exporter console
• Clients have no indication that hooks are running or their progress
• The j CLI provides no feedback during hook execution

Proposed Solution

1. Protocol Changes

Add Exporter Status Enum to client.proto

message Exporter {
  // ... existing fields ...
  optional ExporterStatus status = 4 [(google.api.field_behavior) = OUTPUT_ONLY];
}

enum ExporterStatus {
  EXPORTER_STATUS_UNSPECIFIED = 0;
  EXPORTER_STATUS_AVAILABLE = 1;           // No lease, ready for connections
  EXPORTER_STATUS_BEFORE_LEASE_HOOK = 2;   // Running beforeLease hooks
  EXPORTER_STATUS_LEASED = 3;              // Leased and ready for commands
  EXPORTER_STATUS_AFTER_LEASE_HOOK = 4;    // Running afterLease hooks
}

Add Log Source to jumpstarter.proto

message LogStreamResponse {
  string uuid = 1;
  string severity = 2;
  string message = 3;
  optional LogSource source = 4;  // New optional field
}

enum LogSource {
  LOG_SOURCE_UNSPECIFIED = 0;
  LOG_SOURCE_EXPORTER = 1;       // System/exporter logs
  LOG_SOURCE_DRIVER = 2;          // Driver/device logs
  LOG_SOURCE_HOOK = 3;            // Hook execution logs
  LOG_SOURCE_MESSAGE = 4;        // Message (reserved for future MOTD use)
}

2. Implementation Changes

Exporter Updates

• Track current status using ExporterStatus enum
• Report status through existing GetReport and Status RPCs
• Stream hook logs through LogStream with appropriate LogSource
• Continue blocking driver calls for existing clients, new clients can see status

Hook Executor Updates

• Send hook output to LogStream with correct LogSource field
• Maintain existing execution logic
• Add progress indication in log messages

Client Updates

• Check exporter status when connecting
• Display appropriate waiting messages based on status
• Subscribe to LogStream and filter by source for hook progress
• Allow optional monitoring of after lease hooks (Ctrl+C ends lease, but waits)

3. User Experience Improvements

Before

$ j shell -l device=rpi4
Connecting to exporter...
[Connection appears to hang with no feedback]

After

$ j shell -l device=rpi4
Connecting to exporter...
Waiting for beforeLease hooks to complete...
[Hook INFO] Powering on device...
[Hook INFO] Initializing network interface...
[Hook INFO] Configuring storage...
beforeLease hooks completed successfully!
Connected to device.

[User works with device...]

Releasing lease...
afterLease hooks running (press Ctrl+C to disconnect):
[Hook Output] Saving logs to archive...
[Hook Output] Cleaning up workspace...
afterLease hooks completed.

Hooks Example

Hooks will be able to log messages by adding the JMP_INFO, JMP_WARN, or JMP_ERROR prefix to log messages.

Example hook shell script:

echo "JMP_INFO: This is informational"
echo "JMP_ERROR: This is an error message"
echo "JMP_WARN: This is a warning"

echo "If the type is not specified, it will be an info-level message"

Example client CLI output:

$ j shell -l device=rpi4
Connecting to exporter...
Waiting for beforeLease hooks to complete...
[Hook INFO] This is informational
[Hook ERROR] This is an error message
[Hook WARN] This is a warning
[Hook INFO] If the type is not specified, it will be an info-level message'
beforeLease hooks completed successfully!
Connected to device.

4. Backward Compatibility

• All new protocol fields are optional
• Existing clients continue to work without changes (calls are blocked)
• Graceful degradation for clients that don't understand new status
• No breaking changes to existing APIs

Implementation Plan

Phase 1: Protocol Updates

1. Add enums to client.proto and jumpstarter.proto in the jumpstarter-protocol.
2. Regenerate protobuf code and migrate existing client/exporter
3. Update exporter to track and report status

Phase 2: Hook Integration (#606)

1. Modify hook executor to use LogStream
2. Update exporter status transitions
3. Test status reporting

Phase 3: Client Updates

1. Update j CLI to check and display status
2. Add LogStream filtering by source
3. Implement progress indication

Phase 4: Documentation

1. Update user documentation
2. Add examples of hook status handling
3. Update API documentation

[kirkbrauer]

@mangelajo @NickCao

[mangelajo]

It looks good, I have some comments:

I would change this:

enum ExporterStatus {
  EXPORTER_STATUS_UNSPECIFIED = 0;
  EXPORTER_STATUS_AVAILABLE = 1;           // No lease, ready for connections
  EXPORTER_STATUS_BEFORE_LEASE_HOOK = 2;   // Running beforeLease hooks
  EXPORTER_STATUS_LEASED = 3;               // Leased and ready for commands
  EXPORTER_STATUS_AFTER_LEASE_HOOK = 4;    // Running afterLease hooks
}

to this

I would change this:

enum ExporterStatus {
  EXPORTER_STATUS_UNSPECIFIED = 0;
  EXPORTER_STATUS_AVAILABLE = 1;           // No lease, ready for connections
  EXPORTER_STATUS_BEFORE_LEASE_HOOK = 2;   // Running beforeLease hooks
  EXPORTER_STATUS_LEASE_READY= 3;          // Leased and ready for commands
  EXPORTER_STATUS_AFTER_LEASE_HOOK = 4;    // Running afterLease hooks
}

it's a nit honestly, doesn't change much, but makes it easier for me to think about it, because 2,3,4 are all leased, just in different phases.

For this, what about?

echo "This is informational, it will be an info-level message"
echo "This is an error message" >&2
echo "JMP_WARN: This is a warning". (or just WARNING: ) ? 

This is more natural, but perhaps your design is more explicit, I don't know hhmm. One benefit of separating in different levels is that this can be pushed to both client and server, and server can generate metrics and alerts based on ERROR/WARNING behavior.

On the after behavior, i would make the afterLease waiting behavior conditional on a flag, as it can happen in the background.

$ jmp shell -l device=rpi4 --debug-release
Connecting to exporter...
Waiting for before-lease hooks to complete...
[Hook INFO] Powering on device...
[Hook INFO] Initializing network interface...
[Hook INFO] Configuring storage...
beforeLease hooks completed successfully!
Connected to device {device-name}

$ [User works with device...]

$ exit

Releasing lease...
after-lease hooks running (press Ctrl+C to disconnect):
[Hook Output] Saving logs to archive...
[Hook Output] Cleaning up workspace...
afterLease hooks completed.

[kirkbrauer]

@mangelajo

I agree, we should probably just use stdout and stderr to sort out the INFO and ERROR log messages. The only think I struggle with is the other log levels, hence why I thought maybe making them explicit would make sense. Perhaps we can support both, that way normal scripts that use the standard output will work, but custom scripts can also log DEBUG and WARN log levels. The JMP_ prefix is supposed to be unique, but maybe if we just regex for WARN or DEBUG in the string then we can ingest other types of logs...

I think the ExporterStatus tweaks make sense, I think we should just think about the UI implications of different names as this will probably be surfaced in some sort of UI layer at some point.

[mangelajo]

Perhaps status should also cover other states we know about, like EXPORTER_STATUS_OFFLINE

[kirkbrauer]

We already have the online boolean, but maybe we can just unify it into the status field for simplicity?

jumpstarter-protocol/proto/jumpstarter/client/v1/client.proto

Line 70 in 8d91c4a

bool online = 3 [(google.api.field_behavior) = OUTPUT_ONLY];

[mangelajo]

Yep, I would deprecate that flag (we can keep it for a while), and also make it reflected on the status field.

[mangelajo]

From the meeting, we want to convert the status field on the controller into status conditions for the exporters:

example of current status conditions:

  status:
    conditions:
    - lastTransitionTime: "2025-09-25T13:59:39Z"
      message: Exporter is connected to the Status endpoint
      observedGeneration: 1
      reason: Connect
      status: "True"
      type: Online
    - lastTransitionTime: "2025-07-03T15:27:50Z"
      message: Exporter is connected to the Status endpoint
      observedGeneration: 1
      reason: Connect
      status: "True"
      type: Registered

https://github.com/jumpstarter-dev/jumpstarter-controller/blob/main/api/v1alpha1/exporter_types.go#L46