RDS/Aurora logical refresh component #262
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a new component that automates DBLab full refresh using temporary RDS/Aurora database clones created from snapshots. This enables a hassle-free data sync workflow that doesn't impact production. Features: - Create temporary RDS/Aurora clones from latest automated snapshots - Wait for clone availability with proper timeout handling - Trigger DBLab full refresh via API - Poll refresh status until completion - Clean up temporary clones automatically Deployment options: - AWS Lambda with SAM template and EventBridge scheduling - Standalone CLI binary for cron/manual execution - Docker container for containerized environments Includes comprehensive documentation with IAM policy examples and example configuration files.
Add a fully standalone version of the RDS/Aurora refresh tool that can be built and deployed immediately without requiring a DBLab Engine release. This version has no dependencies on DBLab internal packages. The standalone component includes: - Self-contained Go module with minimal dependencies - Makefile for easy building (CLI, Lambda, Docker) - SAM template for AWS Lambda deployment - Dockerfile for container builds - Comprehensive documentation Can be used immediately by: 1. go build -o rds-refresh . 2. ./rds-refresh -config config.yaml
- Add UpdateSourceConfig method to update DBLab source connection before triggering refresh, ensuring DBLab knows the clone endpoint - Add source database credentials (dbName, username, password) to config for DBLab connection configuration - Remove Lambda-specific code, focusing on container/CLI deployment which better suits the long-running nature of refresh operations - Remove duplicate engine-integrated code (engine/internal/rdsrefresh and engine/cmd/rds-refresh) since standalone component is preferred - Update documentation with container deployment options (Docker, ECS Task, Kubernetes CronJob, CLI with cron) - Update workflow to include config update step before refresh trigger
DBLab expects a flat JSON structure matching ConfigProjection fields,
not a nested structure. The API automatically reloads the config after
update (calls reloadFn internally), so no SIGHUP is needed.
Changed from nested:
{"retrieval": {"dbSource": {"host": "..."}}}
To flat projection format:
{"host": "...", "port": 5432, "dbname": "...", ...}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.