Skip to content

Commit c29bc74

Browse files
jeffreyavenjeffreyaven
authored
Merge pull request #208 from stackql/claude/update-stackql-docs-01UmTDEa3wZKwmXQBYYcqx3D
Claude/update stackql docs 01 um td ea3w z kwm xqby ycqx3 d
2 parents 3d2e94b + c7e84b5 commit c29bc74

File tree

2 files changed

+184
-57
lines changed

2 files changed

+184
-57
lines changed

‎docs/developers/architecture.md‎

Lines changed: 116 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -11,70 +11,135 @@ description: Query and Deploy Cloud Infrastructure and Resources using SQL
1111
image: "/img/stackql-featured-image.png"
1212
---
1313

14-
See also:
14+
See also:
1515
[[ Using StackQL ]](/docs/getting-started/using-stackql) [[ Using Variables ]](/docs/getting-started/variables) [[ Templating ]](/docs/getting-started/templating)
1616

1717
## Overview
1818

19-
The following diagram describes the internals of the StackQL at a high level.
19+
StackQL is a unified SQL interface for querying and managing cloud infrastructure across multiple providers. The following container diagram illustrates the major components of the StackQL system and their interactions.
2020

21-
[![StackQL Component Diagram](/img/stackql-component-diagram.svg)](/img/stackql-component-diagram.png)
21+
```mermaid
22+
flowchart TB
23+
subgraph User["User (DevOps, Analyst, or Automation)"]
24+
direction LR
25+
end
26+
27+
subgraph StackQL["StackQL"]
28+
subgraph EntryPoints["Entry Points"]
29+
CLI["CLI / Shell<br/><i>Go</i>"]
30+
PGWire["PostgreSQL Wire Protocol<br/><i>Go, psql-wire</i>"]
31+
MCP["MCP Server<br/><i>Go</i>"]
32+
end
33+
34+
subgraph Processing["Query Processing"]
35+
Parser["SQL Parser<br/><i>Go, Yacc</i>"]
36+
QueryEngine["Query Engine<br/><i>Go</i>"]
37+
SDK["Provider SDK<br/><i>Go, any-sdk</i>"]
38+
end
39+
40+
subgraph Backend["SQL Backend"]
41+
SQLBackend["SQL Backend<br/><i>Go</i>"]
42+
SQLite[("SQLite<br/><i>Embedded</i>")]
43+
Postgres[("PostgreSQL<br/><i>External, Optional</i>")]
44+
end
45+
end
46+
47+
subgraph External["External Systems"]
48+
Providers["Cloud Providers<br/><i>AWS, GCP, Azure, K8s, GitHub...</i>"]
49+
Registry["StackQL Registry<br/><i>Provider definitions</i>"]
50+
end
51+
52+
User --> CLI
53+
User --> PGWire
54+
User --> MCP
55+
56+
CLI --> Parser
57+
PGWire --> Parser
58+
MCP --> Parser
59+
60+
Parser --> QueryEngine
61+
QueryEngine --> SDK
62+
QueryEngine --> SQLBackend
63+
64+
SDK --> Providers
65+
SDK --> Registry
66+
SDK --> SQLBackend
67+
68+
SQLBackend --> SQLite
69+
SQLBackend --> Postgres
70+
```
2271

23-
### Driver
24-
The StackQL Driver is invoked by either a command being run in the [StackQL Interactive Command Shell](/docs/command-line-usage/shell) or a command or commands being submitted using the [`stackql exec`](/docs/command-line-usage/exec) command. The Driver is responsible for the orchestration of StackQL commands.
72+
## Core Components
2573

26-
### QueryPreProcessor
27-
The QueryPreProcessor is responsible for the preprossing of variables (if provided), using either Json or [Jsonnet](https://jsonnet.org/). For more information see [[ Templating ]](/docs/getting-started/templating).
74+
### Entry Points
2875

29-
### QueryParser
30-
The QueryParser parses the IQL statement and ensures the validity of the syntax provided as well as attribute references and mandatory attributes.
76+
StackQL supports multiple entry points for executing SQL queries:
3177

32-
### ProviderInterface
33-
The ProviderInterface consumes signed extended API specs from the StackQL Provider Registry.
78+
- **CLI / Shell**: An interactive shell and command-line interface for executing SQL queries directly from the terminal. Users can run ad-hoc queries, scripts, or use the interactive shell for exploration.
3479

35-
> StackQL provider interface documents are used to enumerate and define all available resources and their associated methods and attributes for a given provider. This information - collected at execution time - is cached internally for the current session.
80+
- **PostgreSQL Wire Protocol**: Enables connections from standard PostgreSQL clients such as `psql`, DBeaver, or any other PostgreSQL-compatible tool. This allows seamless integration with existing database workflows and BI tools.
3681

37-
### QueryPlanner
38-
The QueryPlanner determines which components of the query will be executed remotely through the relevant Cloud Provider API, and which components will be executed locally (such as aggregate operations).
82+
- **MCP Server**: A Model Context Protocol server that enables AI assistant integration, allowing large language models to query and manage infrastructure through natural language.
3983

40-
### RemoteExecutor
41-
The RemoteExecutor is responsible for interacting with the relevant Cloud Provider's API, this includes the handling of asynchronous operations and paginated responses.
84+
### Query Processing Pipeline
4285

43-
### LocalExecutor
44-
The LocalExector is the local embedded SQL engine responsible for operations on provider resource data, such as scalar functions and aggegation operations.
86+
1. **SQL Parser**: Parses SQL input into an Abstract Syntax Tree (AST) using a custom grammar based on Yacc. The parser (implemented as `stackql-parser`) validates SQL syntax and prepares it for query planning.
4587

46-
## Detailed Design
88+
2. **Query Engine**: The central orchestration layer that plans and executes queries. It determines which operations can be pushed down to the API layer versus which require local SQL processing. The query engine handles:
89+
- Query planning and optimization
90+
- Coordination between API calls and local SQL execution
91+
- Result aggregation and transformation
4792

48-
The following diagram is a detailed description of the `stackql` application
93+
### Provider SDK (any-sdk)
4994

50-
```mermaid
51-
graph TB
52-
OpenapiStackQL("Openapi-StackQL")
53-
StackQLParser("StackQL Parser")
54-
PsqlWire("psql-wire")
55-
56-
Shell[Shell] --> CommandRunner[Command Runner]
57-
Exec[Exec] --> CommandRunner
58-
CommandRunner --> Driver[Driver]
59-
Server[Server] --> Driver
60-
Server --> WireServer[Wire Server]
61-
WireServer --> PsqlWire
62-
Driver --> QuerySubmitter[Query Submitter]
63-
QuerySubmitter --> PlanBuilder[Plan Builder]
64-
65-
PlanBuilder --> InitialPassesScreenerAnalyzer[Mature the AST]
66-
InitialPassesScreenerAnalyzer --> NestedIndirection[Nested Indirection]
67-
NestedIndirection --> IndirectExpansion[Indirect Expansion]
68-
PlanBuilder --> Parser[Parser]
69-
IndirectExpansion --> Parser
70-
Parser --> StackQLParser
71-
72-
PlanBuilder --> RoutePass[Route Pass]
73-
PlanBuilder --> PrimitiveBuilder[Primitive Builder]
74-
PrimitiveBuilder --> PrimitiveGraph[Primitive Graph]
75-
PlanBuilder --> PrimitiveGraph
76-
PrimitiveBuilder --> OpenapiStackQL
77-
RoutePass --> ParameterRouter[Parameter Router]
78-
ParameterRouter --> OpenapiStackQL
79-
RoutePass --> NestingComposition[Nesting / Composition]
80-
```
95+
The Provider SDK is a critical component that manages all interactions with cloud providers. Key responsibilities include:
96+
97+
- **Provider Discovery**: Fetches provider specifications from the StackQL Registry, which contains OpenAPI-based definitions for each supported cloud provider.
98+
99+
- **Authentication**: Handles authentication for each provider, supporting various auth methods including API keys, OAuth tokens, service account credentials, and more. Each provider definition specifies its authentication requirements.
100+
101+
- **API Request Building**: Transforms SQL queries into appropriate REST API calls for each provider, handling endpoint construction, parameter mapping, and request formatting.
102+
103+
- **Pagination**: Automatically handles paginated API responses, abstracting away the complexity of different pagination schemes used by various providers (cursor-based, offset-based, token-based, etc.).
104+
105+
- **Predicate Pushdown**: Where supported, pushes query predicates (WHERE clause conditions) to the provider API to minimize data transfer and improve query performance.
106+
107+
- **Response Transformation**: Converts API responses into relational data that can be processed by the SQL backend.
108+
109+
### SQL Backend
110+
111+
The SQL Backend executes relational operations on materialized API data:
112+
113+
- **SQLite**: An embedded database used for temporary table storage and relational processing. Ideal for ad-hoc queries and smaller datasets.
114+
115+
- **PostgreSQL**: An optional external database for persistent storage, larger datasets, and advanced analytics workloads.
116+
117+
The SQL backend handles operations such as:
118+
- JOIN operations across multiple providers or resources
119+
- Aggregate functions (COUNT, SUM, AVG, etc.)
120+
- Sorting and filtering on materialized data
121+
- Complex SQL expressions and subqueries
122+
123+
## External Systems
124+
125+
### Cloud Providers
126+
127+
StackQL connects to cloud providers (AWS, GCP, Azure, Kubernetes, GitHub, and many more) via their REST APIs. All communication is over HTTPS, and the Provider SDK handles the specifics of each provider's API conventions.
128+
129+
### StackQL Registry
130+
131+
The StackQL Registry hosts provider definitions - OpenAPI-based specifications that describe available resources, methods, and their mappings. Providers are versioned and can be pulled on-demand using the `REGISTRY PULL` command.
132+
133+
## Query Execution Flow
134+
135+
1. User submits a SQL query through one of the entry points (CLI, PostgreSQL protocol, or MCP)
136+
2. The SQL Parser validates and converts the query into an AST
137+
3. The Query Engine analyzes the AST and creates an execution plan
138+
4. For provider operations, the Provider SDK:
139+
- Fetches the relevant provider specification from the Registry (if not cached)
140+
- Authenticates with the cloud provider
141+
- Builds and executes API requests, handling pagination automatically
142+
- Pushes predicates to the API where possible
143+
- Materializes results into the SQL backend
144+
5. The SQL Backend executes any remaining relational operations
145+
6. Results are returned to the user in the requested format

‎docs/developers/developing-stackql-providers.md‎

Lines changed: 68 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,12 @@ description: Query and Deploy Cloud Infrastructure and Resources using SQL
1111
image: "/img/stackql-featured-image.png"
1212
---
1313

14-
See also:
14+
See also:
1515
[[ StackQL Provider Registry ]](/providers) [[ Using a Provider ]](/docs/getting-started/using-a-provider)
1616

1717
## Overview
1818

19-
The StackQL Provider Registry is a crucial component of the StackQL ecosystem. It maintains and manages the 'provider' interface documents which inform the StackQL application about how to interact with various providers like AWS, Azure, Google, etc.
19+
The StackQL Provider Registry is a crucial component of the StackQL ecosystem. It maintains and manages the 'provider' interface documents which inform the StackQL application about how to interact with various providers like AWS, Azure, Google, etc.
2020

2121
The registry workflow involves three key components:
2222

@@ -26,13 +26,75 @@ The registry workflow involves three key components:
2626

2727
3. **Deno Deploy**: The packaged artifacts are then registered and published to the StackQL Provider Registry Artifact Repository via Deno Deploy. These artifacts are available to the StackQL application via the registry API, retrievable using `REGISTRY LIST` or `REGISTRY PULL` commands.
2828

29-
![Your context diagram here]
29+
The following diagram illustrates the provider development and publishing workflow:
30+
31+
```mermaid
32+
sequenceDiagram
33+
participant Dev as Developer
34+
participant GH as GitHub Repository
35+
participant GA as GitHub Actions
36+
participant Reg as StackQL Registry
37+
participant App as StackQL Application
38+
39+
Dev->>GH: Push provider OpenAPI specs
40+
GH->>GA: Trigger on push/PR
41+
GA->>GA: Validate OpenAPI specs
42+
GA->>GA: Run provider tests
43+
GA->>GA: Package & sign artifacts
44+
GA->>Reg: Publish to registry
45+
46+
App->>Reg: REGISTRY LIST / REGISTRY PULL
47+
Reg-->>App: Return provider specs
48+
App->>App: Cache provider locally
49+
```
50+
51+
## StackQL Provider Basics
52+
53+
StackQL providers are OpenAPI-based specifications that define how StackQL interacts with cloud service APIs. Each provider encapsulates the complexity of working with a specific cloud platform, exposing resources as queryable SQL tables.
54+
55+
### What Providers Handle
56+
57+
Providers are responsible for several critical functions that abstract away API complexity:
58+
59+
- **Authentication**: Each provider defines its authentication requirements and methods. This includes support for API keys, OAuth 2.0 tokens, service account credentials, bearer tokens, and provider-specific authentication schemes. Users configure authentication once, and the provider handles token management and request signing.
60+
61+
- **Pagination**: Cloud APIs often return paginated responses for large datasets. Providers automatically handle pagination, abstracting away the differences between cursor-based, offset-based, token-based, and other pagination schemes. Users receive complete result sets without needing to manage pagination logic.
62+
63+
- **Predicate Pushdown**: Where supported by the underlying API, providers push query predicates (WHERE clause conditions) directly to the API. This minimizes data transfer, reduces API calls, and improves query performance by filtering data at the source rather than client-side.
64+
65+
- **Response Transformation**: Providers map API responses to relational table structures, handling nested objects, arrays, and complex data types to present a consistent SQL-queryable interface.
66+
67+
### Provider Structure
68+
69+
A StackQL provider typically includes:
70+
71+
- **Services**: Logical groupings of related resources (e.g., `compute`, `storage`, `networking`)
72+
- **Resources**: Individual API resources exposed as SQL tables (e.g., `instances`, `buckets`, `firewalls`)
73+
- **Methods**: Operations available on resources mapped to SQL verbs (`SELECT`, `INSERT`, `UPDATE`, `DELETE`)
74+
- **Properties**: Resource attributes exposed as table columns
75+
76+
For more detailed implementation information and configuration options, see the [Provider Specification Documentation](https://github.com/stackql/any-sdk/blob/main/docs/provider_spec.md).
3077

3178
## Developing a Provider Locally
3279

33-
To develop a StackQL provider, you'll need a provider's OpenAPI or Swagger specification. These specifications could either be supplied by the provider or generated through scripts such as [__google-discovery-to-openapi__](https://github.com/stackql/google-discovery-to-openapi) or [__stackql-azure-openapi__](https://github.com/stackql/stackql-azure-openapi).
80+
To develop a StackQL provider, you'll need a provider's OpenAPI or Swagger specification. These specifications could either be supplied by the provider or generated through custom scripts.
81+
82+
Once you have the OpenAPI specification, you can use the [__`stackql-provider-utils`__](https://github.com/stackql/stackql-provider-utils) utility project and the [__`stackql-provider-TEMPLATE`__](https://github.com/stackql/stackql-provider-TEMPLATE) template project to generate a StackQL provider document.
83+
84+
## Using Claude Code for Provider Development
85+
86+
[Claude Code](https://docs.anthropic.com/en/docs/claude-code) can assist with StackQL provider development through a specialized skill that provides comprehensive guidance on creating and configuring providers.
87+
88+
The [__StackQL Provider Development Skill__](https://github.com/stackql/any-sdk/blob/main/.claude/skills/stackql-provider-development.md) teaches Claude about:
89+
90+
- **Provider Architecture**: The hierarchical structure organizing providers into services, resources, methods, and operations
91+
- **StackQL Extensions**: All custom `x-stackQL-*` OpenAPI extensions for resource definitions, SQL verb mappings, and query semantics
92+
- **Authentication Configuration**: Setting up API keys, OAuth 2.0, AWS signing, service accounts, and other authentication methods
93+
- **Pagination Strategies**: Configuring token-based, offset-based, and link header pagination
94+
- **Query Optimization**: Implementing predicate pushdown for filtering, sorting, and projection at the API level
95+
- **Response Transformation**: Extracting and transforming nested API responses into flat table structures
3496

35-
Once you have the OpenAPI specification, you can use the [__openapisaurus__](https://github.com/stackql/openapisaurus) utility project to generate a StackQL provider document.
97+
To use this skill, clone the [any-sdk](https://github.com/stackql/any-sdk) repository and work with Claude Code in that directory. Claude will automatically have access to the provider development skill and can help you create new providers, troubleshoot configuration issues, and implement advanced features like custom authentication flows or complex pagination handling.
3698

3799
## Testing Your Provider
38100

@@ -43,4 +105,4 @@ To ensure that your provider works as expected, you can test it using the `dev`
43105
```bash
44106
export DEV_REG="{ \"url\": \"https://registry-dev.stackql.app/providers\" }"
45107
./stackql --registry="${DEV_REG}" shell
46-
```
108+
```

0 commit comments

Comments
 (0)