Home  /  Solutions  /  SQL Query Orchestrator

SQL workflows.
Fully automated.
Zero infrastructure.

Schedule, chain, and monitor complex SQL operations across Amazon Redshift, Aurora, RDS, and Athena — through a visual web portal, without writing a single line of infrastructure code.

6
AWS database types supported out of the box
<5 min
Maximum scheduling latency on any activity
1,000
Concurrent Glue jobs supported simultaneously
2–3 days
Average time from signup to first production activity
The problem

Manual SQL ops are a liability at scale

Data engineering and DBA teams managing complex warehouses run hundreds of SQL operations daily. Without a dedicated orchestration layer, this creates brittle, invisible failure points.

  • Silent failures — nightly jobs fail undetected until users report missing data the next morning
  • No conditional execution — queries run blindly without checking prerequisites or previous results
  • Fragmented tooling — EC2 cron jobs, ad-hoc Lambdas, and SQL agents with no centralised view
  • No performance tracking — no history, no duration alerts, no way to catch gradual degradation
EC2 cron job — nightly refresh
Timed out at 02:17 · No alert sent · Reports empty since 03:00
Ad-hoc Lambda — monthly MV create
Deployed inline · Not versioned · Owner left the team
Shell script — incremental ETL
Hardcoded watermark dates · No retry logic · Runs duplicated data on failure
SQL Query Orchestrator
Scheduled · Conditional · Notified · Fully auditable
Platform capabilities

Everything a DBA team needs in one place

Built on a serverless AWS stack — no servers to manage, no agents to maintain, no vendor lock-in beyond what you're already running.

Cron Scheduling with Timezone Support
Standard 5-field cron expressions with full IANA timezone support. Daylight saving time transitions are handled automatically — your activities run at the right local time, always.
EventBridge Scheduler
Conditional Query Execution
Each query can have a condition evaluated against previous query results. Create objects only if they don't exist, refresh only if data is available, skip steps based on runtime state.
simpleeval engine
Smart Calculated Parameters
Dynamic values resolved at runtime: %YEARMONTH%, %PREVMONTH%, %DAYOFWEEK%, %PREVIOUS_START_DT%, and more. Pass results between queries using %PREV_VALUE%.
12 built-in keywords
Activity Chaining
Define prerequisite and postrequisite activities to build complex dependency graphs. The system automatically detects and halts on circular references before they cause problems.
Recursion detection
Real-Time Email Notifications
Success emails include tabular query results. Failure emails include error details and which query failed. Duration alerts fire when execution exceeds your expected threshold.
Amazon SES
Cross-Account Database Access
Access databases in client AWS accounts via IAM cross-account role assumption. Only temporary STS credentials are used — no permanent passwords, no stored access keys, no security compromise.
STS AssumeRole
Visual 5-Step Activity Wizard
Define activities through an intuitive wizard: basic settings → global parameters → SQL queries with conditions → review → save. A Monaco SQL editor provides syntax highlighting for complex queries.
No CLI required
Execution History & Retention
Every execution is stored in DynamoDB with query-level results, duration, and error details. Configurable retention per activity — older records are automatically purged via TTL.
DynamoDB TTL
Multi-Tenant Isolation
Each client's connections, activities, and results are fully isolated via S3 prefix and DynamoDB partition key. Cognito Identity Pool provides secure portal authentication per tenant.
S3 prefix isolation
Database support

One platform, every AWS database

Native executors for each database type — each tuned to the right access pattern, authentication method, and AWS API.

Data API
Amazon Redshift
Redshift Data API + STS AssumeRole · No VPC required
Data API
Aurora PostgreSQL
RDS Data API + STS AssumeRole · Secrets Manager auth
Data API
Aurora MySQL
RDS Data API + STS AssumeRole · Secrets Manager auth
Direct
RDS PostgreSQL
psycopg2 direct connection · Secrets Manager credentials
Direct
RDS MySQL
pymysql direct connection · Secrets Manager credentials
Athena API
Amazon Athena
Athena API + STS AssumeRole · S3 output location
Coming soon
RDS SQL Server
pymssql direct connection · Same pattern as PostgreSQL / MySQL
How it works

Event-driven execution, end to end

1
Define connections and activities in the portal
Connect your databases using the web portal — no CLI, no JSON files to edit by hand. Then create activities with the 5-step wizard, defining SQL queries, parameters, conditions, and notification addresses.
2
EventBridge polls for due activities every 5 minutes
The Launcher Lambda queries a DynamoDB GSI for all enabled activities where next_run < now. Each due activity triggers a Step Functions execution.
3
Glue resolves parameters and executes SQL
A Python Shell Glue job loads the activity config from S3, resolves all calculated and template parameters, evaluates conditions against previous query results, and executes each SQL query in order via the appropriate database executor.
4
Results stored, notifications sent
Execution results are written to DynamoDB with full per-query detail. Email notifications with tabular results go out via SES. If execution exceeds the expected duration threshold, an alert fires immediately.
5
Chained activities execute in sequence
Postrequisite activities start automatically after the main activity completes. Prerequisite failures prevent the main activity from starting. The recursion detector ensures chains can never loop.
Execution architecture
EventBridge Scheduler
Fires every 5 minutes
Launcher Lambda
DynamoDB GSI query → due activities
Step Functions
Orchestrates Glue job + error catch
Glue Python Shell Job
Parameter resolution → SQL execution
DynamoDB + SES
Results stored · Notifications sent
Use cases

What teams are automating with SQO

From monthly materialized view pipelines to incremental ETL loads — real patterns used in production by Cloudwalker clients.

Use case 01
Automatic Monthly Materialized View Pipeline
On the first of each month, a 7-query activity automatically creates a new MV for the current month (mv_sales_%YEARMONTH%), refreshes the previous month's MV, and reconstructs the UNION view that joins all monthly data — all with conditional execution to check object existence before acting.
%YEARMONTH% Conditional execution Redshift 7 sequential queries
Use case 02
Incremental Data Loading with Automatic Watermarking
Using %PREVIOUS_START_DT%, queries automatically process only new data since the last successful execution. No manual watermark table, no hardcoded dates — the system tracks the last run and injects the correct timestamp into every query.
%PREVIOUS_START_DT% Incremental ETL Redshift / Aurora
Use case 03
Daily Analytical Object Refresh with Duration Monitoring
DBA teams define daily refresh operations for materialized views and summary tables, with prerequisite object-existence checks before execution. expected_duration alerts fire immediately if any activity starts running longer than baseline.
Daily cron Duration alerts Prerequisite checks
Use case 04
Multi-Database Cross-Source Workflows
A single activity can use different connections for different queries — read aggregated data from an Athena data lake, transform it, and write results to Redshift. Each query targets the right database independently without extra plumbing.
Athena → Redshift Multi-connection Cross-source
Configuration reference

Calculated parameters

Dynamic values resolved at execution time — no hardcoded dates, no manual watermark tracking. Use them in SQL queries or global parameters.

Keyword Format Example (2026-04-15) Description
%YEARMONTH%YYYYMM202604Current year and month
%PREVYEARMONTH%YYYYMM202603Previous year and month
%NEXTYEARMONTH%YYYYMM202605Next year and month
%YEAR%YYYY2026Current year
%MONTH%MM04Current month
%PREVMONTH%MM03Previous month number
%DAY%DD15Current day of month
%DAYOFWEEK%1–73ISO day of week (Mon=1, Sun=7)
%PREVIOUS_START_DT%datetime+tz2026-04-14 00:15:00 +02:00Start timestamp of last successful run
%PREVIOUS_END_DT%datetime+tz2026-04-14 00:17:32 +02:00End timestamp of last successful run
%PREV_VALUE%string1Result of last returns: true query
%PREV_VALUE@N%stringResult of query with order_num = N
%PREV_VALUE$name%stringResult of query with the given name
Getting started

Live in 2–3 business days

Cloudwalker handles infrastructure and access provisioning. Your team handles AWS-side setup and activity creation. Both workstreams run in parallel.

Cloudwalker
1
Client registration & user account
Create tenant in the portal, set up user credentials and operating mode
2
Technical data delivery
Provide portal URL, Glue role ARN for trust policy, and IP ranges for Security Group
3
SES production configuration
Configure email service in production mode for your domain — no per-address verification needed
Your team
1
Create IAM cross-account role
Trust policy for Cloudwalker's Glue role + permissions scoped to your database type
2
Security Group & Secrets Manager
Only required for direct connections (PostgreSQL, MySQL, SQL Server) — not needed for Data API or Athena
3
Create connections & activities
Log in to the portal, add connections, build activities using the wizard, run a test
You're live
Test activity confirmed
Activity shows CREATED status, test query executes, email notification arrives
Production activities running
Real workflows scheduled, monitored, and notifying your team automatically
Zero infrastructure to maintain
Fully serverless — no EC2, no agents, no patching. AWS manages the runtime
2–3 days
From first contact to first production activity executing
100% serverless AWS

Built on managed services you already trust

No servers to patch, no daemons to monitor. SQL Query Orchestrator runs entirely on AWS managed services — deployed via CloudFormation in your account.

AWS Glue (Python Shell)
SQL query execution engine with full Python library support
AWS Step Functions
Orchestrates Glue job execution with error catch and retry
AWS Lambda (×4)
Scheduler, launcher, result analyzer, SQL export — Python 3.12 arm64
Amazon EventBridge
5-minute polling schedule triggers the launcher pipeline
Amazon DynamoDB
Activity schedule table + execution results with configurable TTL
Amazon S3
Configuration storage, connection files, and execution log rotation
Amazon SES
HTML email notifications with Cloudwalker branding and tabular results
Amazon Cognito
Portal authentication via User Pool + Identity Pool temporary credentials
AWS CloudFormation
Full infrastructure-as-code — repeatable deployment, no manual setup
AWS IAM + STS
Cross-account database access via temporary role assumption — zero permanent credentials
Next.js + TypeScript
React portal with Monaco SQL editor and shadcn/ui components
AWS Secrets Manager
Database credentials for direct connection types — no passwords in config files
FAQ

Common questions

Six types are live: Amazon Redshift (Data API), Aurora PostgreSQL (RDS Data API), Aurora MySQL (RDS Data API), RDS PostgreSQL (direct), RDS MySQL (direct), and Amazon Athena. RDS SQL Server (pymssql) is planned in the next phase.
Each query has an optional condition field — a Python expression evaluated using the simpleeval library. The most common pattern: query 1 checks whether an object exists (returns 0 or 1), query 2 has condition %PREV_VALUE% != 0. If the condition is false, the query is skipped without error.
You create an IAM role in your AWS account with a trust policy authorising Cloudwalker's Glue role to assume it. The Glue job uses STS to obtain temporary credentials for your role — no permanent passwords or access keys are ever stored or exchanged.
With abort_on_error: true, the activity stops at the first failed query, stores the error in DynamoDB, and sends a failure notification email. With abort_on_error: false, the error is logged and execution continues with the next query.
The EventBridge polling interval is 5 minutes, so the maximum scheduling latency is 5 minutes. Activities will execute within 5 minutes of their scheduled time. For most DBA and data engineering workloads this is more than sufficient — if you need sub-minute precision, this platform is not the right fit.
Yes. A query with returns: true stores the first column of the first result row. Subsequent queries reference it via %PREV_VALUE% (last result), %PREV_VALUE@3% (query by order_num), or %PREV_VALUE$query_name% (query by name).
Retention is configurable per activity via the retention_result field (in months). DynamoDB TTL automatically deletes older records. S3 log files can have additional lifecycle policies applied at the bucket level.
2–3 business days end to end. Cloudwalker's setup (registration, user account, SES, technical data handover) takes 1 day. Your team's AWS-side work (IAM role, optional Security Group and Secrets Manager) takes 1 day. Both can run in parallel, so most clients are running production activities within 3 days of starting.

Stop babysitting cron jobs.

SQL Query Orchestrator automates the repetitive SQL operations your team runs every day — with visibility, notifications, and zero infrastructure overhead.

Request a demo