Conversation
This commit introduces the new component, providing a high-level construct for provisioning and managing Amazon Aurora DSQL clusters. The component supports both single-region and multi-region active-active deployments. For multi-region setups, it automatically handles the creation and peering of clusters across regions. The integration has been configured to provide functions with all necessary connection details for both primary and peer clusters, including ARNs, endpoints, and regions, accessible via the type-safe object. Two new examples have been added to demonstrate usage: - : A basic single-region cluster setup. - : A more advanced example showcasing how to connect to both the primary and peer clusters from a single Lambda function. Additionally, this commit updates the pulumi/aws dependency to v6.83.0
This commit refactors the DSQL component to resolve a critical deletion deadlock in multi-region deployments. The architecture is separated into two components: `Dsql` for single-region cluster management and `DsqlPeering` for coordinating multi-region connections. This new design leverages a Pulumi dynamic provider to orchestrate the simultaneous deletion of peered clusters, which is required by AWS and was the root cause of the deadlock. Key changes: - `Dsql` component is now simplified for single-region clusters. - New `DsqlPeering` component manages the lifecycle of multi-region peering. - A dynamic provider in TypeScript handles the complex creation and deletion logic, ensuring clusters are deleted in a coordinated manner. - The `aws-dsql-multiregion` example is updated to reflect the new, more robust usage pattern.
vimtor
left a comment
There was a problem hiding this comment.
thanks for your contribution @DamienPace15
i've left a bunch of comments and suggestions
thanks again for taking a look at this
| private static publicEndpointFromArn(clusterArn: Output<string>) { | ||
| return clusterArn.apply((arn) => { | ||
| const parts = arn.split(":"); | ||
| const region = parts[3]; | ||
| const clusterId = parts[5].split("/")[1]; | ||
| return `${clusterId}.dsql.${region}.on.aws`; | ||
| }); | ||
| } | ||
|
|
||
| private static regionFromArn(clusterArn: Output<string>) { | ||
| return clusterArn.apply((arn) => { | ||
| const parts = arn.split(":"); | ||
| const region = parts[3]; | ||
| return region; | ||
| }); | ||
| } |
There was a problem hiding this comment.
there are several ARN utilities in the codebase. we should check if any of the existing ones cover this use case
if this is sql specific, the existing code is fine
There was a problem hiding this comment.
I can't find any generic region from arn like utility that works with Output from what I can see. The existing utilities are all service specific parsers for example parseQueueArn, parseLambdaEdgeArn and have some specific logic around that.
There was a problem hiding this comment.
probably it still makes sense to extract service-specific arn functions so we can test them. if you update the branch with origin-dev, you'll see new tests for those functions
you can keep them working with regular strings and just do the apply on the component side.
| ```typescript | ||
| // sst.config.ts | ||
| const cluster = new sst.aws.Dsql('MyCluster', { | ||
| multiRegion: { |
There was a problem hiding this comment.
this property doesn't exist in the component
| import type { Dsql } from "./dsql"; | ||
| import { DsqlPeeringProvider } from "./providers/dsql-peering"; | ||
|
|
||
| export interface DsqlPeeringArgs { |
There was a problem hiding this comment.
i believe we no longer need the custom provider since pulumi has a resource for this: https://www.pulumi.com/registry/packages/aws-v6/api-docs/dsql/clusterpeering/
There was a problem hiding this comment.
Should I be removing the dsql peering component completely?
Using the native pulumi peering component means that I would have to do something like this.
`
const witnessRegion = 'us-west-2';
const primary = new sst.aws.Dsql('Primary', {
witnessRegion: witnessRegion,
});
const peerProvider = new aws.Provider('PeerRegion', {
region: 'us-east-2',
});
const peer = new sst.aws.Dsql(
'Peer',
{ witnessRegion: witnessRegion },
{ provider: peerProvider }
);
// Create peering between clusters
new aws.dsql.ClusterPeering('PrimaryPeering', {
identifier: primary.identifier,
clusters: [peer.arn],
witnessRegion,
});
new aws.dsql.ClusterPeering(
'PeerPeering',
{
identifier: peer.identifier,
clusters: [primary.arn],
witnessRegion,
},
{ provider: peerProvider }
);`
Or should we be keeping the peering component so some of this is abstracted?
There was a problem hiding this comment.
i believe we can remove the custom peering component and either provide this as an example or abstract this logic inside the existing DSQL component
an example of the latter would be:
const primary = new sst.aws.Dsql('Primary', {
witness: { region: 'us-east-1' },
peers: [
{ region: 'us-east-2' },
{ region: 'eu-west-1' },
],
});i'm not super familiar with dsql but this latter approach feels very clean
There was a problem hiding this comment.
don't worry i will fix these before merging the pr.
if you want to fix it deleting it and running bun install should work (in fact if you remove the unnecessary dep no bun changes should appear)
| [ | ||
| "aws:dynamodb/table:Table", | ||
| "aws:rds/instance:Instance", | ||
| "aws:dsql/cluster:Cluster", |
There was a problem hiding this comment.
i wonder if we should also add the peering stuff here? can you confirm?
There was a problem hiding this comment.
If you want to keep the peering resource then yes. It doesn't contain any data.
Currently what it will do in multi region is the following
Before Delete
[Cluster 1] -> [Peering relationship] -> [Cluster 2]
After Delete
[Cluster 1] [Cluster 2]
The peering relationship is easy to recreate if needed.
If we add it then it will keep the multi region relationship.
Is this function strictly for resources that hold data? If so Cluster peering isn't the right spot for this imo.
examples/aws-dsql/sst.config.ts
Outdated
| /* # AWS Aurora DSQL Example | ||
| ## How to use | ||
|
|
||
| 1. **Deploy the stack** | ||
|
|
||
| ```bash | ||
| npm install | ||
| sst deploy | ||
| ``` | ||
|
|
||
| 2. **Test the connection** | ||
| The Lambda function when invoked will connect to the DSQL cluster and run a test query. | ||
|
|
||
| 3. **View the results** | ||
| Check the Lambda function logs to see the successful connection and query results. | ||
|
|
||
| ## Clean up | ||
|
|
||
| ```bash |
There was a problem hiding this comment.
can we restructure these docs to follow the pattern of the others?
if you don't want to bother i will do it before merging
Built off the back of PR #5969, with testing and additional code suggested by @natac13. Also merged in latest dev with the latest 4.x version.
This adds the
sst.aws.Dsqlcomponent for Amazon Aurora DSQL.What's included
Single-region
Creates an Aurora DSQL cluster in the current region. Linked functions get
endpointandregionviaResource.Multi-region
Creates a primary cluster in the current region and a peer cluster in another region. An AWS provider for the peer region is created automatically so you don't need to configure one yourself. The two-way
ClusterPeeringhandshake that DSQL requires is also handled automatically. Linked functions getendpointandregionfor the primary cluster andpeer.endpointandpeer.regionfor the peer. The peer cluster is accessed via its public endpoint which is secured by short-lived IAM auth tokens and TLS.VPC endpoints
You can optionally create AWS PrivateLink interface endpoints inside a VPC. Two types are supported:
managementfor control plane operations andconnectionfor PostgreSQL client connections. Both default tofalsesince they cost money and need to be explicitly enabled. A security group is created automatically that allows inbound TCP/5432 from within the VPC CIDR. When a connection endpoint is created,Resource.MyCluster.endpointautomatically resolves to the private hostname instead of the public one, so your function code doesn't need to change.Dsql.getReference an existing cluster by its identifier, useful for sharing a cluster across stages. Supports both single-region and multi-region clusters via
peerIdentifierandpeerRegion.Tested