Using Versioning for AWS GSIs
No Maintenance Windows Needed
This is a pattern I haven’t seen mentioned anywhere else. We use it at razroo.com and I’m surprised it hasn’t been mentioned anywhere else due to making things run so smoothly.
AWS allows for something called Global Secondary Indexes(GSI). GSIs allow for a new secondary key to be used within DynamoDB beyond the original secondary key.
This allows for direct querying of data, without the need of querying and then filtering thereafter. Performance is faster using GSI’s
Can read more here: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html
The Problem With Global Secondary Indexes
Updating Existing GSI
AWS does not allow for an existing GSI to be updated. If you want to add an additional value for the GSI that can be returned such as firstName and lastName you have to delete an existing GSI.
Why does AWS only allow a GSI to be deleted and re-added, instead of allowing update to happen as well? Well it’s a discussion: https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/229
Right now the API is limited to delete and re-creation.
This can cause problems if have an existing API in production and/or development that uses this API.
GSI Backfilling
Once a GSI is created, AWS will have to run through every row to re-index. This can also cause an index once created, to take 5–10 minutes for this backfilling to complete based on how many rows are in the DB.
Solution — GSI Versioning + Alternating GSIs
So, if you find yourself in a:
- Mid-sized to Large team
- Do not want to interrupt developers in dev
- Want to keep runtime of production at 100%
You can version your new updates as new indexes. For instance user-index would become `user-index-v2`
Once the new version is built on dev, test, and production, can then delete the old GSI and then swap our your template variable in your cloudformation with this new index for a seamless transition.
This is solid architecture.
