Repository navigation
[No QA] Deploy staging web builds to the staging Static Assets Worker - #104104
Conversation
Each Worker version only contains the files uploaded with it, so tabs still running an older build would lose its chunks on every deploy. Files from the current build are never replaced.
The test-staging job becomes a matrix over test-staging and staging. The staging Worker serves no traffic until staging.new.expensify.com is routed to it in Cloudflare, and stays continue-on-error until then.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@stitesExpensify Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button] |
|
hold actually, shouldn't have been out of draft yet |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 45ae241c9c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| done < <(gh release list --repo Expensify/App --exclude-drafts --limit 50 --json tagName \ | ||
| --jq "[.[] | select(.tagName | test(\"-staging$\"))][:$RELEASE_COUNT][].tagName") |
There was a problem hiding this comment.
When a staging deployment is retried after createRelease has already published its release—or when that concurrently running job publishes before this query—the descending release list includes the current release first, so [:$RELEASE_COUNT] downloads the current build plus only nine genuinely previous builds. This silently shortens the promised ten-release compatibility window and can make tabs from the tenth prior release lose their chunks; exclude the current deployment tag before applying the limit. I confirmed from gh release list --help that its default ordering is descending (--order ... (default "desc")).
Useful? React with 👍 / 👎.
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| # `staging` serves no traffic until staging.new.expensify.com is routed to its Worker in Cloudflare. |
There was a problem hiding this comment.
This means the PR can go whenever and we'll make the switch when ready via dashboard
| /** | ||
| * Copies files from previous web builds into the current one when the current build doesn't have them, so tabs still | ||
| * running an older build can load its chunks after a deploy. Files with fixed names exist in every build and are | ||
| * never replaced, so in practice only content-hashed and versioned files are carried forward. | ||
| * | ||
| * node scripts/carryForwardPreviousBuilds.ts <current dist> <previous dist>... | ||
| * | ||
| * List previous builds newest first. | ||
| */ |
There was a problem hiding this comment.
@AndrewGable @roryabraham curious for your thoughts here
so with cloudfront/s3, we leave old deploy files there forever and don't wipe them out
Lets say someone opens staging on build N, then the page knows to load the assets for build N
Later on, we deploy build N+1, and now the assets have a different name but the previous user still has a tab open on the previous version. This works because on S3/CloudFront, we never delete the files that get uploaded.
With Static Workers, we're bundling everything into the worker, meaning the old assets don't exist. To tackle that, it suggested we just keep up to the last 10 releases in the worker bundle available. The alternative is we don't do this at all, and let the app recover by force reloading. Is force reloading a problem?
A retried deploy, or one whose release is published while the job runs, counted its own release as one of the previous 10, so only nine older builds were carried forward. The current release is identified from dist/version.json.
A Worker version only contains the current build, so a tab running an older build gets a 404 for any chunk that has since changed. When the current build misses a file requested by a script or image, the staging Workers now look it up in the new-expensify-web-assets-staging bucket. The current build always takes priority, and navigations never read from the bucket.
Uploading only the new files of each build keeps CI time constant however much history the Workers can serve, and the bucket's 60-day lifecycle rule bounds that history. The upload is skipped with a warning until the R2 credentials are added.
| exit 0 | ||
| fi | ||
| aws configure set default.s3.preferred_transfer_client classic | ||
| aws s3 sync dist s3://new-expensify-web-assets-staging/ \ |
There was a problem hiding this comment.
Do we currently have separate s3 buckets for previous staging and production builds?
There was a problem hiding this comment.
Yeah but we only have staging setup here for now, so left the production one out
| WRANGLER_SEND_METRICS: false | ||
|
|
||
| - name: Verify deploy | ||
| if: ${{ matrix.environment == 'test-staging' }} |
There was a problem hiding this comment.
Why only verify on test-staging?
There was a problem hiding this comment.
We aren't mapping the worker to staging here yet, but rather just deploying to it.
I'll have to update the route in the console after.
The alternative is we manage the route in the wrangler config file here and have this PR act as a staging cutover once it's merged.
Maybe that's fine? I guess I'm being overly cautious that someone won't mess with the route hostname and miss it in a review, because that has the potential to cause a site outage with a bad edit.
There was a problem hiding this comment.
I think it's fine to leave things that won't change often and have a high risk of causing outages (e.g. the host name), on a dashboard that only ring0 has access to change.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 011dc3a2f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| aws configure set default.s3.preferred_transfer_client classic | ||
| aws s3 sync dist s3://new-expensify-web-assets-staging/ \ | ||
| --endpoint-url "https://$CLOUDFLARE_ACCOUNT_ID.r2.cloudflarestorage.com" \ | ||
| --size-only --no-progress \ |
There was a problem hiding this comment.
In the inspected webDeployCloudflareWorkers upload, --size-only causes every unchanged content-hashed file to be skipped because AWS documents that this flag uses object size as the sole sync criterion. Consequently, an asset introduced more than 60 days ago but still referenced by yesterday's build retains its original age and can be deleted by the bucket lifecycle; once today's build removes it, a tab from yesterday immediately gets a 404 instead of the promised fallback. Refresh existing objects on each deploy or namespace them by build and expire the namespaces. See the AWS CLI sync documentation and Cloudflare R2 lifecycle behavior.
Useful? React with 👍 / 👎.
With --size-only, a file was uploaded once and skipped afterwards, so the bucket's 60-day lifecycle rule could delete a font, image or chunk that builds still used. Once a later build changed it, tabs on the previous build got a 404 instead of the fallback. Re-uploading everything restarts each file's expiry while it is in use.
|
All you @roryabraham |
|
Forgot Rory is OOO so I will just merge this so we can keep moving forward |
|
🚧 justinpersaud has triggered a test Expensify/App build. You can view the workflow run here. |
|
🧪🧪 Use the links below to test this adhoc build on Android, iOS, and Web. Happy testing! 🧪🧪
|
|
✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release. |
|
🚀 Deployed to staging by https://github.com/justinpersaud in version: 9.5.14-0 🚀
|
4 participants
Explanation of Change
Move staging over the Cloudflare Static Worker
Fixed Issues
For https://github.com/Expensify/Expensify/issues/688126
PROPOSAL:
Tests
Offline tests
N/A. Deploy pipeline only.
QA Steps
// TODO: These must be filled out, or the issue title must include "[No QA]."
No QA. staging.new.expensify.com and new.expensify.com are still served the same way.
PR Author Checklist
### Fixed Issuessection aboveTestssectionOffline stepssectionQA stepssectionAvatar, I verified the components usingAvatarare working as expected)npm run compress-svg)Avataris modified, I verified thatAvataris working as expected in all cases)Designlabel and/or tagged@Expensify/designso the design team can review the changes.mainbranch was merged into this PR after a review, I tested again and verified the outcome was still expected according to theTeststeps.Screenshots/Videos
Android: Native
Android: mWeb Chrome
iOS: Native
iOS: mWeb Safari
MacOS: Chrome / Safari