Push sources and download translations in same PR/branch

[NBuhinicek]

Hey there 👋

Our current setup with crowdin actions is made out of two scheduled workflows

1. one that pushes sources each night at 2am
2. one that pulls translations each night at 4am

That is not ideal for us since we end up with a PR that needs to be approved in isolation from the real changes and it ends up one day later on production than the actual changes/sources do.

I want to make a workflow were the translations are already pulled to the branch that has the new sources - that way, both the new/updated sources and its translations would end up in the same production deploy. It's a bit surprising that I can't find an example of a workflow so I'm asking for help here 🙏

I tried with

name: "[Crowdin] Upload sources and download translations"

on:
  push:
    paths:
      - "config/locales/**/en_US.yml"
    ..without master, develop and few other branches..

permissions:
  contents: write

jobs:
  crowdin-sync:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Upload Sources to Crowdin
        uses: crowdin/github-action@v2
        with:
          upload_sources: true
          upload_translations: false
          download_translations: false
          create_pull_request: false
          push_translations: false
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
          CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}

      - name: Download Translations from Crowdin
        uses: crowdin/github-action@v2
        with:
          upload_sources: false
          upload_translations: false
          download_translations: true
          crowdin_branch_name: ${{ github.ref_name }}
          localization_branch_name: ${{ github.ref_name }}
          create_pull_request: false
          commit_message: "crowdin: Download translations"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
          CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}        

But it's like the two steps are too quick. Like the TM and AI pre-translation steps configured in the crowdin project are not processed between these two steps.

I saw your AI guide but I don't think that I need this step in the workflow since we have that configured in the project 🤔 or do I?

Can you advise me on how to achieve the workflow that I described?

And also - in this workflow, I should use the crowdin_branch_name because there's 10+ devs working on the same repo. If I figured it out correctly, each branch and it's words count towards the pricing limit of our account. We are working on a repo with 90k source words so making like 10,20 copies (with deleting merged crowdin branches and creating new ones constantly) would increase our pricing a lot. Is this the only way to have a workflow like I described?

[andrii-bodnar]

Hi! That's a great question.

The only thing that comes to mind is using a Webhook-based approach with branching in Crowdin. Here's how the flow would work:

1. A developer pushes a new branch with new source strings to GitHub.
2. The Crowdin GitHub Action uploads this branch to Crowdin for translation.
3. The new strings are translated and approved.
4. Crowdin sends a webhook about the finished translation/proofreading process.
5. The GitHub Action listens for that webhook, downloads the translations, and pushes them to the branch.

The main challenge with this workflow is that the File Fully Translated or File Fully Reviewed events only contain the branchId. It would be difficult to map this ID to the actual branch name on the Action's side.

However, we already have an improvement idea to add a branch name to the webhook payload.

[NBuhinicek]

Thans for the quick answer @andrii-bodnar 🙏

Do we need this webhook based approach if we are relying on AI pre-translations and actually are not approving the translations? We often mark the translations as approved once they are already in production for some period of time. The AI translations are almost always what we need so we don't spend a lot of time on approvals 😅 usually until someone reports a better translation

Figured the settings for not having duplicate strings - that will solve my concern regarding pricing.

[andrii-bodnar]

In this case, you just need the Automatic AI Pre-Translation workflow. It will Pre-Translate the new source strings and immediately push them to GitHub.

Also, please take a look at this discussion - #280.

[NBuhinicek]

Yep, went with that 🤙

Added a separate workflow to delete the crowdin branch on PR merges.

Also, added a workflow that uploads translations when PRs get merged into master - what do you think about that?

name: "[Crowdin] Upload translations"

on:
  push:
    branches:
      - "master"
  workflow_dispatch:

jobs:
  upload-translations:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Upload translations to Crowdin
        uses: crowdin/github-action@v2
        with:
          upload_sources: false
          upload_translations: true
          download_translations: false
          create_pull_request: false
          push_translations: false
        env:
          CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
          CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}

[andrii-bodnar]

@NBuhinicek the translation upload process is primarily for the initial setup when you have existing translations that need to be uploaded to Crowdin.

If you use branches in Crowdin, translations added to your feature branch will migrate to the main branch when the feature branch is deleted. Just make sure the feature branch has been merged on GitHub and synced with Crowdin.

[NBuhinicek]

translations added to your feature branch will migrate to the main branch when the feature branch is deleted

Another question then - how to handle crowdin branches for PRs that don't get merged (just closed instead)? If I'm following you, I can't delete that crowdin branch because it will migrate it's translations to the main branch when it actually shouldn't do that. The PR was just closed, its changes weren't accepted/merged.

[andrii-bodnar]

I believe it's possible to adjust the "Cleanup Crowdin branch" workflow trigger to handle the merge event only.

[NBuhinicek]

Hey @andrii-bodnar, a quick follow-up on this.

The pre-translate-with-AI + branch-delete workflows work great for small repos with a limited number of source files.

However, our frontend application repo has ~18k source strings across 2k JSON files. Each pre-translate-with-AI job runs for about 30 minutes, which doesn't scale well when most PRs introduce only a handful of changes (e.g., 10–30 strings in a few files).

Do you have any suggestions for a workflow that would create a new branch on Crowdin and only push the sources that actually changed in the PR?
Thanks!

[andrii-bodnar]

Thank you for your response and the feature suggestion. I apologize if I misunderstood the initial request; I was assuming the crowdin pre-translate command was the primary source of the delay. To confirm, is the performance issue actually occurring during the "Upload Sources to Crowdin" step?

[andrii-bodnar]

@NBuhinicek we are going to validate the idea of introducing caching to the source upload process, which potentially should drastically reduce execution time for projects with many files when only a few files change.

I'll keep you posted here!

[NBuhinicek]

To confirm, is the performance issue actually occurring during the "Upload Sources to Crowdin" step?

I would say primarily during the “Upload sources” step, but also with translations pull.

This was a workflow run for a commit that changed only 5 sources :/

I'll keep you posted here!

Great, thanks! ❤️

Caching could help, but consider the translations pull step too. My previous proposal is focused on reducing the Crowdin branch size itself. If we could create a Crowdin branch that only contains the changed / needed sources (and their translations), that could be as fast as it gets.

[finebel]

My previous proposal is focused on reducing the Crowdin branch size itself

@NBuhinicek I haven't experimented with it myself (yet), but you might be able to use the crowdin file upload CLI command to achieve this.

[NBuhinicek]

@finebel in that case, I would have to manually do the name: Upload Sources to Crowdin step? 🤔

To create a new branch on crowdin, to go over the changed source files in the branch and to upload them to that branch.

[andrii-bodnar]

Hi @NBuhinicek!

We have just released an experimental caching feature that should drastically reduce the execution time of the upload sources command.

You can find more details about the implementation here: crowdin/crowdin-cli#959.
GH Action documentation: Caching source files for faster uploads.

I look forward to receiving your feedback!

[NBuhinicek]

Hey @andrii-bodnar!

I checked the caching docs out and it sounds like what we need.

I tried to set it up on a branch in my repo. When my workflow runs, the cache step returns
Cache not found for input keys: Linux-crowdin-cache-update/crowdin-cache-upload-files-96032e6a4f799578fffbe8356dd012f8eeb76362f896a7d2d0124326665263d6, Linux-crowdin-cache-update/crowdin-cache-upload-files-. Thought that maybe I should run it again so that the next workflow run would find the branch cache key but got the same msg.

I'm I missing something? Added the cache step and upload_sources_args to the workflow. Only that is mentioned in the docs 🤔

name: '[Crowdin] Pre-translate with AI'

on:
  push:
    paths:
      - 'config/locales/**/en_US.yml'
    branches-ignore:
      - "master"
      - "staging"
      - "latest"
      - "f/pentest"
  workflow_dispatch:

jobs:
  crowdin-process:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Cache Crowdin source files
        uses: actions/cache@v4
        with:
          path: ~/.crowdin/cache.json
          key: ${{ runner.os }}-crowdin-cache-${{ github.ref_name }}-${{ hashFiles('**/crowdin.yml') }}

      - name: Upload Sources to Crowdin
        uses: crowdin/github-action@v2
        with:
          upload_sources: true
          upload_sources_args: '--cache'
          upload_translations: false
          download_translations: false
          create_pull_request: false
          push_translations: false
          crowdin_branch_name: ${{ github.ref_name }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
          CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}

      - name: Pre-translate with AI
        uses: crowdin/github-action@v2
        with:
          command: 'pre-translate'
          command_args: '--method ai --ai-prompt=16007 --branch=${{ github.ref_name }} --translate-untranslated-only'
        env:
          CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
          CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}

      - name: Download Translations from Crowdin
        uses: crowdin/github-action@v2
        with:
          upload_sources: false
          upload_translations: false
          download_translations: true
          create_pull_request: false
          crowdin_branch_name: ${{ github.ref_name }}
          localization_branch_name: ${{ github.ref_name }}
          commit_message: 'crowdin: Download translations'
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          CROWDIN_PROJECT_ID: ${{ secrets.CROWDIN_PROJECT_ID }}
          CROWDIN_PERSONAL_TOKEN: ${{ secrets.CROWDIN_PERSONAL_TOKEN }}

[andrii-bodnar]

Hi @NBuhinicek, thanks for the details! I think I understand what's happening. Perhaps we should use actions/cache@v4 twice: once before the Crowdin action to restore the existing cache, and again after the Crowdin action to save it. This is because the cache.json file will be created during the upload of the source files.