Add a Synapse Quarantined Media exchange #7
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The significant change is in the Dockerfile, where we install poetry so we can install the new exchange (and possibly future ones as well). We try to keep the number of layers relatively small to make the image easier to download/less bloated. There's also some indirection via Docker Compose Secrets here for local testing/debugging. Essentially, we're just trying to run the environment variable from the machine down to the container, which requires secrets for reasons. Unfortunately, secrets can only write to files inside the container, so we have to extract the contents of that file back into the environment. That extraction is done in `startup.sh`.
H-Shay
reviewed
Dec 17, 2025
H-Shay
reviewed
Dec 17, 2025
H-Shay
reviewed
Dec 17, 2025
H-Shay
reviewed
Dec 17, 2025
H-Shay
approved these changes
Dec 17, 2025
H-Shay
left a comment
H-Shay
left a comment
There was a problem hiding this comment.
in a perfect pythonic world it would be cool to see docstrings but it's not a huge deal
Member
Author
|
I've added some basic docstrings and fixed the |
H-Shay
reviewed
Dec 18, 2025
|
|
||
| # Now process remote media | ||
| remote_token = checkpoint.remote_from_token if checkpoint else 0 | ||
| remote_media_mxcs, next_batch = self._fetch(False, remote_token) |
There was a problem hiding this comment.
since _fetch (and _hash) can raise an error, would it make sense to try/catch those calls?
Member
Author
There was a problem hiding this comment.
there's not much we can do if we do catch it, so we let the fetcher inside HMA catch+log it.
|
Just one question, otherwise looks good, thanks for the docstrings! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reviewable commit-by-commit (recommended - some commit messages have additional context)
Requires element-hq/synapse#19308 on the server.
The ExchangeAPI comes from python-threatexchange, which is the underlying code behind HMA. The interface is a bit awkward at first glance, but the short version is:
fetchwhenever it wants more data. We don't want to block this for too long, but as long as required is fine._hashand_fetchMuch of the code is inspired by existing exchanges in python-threatexchange.
This has been tested locally. One day we should figure out how to put CI on this.