add BIDS Conversion pipeline #29

lucie271 · 2025-11-03T15:35:01Z

Overview

This PR introduces the BIDS conversion pipeline for the SAILS dataset.
It automates video processing, stabilization, and conversion from raw videos into a standardized BIDS-compatible structure.

The main script for the BIDS conversion is located at:

src/BIDS_convertor.py

and corresponding tests are available in:

src/tests/test_BIDS_convertor.py

Requirements

Before running the pipeline, please ensure:

Poetry is installed (see instructions in the project’s README.md)
FFmpeg ≥ 6.0 is available in your environment
All Python dependencies are installed via Poetry (poetry install)

For detailed setup instructions, refer to the README.

Usage

To launch the BIDS conversion process, use the submission scripts (located in the folder jobs), with dependency as follows :

jid=$(sbatch --parsable jobs/run_bids_convertor.sh)
sbatch --dependency=afterok:$jid jobs/merge_cleanup.sh

Input and Output

Input videos:

/orcd/data/satra/002/datasets/SAILS/Phase_III_Videos/Videos_from_external_standardized
Output (final BIDS dataset):

/orcd/scratch/bcs/001/sensein/sails/BIDS_data

The pipeline automatically:

Extracts and stabilizes raw videos
Converts metadata into BIDS format
Generates standardized folder structure and derivative files
Produces per-task processing logs and merged summaries

gemini-code-assist · 2025-11-03T15:36:01Z

Summary of Changes

Hello @lucie271, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request delivers a robust and automated BIDS conversion pipeline for the SAILS dataset. It streamlines the entire process from raw video ingestion to the generation of a fully BIDS-compliant dataset, complete with preprocessed videos, extracted audio, and rich metadata. The new architecture supports parallel execution via Slurm, making it highly scalable for large-scale video data processing and ensuring data standardization for future research.

Highlights

BIDS Conversion Pipeline: Introduced a new, comprehensive pipeline to automate the conversion of raw SAILS dataset videos into a standardized BIDS-compatible structure. This includes video processing, stabilization, and metadata conversion.
Parallel Processing with Slurm: Implemented Slurm array job scripts (jobs/run_bids_convertor.sh and jobs/merge_cleanup.sh) to enable efficient, parallel processing of video chunks, significantly speeding up the conversion for large datasets.
Enhanced Video Processing: The core BIDS_convertor.py script has been extensively refactored to include robust video stabilization (using vidstab), denoising, and standardization, along with improved error handling for FFmpeg operations.
Dynamic Metadata and Event File Generation: Improved metadata handling, including dynamic task information in JSON sidecar files and a more comprehensive events.tsv file that incorporates all available behavioral coding data from Excel, or generates dummy data if not available.
Dependency Updates and Configuration: Updated project dependencies (poetry.lock, pyproject.toml) to include pandas, opencv-python, openpyxl, and types-pyyaml. A new configuration file (configs/config_bids_convertor.yaml) centralizes pipeline settings.
Documentation and Usage Instructions: The README.md has been updated with detailed instructions on setting up the Poetry environment, installing FFmpeg with vidstab, and launching the BIDS conversion process using the provided Slurm scripts.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a comprehensive BIDS conversion pipeline. The changes represent a significant refactoring, moving towards a more modular, robust, and parallelizable architecture suitable for a cluster environment. The code quality is generally high, with substantial improvements in error handling and data processing logic. However, there are a few critical and high-severity issues, primarily a hardcoded path in a shell script that severely impacts portability, and some unsafe file operations. I've also included several medium-severity suggestions to improve maintainability, robustness, and test correctness.

jobs/run_bids_convertor.sh

src/sailsprep/BIDS_convertor.py

src/tests/test_BIDS_convertor.py

jobs/merge_cleanup.sh

gemini-code-assist · 2025-11-03T15:38:49Z

src/sailsprep/BIDS_convertor.py

+    if participant_id.endswith(" 2"):
+        participant_id = participant_id[:-2].strip()


This hardcoded logic to handle participant IDs ending in " 2" is very specific to the current dataset's quirks. This is fragile and may not be obvious to future maintainers. Consider making this logic more generic or configurable, or at least add a comment explaining why this specific transformation is necessary.

src/sailsprep/BIDS_convertor.py

src/BIDS_convertor.py

src/sailsprep/BIDS_convertor.py

src/tests/test_BIDS_convertor.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

fabiocat93

great work @lucie271 ! I have left some comments here and there

jobs/merge_cleanup.sh

poetry.lock

src/sailsprep/BIDS_convertor.py

src/BIDS_convertor.py

src/sailsprep/BIDS_convertor.py

fabiocat93 · 2025-11-03T18:08:26Z

src/BIDS_convertor.py

+    dataset_desc = {
+        "Name": "SAILS Phase III Home Videos",
+        "BIDSVersion": "1.9.0",
+        "DatasetType": "domestic videos with audio",


this should simply be "raw" to match with the BIDS rules

this actually might be considered preprocessed if using the standardized folder since those were put through ffmpeg and converted to mp4s to try and standardize all of the filetypes

True.. Would it be better to work with the raw videos folder (not standardized) then ? As the BIDS conversion is also taking care of the mp4 conversion and put through ffmpeg ?

fabiocat93 · 2025-11-03T18:11:52Z

README.md

 - Things
 - These may include a wonderful CLI interface.

 ## Installation


I would add your setup info in here (poetry, ffmpeg, ...)

fabiocat93 · 2025-11-03T18:13:58Z

README.md

 pip install git+https://github.com/sensein/sailsprep.git
 ```

 ## Quick start


I would add in here a list/table of things you can do with sailsprep. that list will include mostly/only your bids_convertor for now. and you can create a link to a second .md file where you explain the details of what that is, how to use it, ...
This way, you don't make the readme.md too crowded but you still promote your contribution and make it findable

wilke0818

overall looks good. mostly just address what fabio has mentioned. if you wanted to clean up teh code a little bit by factoring out some of the BIDs file creation you could look here as an example of how it was done for Bridge2AI with the audio files and such:
https://github.com/sensein/b2aiprep/tree/5ece6c33bd744a3a2b8e578f50c33af8d9e913d0/src/b2aiprep/prepare/resources/b2ai-data-bids-like-template

src/sailsprep/BIDS_convertor.py

wilke0818 · 2025-11-03T19:47:46Z

src/BIDS_convertor.py

+    dataset_desc = {
+        "Name": "SAILS Phase III Home Videos",
+        "BIDSVersion": "1.9.0",
+        "DatasetType": "domestic videos with audio",


this actually might be considered preprocessed if using the standardized folder since those were put through ffmpeg and converted to mp4s to try and standardize all of the filetypes

fabiocat93 · 2025-11-04T18:49:33Z

@lucie271 I have updated some of the dev dependencies as dependabot was suggesting (to fix some vulnerabilities and more). This shouldn't create any big conflict in pyproject.toml and should be easily solvable, but wanted to let you know so that you are aware. if you need any help, please feel free to ask

…into BIDS-conversion Merge pyproject.toml to changes for unit tests

codecov-commenter · 2025-11-04T23:09:08Z

Codecov Report

❌ Patch coverage is 78.94737% with 200 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.03%. Comparing base (9eea44a) to head (78e611e).

Files with missing lines	Patch %	Lines
src/sailsprep/BIDS_convertor.py	66.89%	194 Missing ⚠️
src/tests/test_BIDS_convertor.py	98.35%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #29       +/-   ##
===========================================
+ Coverage   65.59%   80.03%   +14.44%     
===========================================
  Files           5        5               
  Lines         529     1052      +523     
===========================================
+ Hits          347      842      +495     
- Misses        182      210       +28

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fabiocat93

good job!

add BIDS Conversion pipeline

aparnabg and others added 17 commits September 24, 2025 21:01

add bids code

228c945

add log _file.py

739d352

add .sh

1564f16

add bids code

c124e0c

add bids code

725305a

Add participants.tsv population file

cb634ce

Create README.md

ea61c29

Added configuration file for BIDS conversion

74bb9f4

Cleaned src folder

bb918b9

Final script for BIDS conversion added

59b66e3

Added poetry dependencies

27cb826

Modified test for new BIDS convertor script

47227b1

updated README with BIDS-conversion pipeline

acf2e66

final cleaning and merge after script execution

b1d262c

changed number of jobs in submission file

348973b

Added logs to .gitignore

e57bcae

fixed last shell scripts

84f2e35

lucie271 requested a review from fabiocat93 November 3, 2025 15:35

lucie271 requested review from wilke0818 and yibeichan November 3, 2025 15:36

gemini-code-assist bot reviewed Nov 3, 2025

View reviewed changes

lucie271 and others added 4 commits November 3, 2025 10:41

Update jobs/run_bids_convertor.sh

d6ac5b8

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update jobs/merge_cleanup.sh

770e600

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update src/tests/test_BIDS_convertor.py

316f227

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update jobs/merge_cleanup.sh

491a431

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

lucie271 linked an issue Nov 3, 2025 that may be closed by this pull request

PR for BIDS-conversion #24

Closed

fabiocat93 reviewed Nov 3, 2025

View reviewed changes

wilke0818 reviewed Nov 3, 2025

View reviewed changes

untrack poetry.lock and add logs folder

f8ed634

lucie271 added 6 commits November 3, 2025 15:36

BIDS_convertor.py in sailsprep

b50ba76

Fixed little warnings from PR

e16f84b

Fixed warnings in BIDS_convertor.py from PR

04bad65

Cleaned /logs handling

50a80b9

Changed source video to raw folder

0fce931

Update src/tests/test_BIDS_convertor.py

c8699e4

lucie271 and others added 7 commits November 4, 2025 16:06

fixed issues of execution

1e04269

added documentation

9e5c76d

Merge branch 'main' into BIDS-conversion

d80bb46

fixed scripts for unit tests

21003fd

Merge branch 'main' into BIDS-conversion

5bad1f5

Merge branch 'BIDS-conversion' of https://github.com/sensein/sailsprep …

85f3384

…into BIDS-conversion Merge pyproject.toml to changes for unit tests

updated unit test

333f5f4

lucie271 removed the request for review from yibeichan November 4, 2025 23:09

fabiocat93 approved these changes Nov 4, 2025

View reviewed changes

lucie271 added 3 commits November 5, 2025 16:41

Added unit tests

7768f7e

Change number of array

5117e74

Fixed error mypy

78e611e

fabiocat93 merged commit f098f57 into main Nov 6, 2025
6 checks passed

lucie271 deleted the BIDS-conversion branch November 24, 2025 14:52

lucie271 pushed a commit that referenced this pull request Dec 15, 2025

Merge pull request #29 from sensein/BIDS-conversion

ac0946f

add BIDS Conversion pipeline

		if participant_id.endswith(" 2"):
		participant_id = participant_id[:-2].strip()

add BIDS Conversion pipeline #29

add BIDS Conversion pipeline #29

Uh oh!

Conversation

lucie271 commented Nov 3, 2025

Overview

Requirements

Usage

Input and Output

Uh oh!

gemini-code-assist bot commented Nov 3, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fabiocat93 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wilke0818 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fabiocat93 commented Nov 4, 2025

Uh oh!

codecov-commenter commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fabiocat93 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

codecov-commenter commented Nov 4, 2025 •

edited

Loading