Create Fastplong.sh handler with direct concatenation approach

[Copilot]

Problem

The Fastplong.sh handler needs to avoid FIFO-based streaming that can block indefinitely on HPC systems, eliminate duplicate function definitions, and provide proper error handling with immediate disk writes.

Changes

New handler implementation

• Direct file concatenation to disk before processing (no FIFOs)
• Single process_sample_direct() function with explicit parameters: sampleName, sampleFiles, outDirectory, skipAdapter, seqHand
• Automatic PacBio HiFi detection (via filename patterns hifi|ccs) to conditionally skip adapter trimming
• Multi-format input support: gzip, bzip2, uncompressed FASTQ

Error handling

• Input file existence validation
• Concatenation verification (non-empty file check)
• Exit code validation on fastplong execution
• Output file creation confirmation
• Cleanup of temporary files on all paths

Structure

• Follows repository patterns from Quality_Trimming.sh and SAM_Processing_Picard.sh
• GNU parallel for multi-sample processing
• Dependency array declaration for external tooling
• Function exports for parallel execution

# Direct concatenation approach
tmpConcatFile="${out}/${sampleName}_concat_temp.fastq"
for file in "${fileArray[@]}"; do
    gzip -cd "${file}" >> "${tmpConcatFile}"  # Writes immediately to disk
done

# Then process with validation
if ! fastplong --input "${tmpConcatFile}" --output "${outputFile}"; then
    echo "Error: fastplong failed for sample ${sampleName}" >&2
    rm -f "${tmpConcatFile}"
    exit 1
fi

Original prompt

Problem

The Handlers/Fastplong.sh script has several critical issues that can cause it to hang or timeout on HPC systems:

1. FIFO blocking: The script uses named pipes (mkfifo) to stream data, which can block indefinitely if fastplong doesn't consume data fast enough or fails silently
2. Duplicate function definition: The process_sample_stream() function is defined twice (lines 73 and 92), with the second definition missing the "${skipAdapter}" parameter
3. No incremental disk writes: Output is only written after the entire sample completes processing, increasing risk of timeout
4. Missing error handling: If fastplong fails, the FIFO will hang forever with no error message
5. Inconsistent with repository patterns: The script should follow similar patterns to other handlers in the repository (like Quality_Trimming.sh and SAM_Processing_Picard.sh)

Solution

Refactor the Fastplong.sh handler to:

1. Remove FIFO approach: Replace with direct file concatenation approach that writes to disk immediately
2. Remove duplicate function: Keep only one process_sample_direct() function
3. Add error handling: Check exit codes and validate output files were created
4. Follow repository conventions: Match the style and error handling patterns used in other handlers like Quality_Trimming.sh
5. Keep adapter trimming logic: Preserve the existing logic for detecting whether to skip adapter trimming based on PacBio HiFi detection

Implementation Details

The refactored script should:

• Replace process_sample_stream() with process_sample_direct() that concatenates files first, then runs fastplong
• Add validation that output files were created successfully
• Add proper error messages with exit codes
• Clean up temporary concatenated files after processing
• Maintain all existing functionality for adapter trimming detection
• Follow the export function pattern used in other handlers

Files to Modify

• Handlers/Fastplong.sh - Complete refactor of the sample processing logic

Reference

The fix should be similar in structure to how Quality_Trimming.sh handles sample processing, with proper error handling and immediate disk writes.

This pull request was created from Copilot chat.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.