Page Comparison

Demultiplexing and splitting

The directory for the raw sequence data (typically gz compressed; use run_pigz.sh and run_unpigz.sh to compress and decompress with multithreaded pigz, using SLURM) and the parsed and split reads is /project/microbiomegtl/data_queue/seqraw/5ALA2TRNL/rawdata. Files for individual samples will be in /project/microbiomegtl/data_queue/seqraw/5ALA2TRNL/rawdata/sample_fastq/.

Demultiplexing

...

mkdir -p /gscratch/grandol1/5ALA2TRNL/rawdata
cd /gscratch/grandol1/5ALA2TRNL/rawdata
unpigz --to-stdout /project/microbiomegtl/data_queue/seqraw/5ALA2TRNL/rawdata/5AL_Redo_S12TRNL_R1_001.fastqfq.gz | split -l 16000000 -d --suffix-length=3 --additional-suffix=.fastq - 5ALA2TRNL_R1_ ;
unpigz --to-stdout /project/microbiomegtl/data_queue/seqraw/5ALA2TRNL/rawdata/5AL2TRNL_Redo_S1_R2_001.fastqfq.gz | split -l 16000000 -d --suffix-length=3 --additional-suffix=.fastq - 5ALA2TRNL_R2_

making 94 R1 files and 94 R2 files, with structured names (e.g., for the R1 set):

/gscratch/grandol1/5ALA2TRNL/rawdata/5ALA2TRNL_R1_000.fastq
/gscratch/grandol1/5ALA2TRNL/rawdata/5ALA2TRNL_R1_001.fastq
etc.

Stalled here on 1-18-2023

...

run_parse_count_onSplitInput.pl also writes to /gscratch.

NS52TRNL_Demux.csv is used to map MIDS to sample names and projects.

...

The info lines for each read in parsed_NS*2TRNL_R1.fastq and parsed_NS*2TRNL_R2.fastq have the locus, the forward mid, the reverse mid, and the sample name. These can be used with the demux key to separate reads into loci, projects, and samples, in the folder sample_fastq/. The reads are in separate files for each sequenced sample, including replicates. The unique combination of forward and reverse MIDs (for a locus) is part of the filename and allows replicates to be distinguished and subsequently merged.

...

Trim, merge and filter reads

In /project/microbiomegtl/data_queue/seqraw/5ALA2TRNL/tfmergedreads , we used run_slurm_mergereads.plto crawl the project folders and sample files (created in the splitting step above) to merge read pairs, and filter based on base quality. This script conforms to the steps in https://microcollaborative.atlassian.net/wiki/spaces/MICLAB/pages/1123778569/Bioinformatics+v3.0?focusedCommentId=1280377080#comment-1280377080, including trimming primers, and joining unmerged reads. This writes a new set of fasta files for each sample and project, rather than fastq, to be used in subsequent steps. These files are found in the 16S/ and ITS/ folders in tfmergedreads/. For example, see contents of /project/microbiomegtl/data/seqraw/NS52TRNL/tfmergedreads/16S/

Within each of these directories are files for the trimmed, merged, and filtered reads, in subfolders trimmed/, joined/, and unmerged/ (the last one is used as a working directory, should be empty; unmerged reads are filtered and joined and put in joined/ if they can be joined; the joined directory can be empty, if all unmerged reads were coligos for example).

...

I used commands.R in that folder to make a plot of numbers of reads per sample (horizontal axis) and the number reads that were removed because they did not merge, or did meet quality criteria and were filtered out (vertical axis). Purple is for 16S and orange is for ITS. It might be interesting to do that plot for each of the projects in the library (TODO), and possibly to have marginal histograms (or put quantiles on the plots).

...

Make OTU table

In /project/microbiome/data_queue/seq/NS5/otu, I ran run_slurm_mkotu.pl, which I modified to also pick up the joined reads (in addition the merged reads).

Make coligo table

In /project/microbiomegtl/data_queue/seqraw/5ALA2TRNL/coligoISD, /project/microbiomegtl/data/seqraw/5ALA2TRNL/coligoISD, and /project/microbiomegtl/data/seqraw/5ALA2TRNL/coligoISD, there are 16S and ITS directories for all projects. These contain a file named coligoISDtable.txt with counts of the coligos and the ISD found in the trimmed forward reads, per sample. The file run_slurm_mkcoligoISDtable.pl has the code that passes over all of the projects and uses vsearch for making the table.

...