Page Comparison

...

mkdir -p /gscratch/grandol1/ALF16S/rawdata
cd /gscratch/grandol1/ALF16S/rawdata
unpigz --to-stdout /project/gtl/data/raw/ALF1/16S/rawdata/1ALF_16S_P1_1.fq.gz | split -l 16000000 -d --suffix-length=3 --additional-suffix=.fastq - ALF16S_R1_
unpigz --to-stdout /project/gtl/data/raw/ALF1/16S/rawdata/1ALF_16S_P1_2.fq.gz | split -l 16000000 -d --suffix-length=3 --additional-suffix=.fastq - ALF16S_R2_

Stopped here on 8-30-22

making 265 R1 files and 266 R2 files, with structured names (e.g., for the R1 set):

...

In a sequence library’s rawdata/ directory (e.g., /project/gtl/data/raw/ALF1/16S/rawdata/) I made run_aggregate.sh, to run aggregate_usearch_fastx_info.pl with a slurm job. Summaries are written to summary_sample_fastq.csv.

cd /project/gtl/data/raw/ALF1/16S/rawdata/

mv sample_fastq/ /project/gtl/data/raw/ALF1/

cd /project/gtl/data/raw/ALF1/ITS/rawdata/sample_fastq

mv * /project/gtl/data/raw/ALF1/sample_fastq/

Tfmergedreads Perl Issue fixing

cd /project/gtl/data/raw/ALF1/

mkdir rawdata

cd rawdata

cp /project/gtl/data/raw/ALF1/16S/rawdata/sample_fastq ./

cd 16S/ALF16S21

for f in *-*; do mv -i "$f" "${f//-/_}"; done

sed -i 's/-/_/g' *

cd ../../ITS/ALF16S21

for f in *-*; do mv -i "$f" "${f//-/_}"; done

sed -i 's/-/_/g' *

for f in *EMPTY*; do mv -i "$f" "${f//EMPTY/Emp}"; done

for f in *Unknown*; do mv -i "$f" "${f//Unknown/Unk}"; done

for f in *Soil*; do mv -i "$f" "${f//Soil/S}"; done

Trim, merge and filter reads

In /project/gtl/data/raw/ALF1/16S/rawdata/tfmergedreads , we used run_slurm_mergereads.plto crawl the project folders and sample files (created in the splitting step above) to merge read pairs, and filter based on base quality. This script conforms to the steps in https://microcollaborative.atlassian.net/wiki/spaces/MICLAB/pages/1123778569/Bioinformatics+v3.0?focusedCommentId=1280377080#comment-1280377080, including trimming primers, and joining unmerged reads. This writes a new set of fasta files for each sample and project, rather than fastq, to be used in subsequent steps. These files are found in the 16S/ and ITS/ folders in tfmergedreads/. For example, see contents of /project/gtl/data/distributionraw/ALF1/16S/rawdatatfmergedreads/

Within each of these directories are files for the trimmed, merged, and filtered reads, in subfolders trimmed/, joined/, and unmerged/ (the last one is used as a working directory, should be empty; unmerged reads are filtered and joined and put in joined/ if they can be joined; the joined directory can be empty, if all unmerged reads were coligos for example).

...

In /project/gtl/data/raw/ALF1/16S/rawdata/otu, I ran run_slurm_mkotu.pl, which I modified to also pick up the joined reads (in addition the merged reads).

...

In /project/gtl/data/raw/ALF1/16S/coligoISD and /project/gtl/data/raw/ALF1/16S/otu there are 16S and ITS directories for all projects. These contain a file named coligoISDtable.txt with counts of the coligos and the ISD found in the trimmed forward reads, per sample. The file run_slurm_mkcoligoISDtable.pl has the code that passes over all of the projects and uses vsearch for making the table.

5-2-2022 Gregg Randolph transferred all of this to /project/gtl/data/distribution/ALF1/

...