13.1-Pmea-sRNAseq-ShortStack-R1-reads

Use ShortStack (Axtell 2013; Shahid and Axtell 2014; Johnson et al. 2016)to perform alignment of sRNAseq data and annotation of sRNA-producing genes.

The P.meandrina genome will be used as the reference genome.

---

Inputs:

• Requires trimmed sRNAseq files generated by 08.1-Pmea-sRNAseq-trimming-R1-only.Rmd

  • Filenames formatted: *fastp-R1-31bp-auto_adapters-polyG.fq.gz

• P.meandrina genome FastA. See 12-Pmea-sRNAseq-MirMachine.Rmd for download info if needed.

Outputs:

• See ShortStack outputs documentation for full list and detailed descriptions.

Software requirements:

• Utilizes a ShortStack Conda/Mamba environment, per the installation instructions.

Replace with name of your ShortStack environment and the path to the corresponding conda installation (find this after you’ve activated the environment).

E.g.

# Activate environment
conda activate ShortStack4_env

# Find conda path
which conda

---

1 Set R variables

shortstack_conda_env_name <- c("ShortStack-4.0.3_env")
shortstack_cond_path <- c("/home/sam/programs/mambaforge/condabin/conda")

2 Create a Bash variables file

This allows usage of Bash variables across R Markdown chunks.

{
echo "#### Assign Variables ####"
echo ""

echo "# Trimmed FastQ naming pattern"
echo "export trimmed_fastqs_pattern='*fastp-R1-31bp-auto_adapters-polyG.fq.gz'"

echo "# Data directories"
echo 'export deep_dive_dir=/home/shared/8TB_HDD_01/sam/gitrepos/deep-dive'
echo 'export deep_dive_data_dir="${deep_dive_dir}/data"'
echo 'export output_dir_top=${deep_dive_dir}/F-Pmea/output/13.1-Pmea-sRNAseq-ShortStack-R1-reads'
echo 'export trimmed_fastqs_dir="${deep_dive_dir}/F-Pmea/output/08.1-Pmea-sRNAseq-trimming-R1-only/trimmed-reads"'
echo ""

echo "# Input/Output files"
echo 'export genome_fasta_dir=${deep_dive_dir}/F-Pmea/data'
echo 'export genome_fasta_name="Pocillopora_meandrina_HIv1.assembly.fasta"'
echo 'export shortstack_genome_fasta_name="Pocillopora_meandrina_HIv1.assembly.fa"'
echo 'export mirbase_mature_fasta=mature.fa'
echo 'export mirbase_mature_fasta_version=mirbase-mature-v22.1.fa'
echo 'export genome_fasta="${genome_fasta_dir}/${shortstack_genome_fasta_name}"'
echo ""

echo "# External data URLs"
echo 'export mirbase_fasta_url="https://mirbase.org/download_version_files/22.1/"'
echo ""

echo "# Set number of CPUs to use"
echo 'export threads=46'
echo ""

echo "# Initialize arrays"
echo 'export trimmed_fastqs_array=()'


} > .bashvars

cat .bashvars

#### Assign Variables ####

# Trimmed FastQ naming pattern
export trimmed_fastqs_pattern='*fastp-R1-31bp-auto_adapters-polyG.fq.gz'
# Data directories
export deep_dive_dir=/home/shared/8TB_HDD_01/sam/gitrepos/deep-dive
export deep_dive_data_dir="${deep_dive_dir}/data"
export output_dir_top=${deep_dive_dir}/F-Pmea/output/13.1-Pmea-sRNAseq-ShortStack-R1-reads
export trimmed_fastqs_dir="${deep_dive_dir}/F-Pmea/output/08.1-Pmea-sRNAseq-trimming-R1-only/trimmed-reads"

# Input/Output files
export genome_fasta_dir=${deep_dive_dir}/F-Pmea/data
export genome_fasta_name="Pocillopora_meandrina_HIv1.assembly.fasta"
export shortstack_genome_fasta_name="Pocillopora_meandrina_HIv1.assembly.fa"
export mirbase_mature_fasta=mature.fa
export mirbase_mature_fasta_version=mirbase-mature-v22.1.fa
export genome_fasta="${genome_fasta_dir}/${shortstack_genome_fasta_name}"

# External data URLs
export mirbase_fasta_url="https://mirbase.org/download_version_files/22.1/"

# Set number of CPUs to use
export threads=46

# Initialize arrays
export trimmed_fastqs_array=()

3 Load ShortStack conda environment

If this is successful, the first line of output should show that the Python being used is the one in your [ShortStack](https://github.com/MikeAxtell/ShortStack conda environment path.

E.g.

python: /home/sam/programs/mambaforge/envs/mirmachine_env/bin/python

use_condaenv(condaenv = shortstack_conda_env_name, conda = shortstack_cond_path)

# Check successful env loading
py_config()

python:         /home/sam/programs/mambaforge/envs/ShortStack-4.0.3_env/bin/python
libpython:      /home/sam/programs/mambaforge/envs/ShortStack-4.0.3_env/lib/libpython3.10.so
pythonhome:     /home/sam/programs/mambaforge/envs/ShortStack-4.0.3_env:/home/sam/programs/mambaforge/envs/ShortStack-4.0.3_env
version:        3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:39) [GCC 12.3.0]
numpy:          /home/sam/programs/mambaforge/envs/ShortStack-4.0.3_env/lib/python3.10/site-packages/numpy
numpy_version:  1.26.4

NOTE: Python version was forced by use_python() function

4 Download miRBase mature miRNA FastA

# Load bash variables into memory
source .bashvars

wget \
--directory-prefix ${deep_dive_data_dir} \
--recursive \
--no-check-certificate \
--continue \
--no-host-directories \
--no-directories \
--no-parent \
--quiet \
--execute robots=off \
 ${mirbase_fasta_url}/${mirbase_mature_fasta}

# Rename to indicate miRBase FastA version
mv ${deep_dive_data_dir}/${mirbase_mature_fasta} ${deep_dive_data_dir}/${mirbase_mature_fasta_version}

ls -lh "${deep_dive_data_dir}"

total 13M
drwxr-xr-x 2 sam sam 4.0K Nov  7 14:36 blast_dbs
-rw-r--r-- 1 sam sam 3.7M Feb 16 11:36 mirbase-mature-v22.1.fa
-rw-r--r-- 1 sam sam 3.7M Dec  4 11:05 mirbase-mature-v22.1-no_spaces.fa
-rw-r--r-- 1 sam sam 3.7M Nov 17 07:59 mirbase-mature-v22.1-no_U.fa
-rw-r--r-- 1 sam sam 726K Nov  7 14:36 mirgene-mature-all-v2.1.fa
-rw-r--r-- 1 sam sam 726K Nov 17 07:59 mirgene-mature-all-v2.1-no_U.fa

5 Run ShortStack

5.1 Modify genome filename for ShortStack compatability

# Load bash variables into memory
source .bashvars

# Check for FastA file first
# Then create rename file if doesn't exist
if [ -f "${genome_fasta_dir}/${shortstack_genome_fasta_name}" ]; then
  echo "${genome_fasta_dir}/${shortstack_genome_fasta_name} already exists. Nothing to do."
  echo ""
else

  # Copy genome FastA to ShortStack-compatible filename (ending with .fa)
  cp ${genome_fasta_dir}/${genome_fasta_name} ${genome_fasta_dir}/${shortstack_genome_fasta_name}
fi

# Confirm
ls -lh ${genome_fasta_dir}/${shortstack_genome_fasta_name}

/home/shared/8TB_HDD_01/sam/gitrepos/deep-dive/F-Pmea/data/Pocillopora_meandrina_HIv1.assembly.fa already exists. Nothing to do.

-rw-r--r-- 1 sam sam 360M Feb 16 10:28 /home/shared/8TB_HDD_01/sam/gitrepos/deep-dive/F-Pmea/data/Pocillopora_meandrina_HIv1.assembly.fa

5.2 Excecute ShortStack command

Uses the --dn_mirna option to identify miRNAs in the genome, without relying on the --known_miRNAs.

This part of the code redirects the output of time to the end of shortstack.log file.

• ; } \ 2>> ${output_dir_top}/shortstack.log

# Load bash variables into memory
source .bashvars

# Make output directory, if it doesn't exist
mkdir --parents "${output_dir_top}"

# Create array of trimmed FastQs
trimmed_fastqs_array=(${trimmed_fastqs_dir}/${trimmed_fastqs_pattern})


# Pass array contents to new variable as space-delimited list
trimmed_fastqs_list=$(echo "${trimmed_fastqs_array[*]}")


###### Run ShortStack ######
{ time \
ShortStack \
--genomefile "${genome_fasta}" \
--readfile ${trimmed_fastqs_list} \
--known_miRNAs ${deep_dive_data_dir}/${mirbase_mature_fasta_version} \
--dn_mirna \
--threads ${threads} \
--outdir ${output_dir_top}/ShortStack_out \
&> ${output_dir_top}/shortstack.log ; } \
2>> ${output_dir_top}/shortstack.log

5.3 Check runtime

# Load bash variables into memory
source .bashvars

tail -n 3 ${output_dir_top}/shortstack.log \
| grep "real" \
| awk '{print "ShortStack runtime:" "\t" $2}'

ShortStack runtime: 54m16.865s

6 Results

6.1 ShortStack synopsis

# Load bash variables into memory
source .bashvars

tail -n 25 ${output_dir_top}/shortstack.log

Writing final files

Found a total of 35 MIRNA loci


Non-MIRNA loci by DicerCall:
N 7698
22 33
23 32
24 14
21 11

Creating visualizations of microRNA loci with strucVis
<<< WARNING >>>
Do not rely on these results alone to annotate new MIRNA loci!
The false positive rate for de novo MIRNA identification is low, but NOT ZERO
Insepct each mirna locus, especially the strucVis output, and see
https://doi.org/10.1105/tpc.17.00851 , https://doi.org/10.1093/nar/gky1141

Fri 16 Feb 2024 11:25:38 -0800 PST
Run Completed!

real    54m16.865s
user    1086m20.043s
sys 324m13.979s

ShortStack identified 35 miRNAs.

6.2 Inspect Results.txt

# Load bash variables into memory
source .bashvars

head ${output_dir_top}/ShortStack_out/Results.txt

echo ""
echo "----------------------------------------------------------"
echo ""

echo "Nummber of potential loci:"
awk '(NR>1)' ${output_dir_top}/ShortStack_out/Results.txt | wc -l

Locus   Name    Chrom   Start   End Length  Reads   DistinctSequences   FracTop Strand  MajorRNA    MajorRNAReads   Short   Long    21  22  23  24  DicerCall   MIRNA   known_miRNAs
Pocillopora_meandrina_HIv1___Sc0000000:9091-9521    Cluster_1   Pocillopora_meandrina_HIv1___Sc0000000  9091    9521    431 11370   416 1.0 +   GGGGGUAUAGCUCAGUGGUAGA  3774    1971    3797    682 4314    189 417 N   N   NA
Pocillopora_meandrina_HIv1___Sc0000000:53578-53997  Cluster_2   Pocillopora_meandrina_HIv1___Sc0000000  53578   53997   420 434 30  0.9976958525345622  +   GCCUAAGUUGCUUGGAACA 139 432 2   0   0   0   0   N   N   NA
Pocillopora_meandrina_HIv1___Sc0000000:150243-150718    Cluster_3   Pocillopora_meandrina_HIv1___Sc0000000  150243  150718  476 2661    283 0.000751597143930853    -   UGGCUAUGAUGAAAAUGACU    327 1002    356 629 369 138 167 N   N   NA
Pocillopora_meandrina_HIv1___Sc0000000:173728-174150    Cluster_4   Pocillopora_meandrina_HIv1___Sc0000000  173728  174150  423 1253    72  0.9960095770151636  +   UUUGAUUGCUGUGAUCUGGUUG  369 120 2   43  623 435 30  22  N   NA
Pocillopora_meandrina_HIv1___Sc0000000:187562-188072    Cluster_5   Pocillopora_meandrina_HIv1___Sc0000000  187562  188072  511 183 34  0.453551912568306   .   AUAAAUGUCACUACAAGAAACCUGAAAUCGU 25  2   175 1   1   2   2   N   N   NA
Pocillopora_meandrina_HIv1___Sc0000000:485727-486254    Cluster_6   Pocillopora_meandrina_HIv1___Sc0000000  485727  486254  528 327 147 1.0 +   GAUGGGUGUUAUUACUCCUCAGACAGAC    48  111 180 7   11  3   15  N   N   NA
Pocillopora_meandrina_HIv1___Sc0000000:525310-527341    Cluster_7   Pocillopora_meandrina_HIv1___Sc0000000  525310  527341  2032    14555   3093    0.184610099622123   -   UUUUCGUCACUUUCUUCAGCCUCAGAGU    947 210 13371   53  108 310 503 N   N   NA
Pocillopora_meandrina_HIv1___Sc0000000:541262-541723    Cluster_8   Pocillopora_meandrina_HIv1___Sc0000000  541262  541723  462 725 137 0.07862068965517241 -   UUGGACGAAAUUUCGAGGUUCACACUCGUU  80  0   719 1   3   1   1   N   N   NA
Pocillopora_meandrina_HIv1___Sc0000000:551887-552689    Cluster_9   Pocillopora_meandrina_HIv1___Sc0000000  551887  552689  803 833 274 0.20168067226890757 .   UUCAAGGAGGAUGUAUCCACUGUCAUU 90  27  748 12  8   18  20  N   N   NA

----------------------------------------------------------

Nummber of potential loci:
7823

Column 20 of the Results.txt file identifies if a cluster is a miRNA or not (Y or N).

# Load bash variables into memory
source .bashvars

echo "Number of loci characterized as miRNA:"
awk '$20=="Y" {print $0}' ${output_dir_top}/ShortStack_out/Results.txt \
| wc -l
echo ""

echo "----------------------------------------------------------"

echo ""
echo "Number of loci _not_ characterized as miRNA:"
awk '$20=="N" {print $0}' ${output_dir_top}/ShortStack_out/Results.txt \
| wc -l

Number of loci characterized as miRNA:
35

----------------------------------------------------------

Number of loci _not_ characterized as miRNA:
7788

Column 21 of the Results.txt file identifies if a cluster aligned to a known miRNA (miRBase) or not (Y or NA).

Since there are no miRNAs, the following code will not print any output.

The echo command after the awk command is simply there to prove that the chunk executed.

# Load bash variables into memory
source .bashvars

echo "Number of loci matching miRBase miRNAs:"
awk '$21!="NA" {print $0}' ${output_dir_top}/ShortStack_out/Results.txt \
| wc -l
echo ""

echo "----------------------------------------------------------"

echo ""
echo "Number of loci _not_ matching miRBase miRNAs:"
awk '$21=="NA" {print $0}' ${output_dir_top}/ShortStack_out/Results.txt \
| wc -l

Number of loci matching miRBase miRNAs:
78

----------------------------------------------------------

Number of loci _not_ matching miRBase miRNAs:
7746

Although there are loci with matches to miRBase miRNAs, ShortStack did not annotate these clusters as miRNAs likely because they do not also match secondary structure criteria.

6.2.1 Directory tree of all ShortStack outputs

Many of these are large (by GitHub standards) BAM files, so will not be added to the repo.

Additionally, it’s unlikely we’ll utilize most of the other files (bigwig) generated by ShortStack.

# Load bash variables into memory
source .bashvars

tree -h ${output_dir_top}/

/home/shared/8TB_HDD_01/sam/gitrepos/deep-dive/F-Pmea/output/13.1-Pmea-sRNAseq-ShortStack-R1-reads/
├── [ 21K]  shortstack.log
└── [ 20K]  ShortStack_out
    ├── [ 28K]  alignment_details.tsv
    ├── [656K]  Counts.txt
    ├── [160K]  known_miRNAs.gff3
    ├── [1.8M]  known_miRNAs_unaligned.fasta
    ├── [6.9M]  merged_alignments_21_m.bw
    ├── [7.0M]  merged_alignments_21_p.bw
    ├── [6.7M]  merged_alignments_22_m.bw
    ├── [6.8M]  merged_alignments_22_p.bw
    ├── [ 12M]  merged_alignments_23-24_m.bw
    ├── [ 13M]  merged_alignments_23-24_p.bw
    ├── [1.1G]  merged_alignments.bam
    ├── [150K]  merged_alignments.bam.csi
    ├── [ 53M]  merged_alignments_other_m.bw
    ├── [ 54M]  merged_alignments_other_p.bw
    ├── [ 24M]  merged_alignments_sRNA-POC-47-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bw
    ├── [ 34M]  merged_alignments_sRNA-POC-48-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bw
    ├── [ 26M]  merged_alignments_sRNA-POC-50-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bw
    ├── [ 41M]  merged_alignments_sRNA-POC-53-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bw
    ├── [ 56M]  merged_alignments_sRNA-POC-57-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bw
    ├── [ 13K]  mir.fasta
    ├── [994K]  Results.gff3
    ├── [1.5M]  Results.txt
    ├── [183M]  sRNA-POC-47-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam
    ├── [174K]  sRNA-POC-47-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam.csi
    ├── [206M]  sRNA-POC-48-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam
    ├── [171K]  sRNA-POC-48-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam.csi
    ├── [184M]  sRNA-POC-50-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam
    ├── [176K]  sRNA-POC-50-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam.csi
    ├── [250M]  sRNA-POC-53-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam
    ├── [171K]  sRNA-POC-53-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam.csi
    ├── [247M]  sRNA-POC-57-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam
    ├── [161K]  sRNA-POC-57-S1-TP2_R1_001.fastp-R1-31bp-auto_adapters-polyG.bam.csi
    └── [4.0K]  strucVis
        ├── [ 12K]  Cluster_1009.ps
        ├── [ 35K]  Cluster_1009.txt
        ├── [ 12K]  Cluster_1112.ps
        ├── [ 36K]  Cluster_1112.txt
        ├── [ 11K]  Cluster_1171.ps
        ├── [3.8K]  Cluster_1171.txt
        ├── [ 11K]  Cluster_1183.ps
        ├── [ 11K]  Cluster_1183.txt
        ├── [ 12K]  Cluster_1232.ps
        ├── [8.4K]  Cluster_1232.txt
        ├── [ 12K]  Cluster_1432.ps
        ├── [ 27K]  Cluster_1432.txt
        ├── [ 13K]  Cluster_1437.ps
        ├── [ 78K]  Cluster_1437.txt
        ├── [ 13K]  Cluster_1961.ps
        ├── [ 50K]  Cluster_1961.txt
        ├── [ 12K]  Cluster_20.ps
        ├── [ 33K]  Cluster_20.txt
        ├── [ 12K]  Cluster_2143.ps
        ├── [ 44K]  Cluster_2143.txt
        ├── [ 12K]  Cluster_2144.ps
        ├── [ 34K]  Cluster_2144.txt
        ├── [ 12K]  Cluster_3043.ps
        ├── [ 10K]  Cluster_3043.txt
        ├── [ 12K]  Cluster_3093.ps
        ├── [5.7K]  Cluster_3093.txt
        ├── [ 12K]  Cluster_3095.ps
        ├── [8.4K]  Cluster_3095.txt
        ├── [ 11K]  Cluster_3115.ps
        ├── [ 14K]  Cluster_3115.txt
        ├── [ 11K]  Cluster_3258.ps
        ├── [6.5K]  Cluster_3258.txt
        ├── [ 11K]  Cluster_3700.ps
        ├── [ 33K]  Cluster_3700.txt
        ├── [ 12K]  Cluster_396.ps
        ├── [ 47K]  Cluster_396.txt
        ├── [ 11K]  Cluster_4008.ps
        ├── [ 15K]  Cluster_4008.txt
        ├── [ 11K]  Cluster_41.ps
        ├── [3.8K]  Cluster_41.txt
        ├── [ 11K]  Cluster_429.ps
        ├── [ 15K]  Cluster_429.txt
        ├── [ 12K]  Cluster_4423.ps
        ├── [5.4K]  Cluster_4423.txt
        ├── [ 12K]  Cluster_4524.ps
        ├── [ 15K]  Cluster_4524.txt
        ├── [ 12K]  Cluster_4871.ps
        ├── [ 40K]  Cluster_4871.txt
        ├── [ 12K]  Cluster_4873.ps
        ├── [ 57K]  Cluster_4873.txt
        ├── [ 12K]  Cluster_4874.ps
        ├── [ 27K]  Cluster_4874.txt
        ├── [ 12K]  Cluster_4875.ps
        ├── [ 34K]  Cluster_4875.txt
        ├── [ 12K]  Cluster_4876.ps
        ├── [9.0K]  Cluster_4876.txt
        ├── [ 12K]  Cluster_5010.ps
        ├── [ 71K]  Cluster_5010.txt
        ├── [ 12K]  Cluster_5210.ps
        ├── [7.1K]  Cluster_5210.txt
        ├── [ 12K]  Cluster_5279.ps
        ├── [ 33K]  Cluster_5279.txt
        ├── [ 12K]  Cluster_6115.ps
        ├── [ 19K]  Cluster_6115.txt
        ├── [ 12K]  Cluster_6255.ps
        ├── [ 18K]  Cluster_6255.txt
        ├── [ 12K]  Cluster_6981.ps
        ├── [ 23K]  Cluster_6981.txt
        ├── [ 12K]  Cluster_833.ps
        └── [ 33K]  Cluster_833.txt

2 directories, 103 files

---

Citations

Axtell, Michael J. 2013. “ShortStack: Comprehensive Annotation and Quantification of Small RNA Genes.” RNA 19 (6): 740–51. https://doi.org/10.1261/rna.035279.112.

Johnson, Nathan R, Jonathan M Yeoh, Ceyda Coruh, and Michael J Axtell. 2016. “Improved Placement of Multi-Mapping Small RNAs.” G3 Genes|Genomes|Genetics 6 (7): 2103–11. https://doi.org/10.1534/g3.116.030452.

Shahid, Saima, and Michael J. Axtell. 2014. “Identification and Annotation of Small RNA Genes Using ShortStack.” Methods 67 (1): 20–27. https://doi.org/10.1016/j.ymeth.2013.10.004.