---
author: Sam White
toc-title: Contents
toc-depth: 5
toc-location: left
layout: post
title: RepeatMasker - C.gigas Rosling NCBI Genome GCA_902806645.1 on Mox
date: '2021-05-04 11:33'
tags: 
  - GCA_902806645.1
  - mox
  - repeatmasker
  - Crassostrea gigas
  - Pacific oyster
categories: 
  - 2021
  - Miscellaneous
---
Decided to [tackle this GitHub Issue about creating a transposable elements IGV track with the new Roslin _C.gigas_ genome](https://github.com/RobertsLab/resources/issues/1141), since it had been sitting for a while and I have code sitting around that's ready to roll for this type of thing.

Downloaded the NCBI [_Crassostrea gigas_ (Pacific oyster)](http://en.wikipedia.org/wiki/Pacific_oyster) genome assembly [GCA_902806645.1](https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/902/806/645/GCA_902806645.1_cgigas_uk_roslin_v1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.gz) and verified the MD5 checksum (not shown).

NOTE: The above listed NCBI assembly is from the "GenBank" assembly. There is another version, [GCF_902806645.1](https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/902/806/645/GCF_902806645.1_cgigas_uk_roslin_v1/GCF_902806645.1_cgigas_uk_roslin_v1_genomic.fna.gz), from NCBI RefSeq. As far as I can tell, the only difference between the two is the sequence IDs; numbers of sequences and their lengths are the same.

Analysis was performed using [RepeatMasker](https://www.repeatmasker.org/) with two different species settings for comparison, if someone is interested.

- "all"

- "[_Crassostrea gigas_ (Pacific oyster)](http://en.wikipedia.org/wiki/Pacific_oyster)"

The analysis was run on Mox.

SBATCH script (GitHub):

- [20210504_cgig_repeatmasker_roslin-GCA_902806645.1.sh](https://github.com/RobertsLab/sams-notebook/blob/master/sbatch_scripts/20210504_cgig_repeatmasker_roslin-GCA_902806645.1.sh)

```shell
#!/bin/bash
## Job Name
#SBATCH --job-name=20210504_cgig_repeatmasker_roslin-GCA_902806645.1
## Allocation Definition
#SBATCH --account=coenv
#SBATCH --partition=coenv
## Resources
## Nodes
#SBATCH --nodes=1
## Walltime (days-hours:minutes:seconds format)
#SBATCH --time=6-00:00:00
## Memory per node
#SBATCH --mem=120G
##turn on e-mail notification
#SBATCH --mail-type=ALL
#SBATCH --mail-user=samwhite@uw.edu
## Specify the working directory for this job
#SBATCH --chdir=/gscratch/scrubbed/samwhite/outputs/20210504_cgig_repeatmasker_roslin-GCA_902806645.1

# Script to run RepeatMasker 4.1.0 on "Roslin" C.gigas NCBI genome assembly GCA_902806645.1


###################################################################################
# These variables need to be set by user

# Set working directory
wd=$(pwd)

# Set number of CPUs to use
threads=40

# Input/output files
source_genome_fasta=/gscratch/srlab/sam/data/C_gigas/genomes/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna
genome_fasta=GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna

# Programs
## Minimap2
repeat_masker=/gscratch/srlab/programs/RepeatMasker-4.1.0/RepeatMasker

# Species array (used for RepeatMasker species setting)
species=("all" "crassostrea gigas")

# Programs associative array
declare -A programs_array
programs_array=(
[repeat_masker]=${repeat_masker} \
)



###################################################################################

# Exit script if any command fails
set -e


# Generate checksum for "new" FastA
md5sum ${source_genome_fasta} > genome_fasta.md5

for species in "${species[@]}"
do

  # Check species name and create appropriate directory naem
  if [ "${species}" = "crassostrea gigas" ]; then

    mkdir "repeatmasker-species_C.gigas_roslin-GCA_902806645.1" && cd $_
    rsync -av ${source_genome_fasta} .

    else
    mkdir "repeatmasker-species_all_roslin-GCA_902806645.1" && cd $_
    rsync -av ${source_genome_fasta} .

  fi

  # Run RepeatMasker
  # Uses all species
  # Generates GFF output
  # 'excln' calculates repeat densities excluding runs of X/N >20bp
  ${programs_array[repeat_masker]} \
  ${genome_fasta} \
  -species "${species}" \
  -parallel ${threads} \
  -gff \
  -excln
  
  # Remove the genome FastA file
  rm ${genome_fasta}

  cd ${wd}

done

###################################################################################

# Capture program options
if [[ "${#programs_array[@]}" -gt 0 ]]; then
  echo "Logging program options..."
  for program in "${!programs_array[@]}"
  do
    {
    echo "Program options for ${program}: "
    echo ""
    # Handle samtools help menus
    if [[ "${program}" == "samtools_index" ]] \
    || [[ "${program}" == "samtools_sort" ]] \
    || [[ "${program}" == "samtools_view" ]]
    then
      ${programs_array[$program]}

    # Handle DIAMOND BLAST menu
    elif [[ "${program}" == "diamond" ]]; then
      ${programs_array[$program]} help

    # Handle NCBI BLASTx menu
    elif [[ "${program}" == "blastx" ]]; then
      ${programs_array[$program]} -help
    fi
    ${programs_array[$program]} -h
    echo ""
    echo ""
    echo "----------------------------------------------"
    echo ""
    echo ""
  } &>> program_options.log || true

    # If MultiQC is in programs_array, copy the config file to this directory.
    if [[ "${program}" == "multiqc" ]]; then
      cp --preserve ~/.multiqc_config.yaml multiqc_config.yaml
    fi
  done
fi

# Document programs in PATH (primarily for program version ID)
{
  date
  echo ""
  echo "System PATH for $SLURM_JOB_ID"
  echo ""
  printf "%0.s-" {1..10}
  echo "${PATH}" | tr : \\n
} >> system_path.log

echo "Finished logging system PATH"
```

---

# RESULTS

Was much faster than I expected, as previous runs on Emu/Roadrunner have taken _days_. This was just 1.5hrs:

![RepeatMasker runtime for GCA_902806645.1 on Mox](https://github.com/RobertsLab/sams-notebook/blob/master/images/screencaps/20210504_cgig_repeatmasker_roslin-GCA_902806645.1_runtime.png?raw=true)

Important to note is the stark differences between results with the species set to "all" vs. "[_Crassostrea gigas_ (Pacific oyster)](http://en.wikipedia.org/wiki/Pacific_oyster)". For example, the number of "Retroelements" identified with the "all" setting was 14,694 vs 23 with the "[_Crassostrea gigas_ (Pacific oyster)](http://en.wikipedia.org/wiki/Pacific_oyster)" setting!

Output folder:

- [20210504_cgig_repeatmasker_roslin-GCA_902806645.1/](https://gannet.fish.washington.edu/Atumefaciens/20210504_cgig_repeatmasker_roslin-GCA_902806645.1/)

##### Species Setting: [_Crassostrea gigas_ (Pacific oyster)](http://en.wikipedia.org/wiki/Pacific_oyster)

  - RepeatMasker GFF:

    - [20210504_cgig_repeatmasker_roslin-GCA_902806645.1/repeatmasker-species_C.gigas_roslin-GCA_902806645.1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.out.gff](https://gannet.fish.washington.edu/Atumefaciens/20210504_cgig_repeatmasker_roslin-GCA_902806645.1/repeatmasker-species_C.gigas_roslin-GCA_902806645.1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.out.gff)

  - Summary Table (text):

    - [20210504_cgig_repeatmasker_roslin-GCA_902806645.1/repeatmasker-species_C.gigas_roslin-GCA_902806645.1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.tbl](https://gannet.fish.washington.edu/Atumefaciens/20210504_cgig_repeatmasker_roslin-GCA_902806645.1/repeatmasker-species_C.gigas_roslin-GCA_902806645.1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.tbl)

    ```
    ==================================================
    file name: GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna
    sequences:           236
    total length:  647887097 bp  (647868030 bp excl N/X-runs)
    GC level:         33.50 %
    bases masked:   12745378 bp ( 1.97 %)
    ==================================================
                  number of      length   percentage
                  elements*    occupied  of sequence
    --------------------------------------------------
    Retroelements           23         1649 bp    0.00 %
      SINEs:               23         1649 bp    0.00 %
      Penelope              0            0 bp    0.00 %
      LINEs:                0            0 bp    0.00 %
        CRE/SLACS            0            0 bp    0.00 %
        L2/CR1/Rex          0            0 bp    0.00 %
        R1/LOA/Jockey       0            0 bp    0.00 %
        R2/R4/NeSL          0            0 bp    0.00 %
        RTE/Bov-B           0            0 bp    0.00 %
        L1/CIN4             0            0 bp    0.00 %
      LTR elements:         0            0 bp    0.00 %
        BEL/Pao             0            0 bp    0.00 %
        Ty1/Copia           0            0 bp    0.00 %
        Gypsy/DIRS1         0            0 bp    0.00 %
          Retroviral        0            0 bp    0.00 %

    DNA transposons          0            0 bp    0.00 %
      hobo-Activator        0            0 bp    0.00 %
      Tc1-IS630-Pogo        0            0 bp    0.00 %
      En-Spm                0            0 bp    0.00 %
      MuDR-IS905            0            0 bp    0.00 %
      PiggyBac              0            0 bp    0.00 %
      Tourist/Harbinger     0            0 bp    0.00 %
      Other (Mirage,        0            0 bp    0.00 %
        P-element, Transib)

    Rolling-circles          0            0 bp    0.00 %

    Unclassified:            2           84 bp    0.00 %

    Total interspersed repeats:        1733 bp    0.00 %


    Small RNA:              68        12259 bp    0.00 %

    Satellites:              0            0 bp    0.00 %
    Simple repeats:     231620     11214786 bp    1.73 %
    Low complexity:      32034      1516600 bp    0.23 %
    ==================================================

    * most repeats fragmented by insertions or deletions
      have been counted as one element
      Runs of >=20 X/Ns in query were excluded in % calcs


    The query species was assumed to be crassostrea gigas
    RepeatMasker Combined Database: Dfam_3.1
                                            
    run with rmblastn version 2.10.0+
    ```
    
##### Species Setting: All

- RepeatMasker GFF:

  - [20210504_cgig_repeatmasker_roslin-GCA_902806645.1/repeatmasker-species_all_roslin-GCA_902806645.1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.out.gff](https://gannet.fish.washington.edu/Atumefaciens/20210504_cgig_repeatmasker_roslin-GCA_902806645.1/repeatmasker-species_all_roslin-GCA_902806645.1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.out.gff)

- Summary Table (text):

  - [20210504_cgig_repeatmasker_roslin-GCA_902806645.1/repeatmasker-species_all_roslin-GCA_902806645.1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.tbl](https://gannet.fish.washington.edu/Atumefaciens/20210504_cgig_repeatmasker_roslin-GCA_902806645.1/repeatmasker-species_all_roslin-GCA_902806645.1/GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna.tbl)

```
==================================================
file name: GCA_902806645.1_cgigas_uk_roslin_v1_genomic.fna
sequences:           236
total length:  647887097 bp  (647868030 bp excl N/X-runs)
GC level:         33.50 %
bases masked:   19522553 bp ( 3.01 %)
==================================================
               number of      length   percentage
               elements*    occupied  of sequence
--------------------------------------------------
Retroelements        14694      4937654 bp    0.76 %
   SINEs:              585        34638 bp    0.01 %
   Penelope            262        28681 bp    0.00 %
   LINEs:             7164      1209298 bp    0.19 %
    CRE/SLACS            0            0 bp    0.00 %
     L2/CR1/Rex       1266       159741 bp    0.02 %
     R1/LOA/Jockey     157        13339 bp    0.00 %
     R2/R4/NeSL        129        65251 bp    0.01 %
     RTE/Bov-B         980       263384 bp    0.04 %
     L1/CIN4          1582       175410 bp    0.03 %
   LTR elements:      6945      3693718 bp    0.57 %
     BEL/Pao           863      1015388 bp    0.16 %
     Ty1/Copia          38        10630 bp    0.00 %
     Gypsy/DIRS1      3578      2423374 bp    0.37 %
       Retroviral     1416        90219 bp    0.01 %

DNA transposons      13542      2099703 bp    0.32 %
   hobo-Activator     3213       229649 bp    0.04 %
   Tc1-IS630-Pogo      560        47379 bp    0.01 %
   En-Spm                0            0 bp    0.00 %
   MuDR-IS905            0            0 bp    0.00 %
   PiggyBac             30         2479 bp    0.00 %
   Tourist/Harbinger   711       100695 bp    0.02 %
   Other (Mirage,       37         1842 bp    0.00 %
    P-element, Transib)

Rolling-circles       2594       517630 bp    0.08 %

Unclassified:          508        42673 bp    0.01 %

Total interspersed repeats:     7080030 bp    1.09 %


Small RNA:            2485       177888 bp    0.03 %

Satellites:           1212       172569 bp    0.03 %
Simple repeats:     223699     10101220 bp    1.56 %
Low complexity:      31668      1475381 bp    0.23 %
==================================================

* most repeats fragmented by insertions or deletions
  have been counted as one element
  Runs of >=20 X/Ns in query were excluded in % calcs


The query species was assumed to be root          
RepeatMasker Combined Database: Dfam_3.1
                                         
run with rmblastn version 2.10.0+
```