---
author: Sam White
toc-title: Contents
toc-depth: 5
toc-location: left
date: 2016-02-05 04:48:50+00:00
layout: post
slug: data-management-high-throughput-sequencing-data
title: Data Management - High-throughput Sequencing Data
categories:
  - 2016
  - Miscellaneous
tags:
  - fusion table
  - nightingales
  - owl
---

We've had a recent influx of sequencing data, which is great, but it created a bit of a backlog documenting what we've received.

I updated our Google Sheet (Nightingales) with the data from [geoduck genome sequencing data from BGI](https://robertslab.github.io/sams-notebook/posts/2016/2016-01-27-data-received-panopea-generosa-genome-sequencing-files-from-bgi.html), [Olympia oyster genome sequencing data from BGI](https://robertslab.github.io/sams-notebook/2016/01/27/data-received-ostrea-lurida-genome-sequencing-files-from-bgi/), and [MBD bisulfite sequencing data from ZymoResearch](https://robertslab.github.io/sams-notebook/2016/02/03/data-received-ostrea-lurida-mbd-enriched-bs-seq/).

I also fixed the :FileLocation" column by replacing the "HYPERLINK" function with "CONCATENATE".

Google Sheet: [Nightingales](https://docs.google.com/spreadsheets/d/1_XqIOPVHSBVGscnjzDSWUeRL7HUHXfaHxVzec-I-8Xk/pubhtml)





After updating the Nightingales Google Sheet, I updated the corresponding Google Fusion Table (also called Nightingales).

To update the Fusion Table, you have to do the following:




    
  * delete all rows in the Nightingales Google Fusion Table (Edit > Delete all rows)

    
  * Import data from the Nightingales Google Spreadsheet (File > Import more rows...)



Fusion Table: [Nightingales](https://www.google.com/fusiontables/embedviz?viz=GVIZ&t=TABLE&q=select+col0%2C+col1%2C+col2%2C+col3%2C+col4%2C+col5%2C+col6%2C+col7%2C+col8%2C+col9%2C+col10%2C+col11%2C+col12+from+13IxnqIZ_2Xpz_HE-3YcnU_egASYz9ZlA0PYIDGLN+order+by+col1+desc&containerId=googft-gviz-canvas)



At initial glance, the Fusion Table appears the same as the Google Sheet. However, if you follow the link to the full Fusion Table, it offers some unique ways to visually explore the data contained in the Fusion Table.



After that I decided to deal with the fact that many of the directories on Owl (https://owl.fish.washington.edu/nightingales/) lack readme files and subsequent information about the sequencing files in those folders.

So, I took an inordinate amount of time to write a script that would automate as much of the process as I could think of.


The goal of the script is to perform the following:





  * Identify folders that do not have readme files.



  * Identify folders that do not have checksum files.



  * Create readme files in those directories lacking readme files



  * Append the directory path to each new readme file



  * Append sequencing file names and corresponding read counts to the new readme files






Will run the script. Hope it works...
