HowTo: Access SRA Data

 HowTo: Access SRA Data

https://github.com/ncbi/sra-tools/wiki/HowTo:-Access-SRA-Data 

 

use the tool prefetch included in the SRA Toolkit.

 

xample of prefetch usage:

$ prefetch SRR1482462
Maximum file size download limit is 20,971,520KB

2015-02-19T13:20:06 prefetch.2.4.4: 1) Downloading 'SRR1482462'...
2015-02-19T13:20:06 prefetch.2.4.4:  Downloading via fasp...
2015-02-19T13:20:32 prefetch.2.4.4:  fasp download succeed
2015-02-19T13:20:32 prefetch.2.4.4: 1) 'SRR1482462' was downloaded successfully
2015-02-19T13:20:35 prefetch.2.4.4: 'SRR1482462' has 22 dependencies
2015-02-19T13:20:36 prefetch.2.4.4: 2) Downloading 'ncbi-acc:NC_000067.5?vdb-ctx=refseq'...
2015-02-19T13:20:36 prefetch.2.4.4:  Downloading via fasp...
2015-02-19T13:20:41 prefetch.2.4.4:  fasp download succeed
2015-02-19T13:20:41 prefetch.2.4.4: 2) 'ncbi-acc:NC_000067.5?vdb-ctx=refseq' was downloaded successfully
2015-02-19T13:20:41 prefetch.2.4.4: 3) Downloading 'ncbi-acc:NC_000068.6?vdb-ctx=refseq'...
2015-02-19T13:20:41 prefetch.2.4.4:  Downloading via fasp...
2015-02-19T13:20:46 prefetch.2.4.4:  fasp download succeed
2015-02-19T13:20:46 prefetch.2.4.4: 3) 'ncbi-acc:NC_000068.6?vdb-ctx=refseq' was downloaded successfully
2015-02-19T13:20:46 prefetch.2.4.4: 4) Downloading 'ncbi-acc:NC_000069.5?vdb-ctx=refseq'...
2015-02-19T13:20:46 prefetch.2.4.4:  Downloading via fasp...
2015-02-19T13:20:51 prefetch.2.4.4:  fasp download succeed
2015-02-19T13:20:51 prefetch.2.4.4: 4) 'ncbi-acc:NC_000069.5?vdb-ctx=refseq' was downloaded successfully
...

As can be seen from the output above, prefetch performs several steps:

1. check the size of the file being downloaded
   If the file is very large, prefetch must be given a higher download limit, e.g.:
   $ prefetch --max-size 100000000 SRR1482462

2. download the requested file
   The file is downloaded using Aspera if available on your system, or HTTPS otherwise.

3. put the file into its proper place
   The file is downloaded into your designated cache area. This permits VDB name resolution to work as designed.

4. recursively download missing external reference sequences
   Most SRA files require additional sequence files in order to reconstruct original reads. prefetch ensures that you not only download the main file but all of its dependencies.

5. access dbGaP encrypted data
   prefetch will make use of download and decryption keys that have been added to SRA Toolkit configuration to obtain authorization for the download in addition to performing all of the steps above. (N.B. In order to access dbGaP data, you will need to change directory or "cd" to the dbGaP project's workspace.)

prefetch will also operate on existing, previously downloaded files to recursively download any missing external reference sequences.

 

Download FASTQ files from European Nucleotide Archive (ENA)

https://github.com/wwood/ena-fast-download

Requirements

aspera client : https://downloads.asperasoft.com/en/downloads/8?list
curl
Python 3

# set path for aspera. check your aspera directory path
PATH=$PATH:/home/lee/.aspera/connect/bin
export PATH

usage: ena-fast-download.py [-h] [--output_directory OUTPUT_DIRECTORY]
                            [--ssh_key SSH_KEY ( for OSX) ]
                            run_identifier

ena-fast-download.py --output_directory /output/directory ERR1739691

parallel-fastq-dump

https://github.com/rvalieris/parallel-fastq-dump

parallel fastq-dump wrapper

How to access data in SRA

Good series of posts on how to access data in SRA

https://edwards.sdsu.edu/research/lab-blog/sra/

SRA Toolkit

update : 20220916

update your SRA toolkit

https://github.com/ncbi/sra-tools/wiki

 

 

https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=toolkit_doc

SRA Toolkit Installation and Configuration Guide

Protected Data Usage Guide

Frequently Used Tools:

fastq-dump: Convert SRA data into fastq format

prefetch: Allows command-line downloading of SRA, dbGaP, and ADSP data

sam-dump: Convert SRA data to sam format

sra-pileup: Generate pileup statistics on aligned SRA data

vdb-config: Display and modify VDB configuration information

vdb-decrypt: Decrypt non-SRA dbGaP data ("phenotype data")

Additional Tools:

abi-dump: Convert SRA data into ABI format (csfasta / qual)

illumina-dump: Convert SRA data into Illumina native formats (qseq, etc.)

sff-dump: Convert SRA data to sff format

sra-stat: Generate statistics about SRA data (quality distribution, etc.)

vdb-dump: Output the native VDB format of SRA data.

vdb-encrypt: Encrypt non-SRA dbGaP data ("phenotype data")

vdb-validate: Validate the integrity of downloaded SRA data

SRAtoolkit - fastq-dump

https://edwards.sdsu.edu/research/fastq-dump/

good review for how to use fastq-dump option


https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=toolkit_doc&f=fastq-dump

fastq-dump: Convert SRA data into fastq format

Usage:

fastq-dump [options] <path/file> [<path/file> ...]

fastq-dump [options] <accession>

Frequently Used Options:

General:
-h	|	--help	Displays ALL options, general usage, and version information.
-V	|	--version	Display the version of the program.
Data formatting:
		--split-files	Dump each read into separate file. Files will receive suffix corresponding to read number.
		--split-spot	Split spots into individual reads.
		--fasta <[line width]>	FASTA only, no qualities. Optional line wrap width (set to zero for no wrapping).
-I	|	--readids	Append read id after spot id as 'accession.spot.readid' on defline.
-F	|	--origfmt	Defline contains only original sequence name.
-C	|	--dumpcs <[cskey]>	Formats sequence using color space (default for SOLiD). "cskey" may be specified for translation.
-B	|	--dumpbase	Formats sequence using base space (default for other than SOLiD).
-Q	|	--offset <integer>	Offset to use for ASCII quality scores. Default is 33 ("!").
Filtering:
-N	|	--minSpotId <rowid>	Minimum spot id to be dumped. Use with "X" to dump a range.
-X	|	--maxSpotId <rowid>	Maximum spot id to be dumped. Use with "N" to dump a range.
-M	|	--minReadLen <len>	Filter by sequence length >= <len>
		--skip-technical	Dump only biological reads.
		--aligned	Dump only aligned sequences. Aligned datasets only; see sra-stat.
		--unaligned	Dump only unaligned sequences. Will dump all for unaligned datasets.
Workflow and piping:
-O	|	--outdir <path>	Output directory, default is current working directory ('.').
-Z	|	--stdout	Output to stdout, all split data become joined into single stream.
		--gzip	Compress output using gzip.
		--bzip2	Compress output using bzip2.

Use examples:

fastq-dump -X 5 -Z SRR390728

Prints the first five spots (-X 5) to standard out (-Z). This is a useful starting point for verifying other formatting options before dumping a whole file.

fastq-dump -I --split-files SRR390728

Produces two fastq files (--split-files) containing ".1" and ".2" read suffices (-I) for paired-end data.

fastq-dump --split-files --fasta 60 SRR390728

Produces two (--split-files) fasta files (--fasta) with 60 bases per line ("60" included after --fasta).

fastq-dump --split-files --aligned -Q 64 SRR390728

Produces two fastq files (--split-files) that contain only aligned reads (--aligned; Note: only for files submitted as aligned data), with a quality offset of 64 (-Q 64) Please see the documentation on vdb-dump if you wish to produce fasta/qual data.

Possible errors and their solution:

fastq-dump.2.x err: item not found while constructing within virtual database module - the path '<path/SRR*.sra>' cannot be opened as database or table

This error indicates that the .sra file cannot be found. Confirm that the path to the file is correct.

fastq-dump.2.x err: name not found while resolving tree within virtual file system module - failed SRR*.sra

The data are likely reference compressed and the toolkit is unable to acquire the reference sequence(s) needed to extract the .sra file. Please confirm that you have tested and validated the configuration of the toolkit. If you have elected to prevent the toolkit from contacting NCBI, you will need to manually acquire the reference(s) here

How To Download All Sra Samples At Once

using SQLlite3
https://edwards.sdsu.edu/research/getting-data-from-the-sra/



using linux wget 
details in http://seqanswers.com/forums/archive/index.php/t-30625.html

wget -r -nd -nH ftp://file_address

for example,
wget -r -nd -nH ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByStudy/sra/SRP/SRP103/SRP103124

Address is changed : edit 20210720
https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR11780909/SRR11780909

To download specific SRA files in the list

Store the SRR096001-SRR096999 #s that would want to download in a file:
for example:
cat > SRR_2_download
SRR096023
SRR096072
SRR096074

for i in $(cat SRR_2_download);do wget -r -nd -nH ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/litesra/SRR/SRR096/$i/*; done

for i in $(cat SRR_2_download);do wget https://sra-pub-run-odp.s3.amazonaws.com/sra/$i/$i; done

 
This will download the specific SRA files that are listed in the file SRR_2_download

using R
details in www.biostars.org/p/93494/

Run R
source('http://bioconductor.org/biocLite.R')
biocLite('SRAdb')
library(SRAdb)
biocLite('DBI')
library(DBI)

srafile = getSRAdbFile() # Download & Unzip Last Version Of SRAmetadb.Sqlite.Gz From Server to working directory

# Once you download SRAmetadb.Sqlite.Gz, set SRAmetadb.sqlite file to variable srafile
# SRAmetadb.Sqlite.Gz is big. Re-download if you need updated version

srafile <- 'SRAmetadb.sqlite' 

con = dbConnect(RSQLite::SQLite(), srafile)
listSRAfile('SRP026197',con)
getSRAfile('SRP026197',con,fileType='sra')




## dump SRA file to fastq.gz  (required SRAtoolkit)
Run in linux
fastq-dump -O /output/dir --gzip ./SRR2047462.sra #sra file dump to SRR2047462.fastq.gz

library(GEOquery)
gse <- getGEO('GSE48138') # retrieves a GEO list set for your SRA id.
## see what is in there:
show(gse)
# There are 2 sets of samples for that ID
##  what you want is table a with SRR to download and some sample information:
## lets see what the first set contains:
df <- as.data.frame(gse[[1]])
head(df)