All our alignment files are in BAM or CRAM format. BAM is a standard alignment format which was defined by the 1000 Genomes consortium and has since seen wide community adoption, whereas CRAM is a compressed version of this. This compression is driven by the reference the sequence data is aligned to.
The CRAM file format was designed by the EBI to reduce the disk footprint of alignment data in these days of ever-increasing data volumes.
The CRAM files the 1000 Genomes project distributes are lossy cram files which reduce the base quality scores using the Illumina 8-bin compression scheme as described in the lossy compression section on the cram usage page
There is a github page where the format of CRAM file is discussed and help can be found.
CRAM files can be read using many Picard tools and work is being done to ensure samtools can also read the file format natively.
The bam file names look like:
NA00000.location.platform.population.analysis_group.YYYYMMDD.bam
The bai index and bas statistics files are also named in the same way.
The name includes the individual sample ID, where the sequence is mapped to, if the file has only contains mapping to a particular chromosome that is what the name contains otherwise, mapped means the whole genome mapping and unmapped means the reads which failed to map to the reference (pairs where one mate mapped and the other didn’t stay in the mapped file), the sequencing platform, the ethnicity of the sample using our three letter population code, the sequencing strategy. The date matches the date of the sequence used to build the bams and can also be found in the sequence.index filename.
The unmapped bams contain all the reads for the given individual which could not be placed on the reference genome. It contains no mapping information
Please note that any paired end sequence where one end successfully maps but the other does not both reads are found in the mapped bam
Bas files are statistics we generate for our alignment files which we distribute alongside our alignment files.
These are readgroup level statistics in a tab delimited manner and are described in this README
Each mapped and unmapped bam file has an associated bas file and we provide them collected together into a single file in the alignment_indices directory, dated to match the alignment release.
We describe our sequence meta data in sequence index files. The index for data from the 1000 Genomes Project can be found in the 1000 Genomes data collection directory. Additional indices are present for data in other data collections. Our old index files which describe the data used in the main project can be found in the historical_data directory
Sequence index files are tab delimited files and frequently contain these columns:
| Column | Title | Description |
|---|---|---|
| 1 | FASTQ_FILE | path to fastq file on ftp site or ENA ftp site |
| 2 | MD5 | md5sum of file |
| 3 | RUN_ID | SRA/ERA run accession |
| 4 | STUDY_ID | SRA/ERA study accession |
| 5 | STUDY_NAME | Name of study |
| 6 | CENTER_NAME | Submission centre name |
| 7 | SUBMISSION_ID | SRA/ERA submission accession |
| 8 | SUBMISSION_DATE | Date sequence submitted, YYYY-MM-DD |
| 9 | SAMPLE_ID | SRA/ERA sample accession |
| 10 | SAMPLE_NAME | Sample name |
| 11 | POPULATION | Sample population, this is a 3 letter code as defined in README_populations.md |
| 12 | EXPERIMENT_ID | Experiment accession |
| 13 | INSTRUMENT_PLATFORM | Type of sequencing machine |
| 14 | INSTRUMENT_MODEL | Model of sequencing machine |
| 15 | LIBRARY_NAME | Library name |
| 16 | RUN_NAME | Name of machine run |
| 17 | RUN_BLOCK_NAME | Name of machine run sector (This is no longer recorded so this column is entirely null, it was left in so as not to disrupt existing sequence index parsers) |
| 18 | INSERT_SIZE | Submitter specified insert size |
| 19 | LIBRARY_LAYOUT | Library layout, this can be either PAIRED or SINGLE |
| 20 | PAIRED_FASTQ | Name of mate pair file if exists (Runs with failed mates will have a library layout of PAIRED but no paired fastq file) |
| 21 | WITHDRAWN | 0/1 to indicate if the file has been withdrawn, only present if a file has been withdrawn |
| 22 | WITHDRAWN_DATE | This is generally the date the file is generated on |
| 23 | COMMENT | comment about reason for withdrawal |
| 24 | READ_COUNT | read count for the file |
| 25 | BASE_COUNT | basepair count for the file |
| 26 | ANALYSIS_GROUP | the analysis group of the sequence, this reflects sequencing strategy. For 1000 Genomes Project data, this includes low coverage, high coverage, exon targeted and exome to reflect the two non low coverage pilot sequencing strategies and the two main project sequencing strategies used by the 1000 Genomes Project. |
The sequence.index file contains a list of all the sequence data produced by the project, pointers to the file locations on the ftp site and also all the meta data associated with each sequencing run.
For the phase 3 analysis the consortium has decided to only use Illumina platform sequence data with reads of 70 base pairs or longer. The analysis.sequence.index file contains only the active runs which match this criterion. There are withdrawn runs in this index. These runs are withdrawn because either: * They have insufficient raw sequence to meet our 3x non duplicated aligned coverage criteria for low coverage alignments. * After the alignment has been run they have failed our post alignment quality controls for short indels. * Contamination. * They do not meet our coverage criteria.
Since the alignment release based on 20120522, we have only released alignments based on the analysis.sequence.index
The easiest way to find the alignment files you’re looking for is with the Data Portal. You can search for individuals, populations and data collections, and filter the files by data type and technologies. This will give you locations of the files, which you can use to download directly, or to export a list to use with a download manager.
You can find an index of our alignments in our alignment.index file. There are dated versions of these files and statistics surrounding each alignment release in the alignment_indicies directory. Please note with few exceptions we only keep the most recent QC passed alignment for each sample on the ftp site.
The easiest way to find the sequence files you’re looking for is with the Data Portal. You can search for individuals, populations and data collections, and filter the files by data type and technologies. This will give you locations of the files, which you can use to download directly, or to export a list to use with a download manager.
You can get a subsection of the VCF or BAM files using the Ensembl Data Slicer tool. This tool gives you a web interface requesting the URL of any VCF file and the genomic location you wish to get a sub-slice for. This tool also works for BAM files. This tool also allows you to filter the file for particular individuals or populations if you also provide a panel file.
You can also subset VCFs using tabix on the command line, e.g.
tabix -h ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20100804/ALL.2of4intersection.20100804.genotypes.vcf.gz 2:39967768-39967768
Specifications for the VCF format, and a C++ and Perl tool set for VCF files can be found at vcftools on sourceforge
Please note that all our VCF files using straight intergers and X/Y for their chromosome names in the Ensembl style rather than using chr1 in the UCSC style. If you request a subsection of a vcf file using a chromosome name in the style chrN as shown below it will not work.
tabix -h ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/release/20100804/ALL.2of4intersection.20100804.genotypes.vcf.gz chr2:39967768-39967768
You can subset alignment files with samtools on the command line, e.g.
samtools view -h ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase1/data/HG00154/alignment/HG00154.mapped.ILLUMINA.bwa.GBR.low_coverage.20101123.bam 17:7512445-7513455
Samtools supports streaming files and piping commands together both using local and remote files. You can get more help with samtools from the samtools help mailing list
The analysis tools used for samples in a given data collection can vary depending on the technologies (i.e. PacBio, exome, etc.) that were used to generate data for any given sample. The technologies may not be the same across all samples in a collection.
Generally, however, for any given analysis the data types will be the same or very similar and will have been analysed in a consistent manner.
For data collections where the analysis has been published, the publication will give details of what methods were used. Checking this may involve looking at the supplementary material. We list publcations on our data collections page. In addition, for analyses which may not have been published, you will find README files on our FTP site, in the directories for each data collection, that provide further information.
The chr 11 and chr 20 alignment files are put in place to give the 1000 Genomes analysis group a small section of the genome to run test analyses on before committing to a particular strategy to run across the whole genome. Everything in the chr 11 and chr 20 files is also represented in the mapped bam file. To get a complete view of what data we aligned you only need to download the mapped and unmapped bams, the chr 11 and chr 20 bams are there as a convenience to the analysis group.