Please consider adding support for multi-thread support

[dbrami]

Being able to scatter/gather the work over multiple CPU cores would really help speed-up your script.

[susannasiebert]

Thank you for your interest in VAtools. All of the tools in this toolkit are just IO for the most part so shouldn't take very long. Is there a specific tool that has been running slow for you?

[dbrami]

Hi,
Indeed, "vcf-info-annotator" has been working much slower than hoped. My decompressed VCF file is about 75 GB. The program was writing about 1 MB every 5 min on a beefy AWS machine.
I know it's not a lot of info. Let me know what you would like to see in order to figure out how use your script more efficiently or if some code fixes can speed things up.
Thanks

[susannasiebert]

ah ok. I've definitely never run it on that large of a file. I'll have a look to see how things can be improved.

[lukaas33]

I am encountering the same issue. A VCF of 11GB has been running for over 12 hours now.

[susannasiebert]

How big is your TSV? That file is being read into memory so you'll want to make sure that you have at least that amount of memory available. You're process is probably stuck swapping memory and not actually doing any/much work.

[lukaas33]

The TSV is only 3MB, the gtf around 300MB.

Here is my top output:

It seems cpu is used maximally as well as the memory (64GB total).

The exact command that I am using is vcf-expression-annotator -s sample -o /shared_dir/temp.vcf /shared_dir/neoantigen.vep.vcf /shared_dir/neoantigen.transcript.abundance.gtf stringtie transcript && vcf-expression-annotator -s sample -o /shared_dir/neoantigen.stringtie.vcf /shared_dir/temp.vcf /shared_dir/neoantigen.gene.abundance.tsv stringtie gene

I am using the Docker container.

[susannasiebert]

That's strange. I'm not sure why you are seeing multiple processes either. Do you see the same behavior when you run the two steps as separate commands?

The gtf parsing library we are using uses pandas underneath the hood, which, unfortunately, can use up a lot of memory (more than expected) because of the way it stores some data. Would you mind sending me your gtf file so I can play around with it?

[lukaas33]

When using the && separator these commands are executed seperately.
I also tried to run only the first command but this also took too long.

I will email you these files.

[susannasiebert]

Ok, this is definitely not an issue with the GTF file. It's able to read it in just fine but you have over 4.5 million VCF entries so processing just takes a while. I'm not sure if there is a good programatic way to fix this, tbh, while still preserving the ordering of the VCF. You could try manually splitting the VCF into smaller subsets before running them through the annotator.

[lukaas33]

Ah thank you.
Perhaps another approach would be to filter the VCF or limit the contigs I align to upstream.
But not really an issue with this tool then.

[chrisamiller]

I've looked into this a bit, and most of the performance hits come from the use of the vcfpy library, which is convenient in the way it splits apart the VCF and allows addressing fields, but is also quite slow. Using pysam would be quite a bit faster, I believe, but also would require a rewrite of a lot of the code, making it more difficult to maintain, and would also be much more picky about requiring things like ##contig headers on the VCF. I don't think it's feasible at this point.

That said, I did profile the code and found the following results:

(medium.vcf, 746K records)
157s total. 0.21ms/record.

Profiled with cProfile on medium.vcf + medium_readcounts_snvs.tsv (SNV mode, SAMPLE_01).

Category	Time	%
vcfpy write cycle	~61s	39%
vcfpy read cycle	~43s	27%
parse_brct_field (readcount field parsing)	24s	15%
parse_bam_readcount_file (file load)	5s	3%
parse_to_bam_readcount redundant calls	2s	1%

I'm working on a PR that speeds up some of the parsing a bit, but it's not going to be a huge improvement. I think for now, the answer is, if you have a massive VCF (many millions of lines), this is probably not the right tool.