Request for the example datasets?

[conniecl]

Hello authors, are there any example datasets for model training? Or how can I generate the correct input file format when training the model?

[AayushGrover]

Hello,

Thank you for your interest in our work.

An example training script is available here: tutorials/train.py.

This script requires the following four input files:

1. signal_file – The ATAC-seq bigwig used as the target for model training. For the GM12878 cell line, it can be found on ENCODE (ENCFF667MDI).
2. genome – The reference genome used as the input to the model. For hg38, it can be downloaded from the UCSC Genome Browser (hg38.fa.gz).
3. blacklist_file – A BED file listing regions to ignore during training. For hg38, this can be obtained from the Basenji2 manuscript (blacklist file).
4. unmap_file – A BED file containing unmappable regions. For hg38, this can also be obtained from the Basenji2 manuscript (unmap file).

With these four files, you should be able to train any of the seven models described in our manuscript. More details on model training can be found on our Wiki page.

Please let us know if you have any further questions.

[conniecl]

Thank you for your kind response. I'm also wondering if there are any suggestions about the GPU requirement for the model training for a typical human ATAC-seq dataset?

[AayushGrover]

While all models discussed in this work can be trained on a single RTX2080Ti GPU, having a system with 2 or 4 GPUs (RTX2080Ti or better) will significantly accelerate training. To use multiple GPUs (say, 4 for example), you can set n_gpus = 4 as shown in tutorials/train.py.

In our preprint, we have mentioned the training runtime of each method on human ATAC-seq datasets using a single RTX2080Ti GPU (Figure 2d).

Please let us know if you have any further questions.

[conniecl]

Thank you again for your kind reply. I try to run a trainning in RTX 4090 24G with my dataset (will try to use the dataset you provided to see if the bug happened again), but the trainning stops with the following errors. And I cannot find my GPU with nvidia-smi. I suspect it may be down due to the memory issues or something unexpected? But how can I reduce the memory usage during the training step? And the log file was also attacted.

test_gpu.log

Traceback (most recent call last):
  File "train_maize.py", line 49, in <module>
    main()
  File "train_maize.py", line 39, in main
    asap.train_model(
  File "/mnt/sda/lchen/software/anaconda3/envs/atac-asap/lib/python3.8/site-packages/asap/task.py", line 77, in train_model
    trainer.fit(train_dset=train_dataset, val_dset=val_dataset, nr_epochs=max_epochs, learning_rate=learning_rate)
  File "/mnt/sda/lchen/software/anaconda3/envs/atac-asap/lib/python3.8/site-packages/asap/trainer/trainer.py", line 92, in fit
    _fit(    
  File "/mnt/sda/lchen/software/anaconda3/envs/atac-asap/lib/python3.8/site-packages/asap/trainer/trainer.py", line 335, in _fit
    train_log_payload = _train_epoch(
  File "/mnt/sda/lchen/software/anaconda3/envs/atac-asap/lib/python3.8/site-packages/asap/trainer/trainer.py", line 403, in _train_epoch
    X_i = X_i.to(rank)
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[AayushGrover]

Sorry to hear that you are facing this issue.

I cannot find my GPU with nvidia-smi.

Could you please elaborate on this? Generally, if you are unable to see any GPU with nvidia-smi, it seems to be a driver issue, rather than an issue with the package.

To ensure that pytorch is installed correctly, could you please share the output of the following command:

python -c "import torch; print(f'PyTorch Version: {torch.__version__}'); print(f'PyTorch CUDA Version: {torch.version.cuda}'); print(f'CUDA Available: {torch.cuda.is_available()}"

In case it is actually a memory issue, one can reduce the memory usage training by setting a smaller batch size (default: 64) within asap.train_model function.

[conniecl]

Hi @AayushGrover
Thank you very much for your help before. After a long time of effort (AS WELL AS FIXING THE GPU), I eventually got the prediction results from one of my ATAC-seq dataset. However, I faced a branch of bugs when following the procedures that provided in wiki page.

Mostly bugs were came from the format of "chromosome", which I provided was: 1, 2, 3.... (both the reference, bw signal, VCF and ect). But it seems like some functions in ASAP only accept the "chromosomes" like "chr1, chr2, chr3" etc (sorry about lost my mind how many places I have edited in the code, and cannot provide the detailed line here), is there any chance that ASAP can handle the chromosome format more flexibly?

Besides, I also have trouble in interpreting the output files, like in predict_snv_atac.py and export_predictions.py. For example, for the output file of predict_snv_atac.py (pasted as test_snv_predictions.csv below), I suspected the last two columns are the prediction score for ref allele and alt allele, and I need average then so that I can get a final comparison about the impact of the ref and alt allele?

test_snv_predictions.csv

Moreover, I'm wondering if you could provide some test files and a more detailed explanation about the input and output format of ASAP?

[AayushGrover]

Hi @conniecl,

I'm sorry that you had to manually debug the chromosome formatting issue.
To help me better understand the changes you made and identify which functions need special handling for chromosome names, could you please submit a new pull request with your fixes? That would be very helpful.

Thank you for pasting the output file of predict_snv_atac.py. The last two columns contain the ATAC-seq predictions for the reference allele and the alternate allele, respectively. These predictions are generated using a 2048 bp DNA sequence centered on the SNV as input.
To compute the fold change between the reference and alternate predictions:

• Since the models predict log-transformed ATAC-seq, we first exponentiate the predictions.
• We then apply 1D Gaussian smoothing to both prediction profiles.
• Finally, we calculate the fold change using the values in the central bin.

To make it easier to interpret the SNV prediction outputs, we've added a new Jupyter notebook that demonstrates how to work with the results. As an example, a pair of input and output files are also added here. Using the SNV predictions provided by you, the notebook can be used to generate plots like this:

The export_predictions.py script outputs ATAC-seq predictions as BigWig files, one for each chromosome specified in the chroms variable (see the example in the tutorial: tutorials/export_predictions.py).
These BigWig files contain predictions that have already been exponentiated, so they are ready to use directly without any additional processing.

The outputs of predict_snv_atac.py and export_predictions.py are now explained in more detail on the Wiki page. Thank you for the extremely helpful suggestions!

Please let us know if you have any further questions.

[conniecl]

Hi @AayushGrover
Thanks a lot for your explanation. I'll check the results later. And have created a new pull in #3, some related scripts were also provided, hope that will be useful.