Reproducibility of MSA

[LivC193]

Hi congrats on continuing to improve your results. I am using your ESM1b model and I was exited to read your MSA Transformer paper. I am trying to reproduce your .a3m files that you placed in examples folder https://github.com/facebookresearch/esm/blob/master/examples/1a3a_1_A.a3m , but I cannot do it. From your paper you say you use all the default parameters except for number of iterations, which is set to 3. These are my parameters:
E-value : 1e-3
No of iterations: 3
Min Prob in hitlist: 20
Max target hits: 250

hhblits.zip

[joshim5]

Hi @LivC182, thanks for the question! The resulting MSAs will be sensitive to the database you provide to HHblits. Which dataset are you using to build the a3ms? We used UniClust30-2017-10.

[LivC193]

@joshim5 you are completely right it was the database. i thought I was using the same but I wasn't

[tomsercu]

@ryan-224 also asked about MSA construction;
In this thread are already some pointers; we use hhblits, an alternative is the HMMer suite.

[ryan-224]

Thanks @tomsercu
I am able to get some MSAs to test the model now. It works well for the provided examples.

One more questions, if I want to get the token representations from the MSAs, which repr_layer should I choose? 11 or 13?

[joshim5]

The MSA Transformer has 12 layers. The 14 hidden representations (0...13) that can be returned correspond to:

• 0: token + positional embedding
• 1-12: output of layer i
• 13: layer_norm(output of layer 12)

When pre-training the model, the output corresponding to repr_layer[13] is passed to a language modeling head, which itself applies linear(layer_norm(gelu(linear(repr_layer[13])))).

The best representation to use may depend on the task, so we encourage you try experiment with multiple options. Knowing nothing about your problem, repr_layer[13] is the most natural choice.

[xf-omics]

Hi, can you advise how to get the layer_norm (repr_layer[13])? I tried this command below, but there is no representations (13). Is repr_layer[13] the same as repr_layer[12]? Thank you!
out = msa_transformer(msa_batch_tokens, repr_layers=[13])

[AlexanderKroll]

Hi everyone,
I am facing a similar problem as LivC182 did, but unfortunately, I couln’t figure out the issue, yet. I am currently trying to reproduce the MSA from the uploaded example “1a3a_1_A.a3m”, but my a3m-file is very different to the uploaded one. I think I am doing everything as described here and in your paper to create the file:

• I am using HHsuite version 3.1.0 (downloaded from here: https://github.com/soedinglab/hh-suite/releases/tag/v3.1.0c)
• I am using the uniclust30_2017_10 database (I downloaded the UniRef30_2020_01_hhsuite.tar.gz fie from here: http://wwwuser.gwdg.de/~compbiol/uniclust/2020_01/)
• I set the number of iteration to three

To create the MSA, I am using the following command:
hhblits -i test_seq.fasta -oa3m test_seq.a3m -n 3 -d /uniclust30_2017_10/uniclust30_2017_10
where “test_seq.fasta” contains only the first sequence of the “1a3a_1_A.a3m”-file.

I would be very happy if anyone has an idea on what the issue might be for not getting the same alignment.

[AlexanderKroll]

Hi @LivC182 ,
it seems that your were able to reproduce the resulst from the file https://github.com/facebookresearch/esm/blob/master/examples/1a3a_1_A.a3m . Could you maybe have a looked at my description above and see if you spot any difference to your procedere? That would be great, thanks!

[robert-verkuil]

Hi @AlexanderKroll,

The a3m files in the examples directory here are not samples from the pre-training dataset used to train the MSA Transformer, they are instead samples from the trRosetta dataset used to benchmark it.

Given this, your hhblits command shouldn't reproduce the a3m file given here. Sorry for the confusion on this. We will update the notebook to make this clear.

That being said, your hhblits command looks good, in terms of generating the MSA's used during MSA Transformer pre-training:
hhblits -i test_seq.fasta -oa3m test_seq.a3m -n 3 -d /uniclust30_2017_10/uniclust30_2017_10
(not UniRef30_2020_01)

[AlexanderKroll]

Hi @robert-verkuil,

thanks a lot for your response and for clarifying that my hhblits output shouldn't reproduce the given a3m file!

[tomsercu]

We apologize for the confusion here. In order to confirm your msa generation pipeline, we added some more MSAs in the new release. See https://github.com/facebookresearch/esm/tree/master/examples

[lzhangUT]

@joshim5 Hi,
I am trying to generate MSA following your steps. I read the discussion here, that's useful, but i am a little bit confused here, in your paper ' , you mentioned that "We compare with EVMutation [4] and DeepSequence [20], state-of-the-art unsupervised variant prediction methods. Since the published versions of these methods use MSAs generated
from an earlier version of Uniref100, we generate new MSAs using the EVMutation methodology
and the version of Uniref100 concurrent with our pretraining dataset. We train the replications of
EVMutation and DeepSequence using their open source code. The same MSAs are also used in the
few-shot experiments with MSA Transformer, and for unsupervised fine-tuning experiments with
ESM-1v.", and in EVmutation, they use jackhmmer for MSA, I think you used that as well to generate your MSA.
however, here in your examples here, you used HHblits to generate MSA. so I am not sure which one to follow. I am asking because when I use jackhmmer for MSA, I get stockholm file, and in which the alignments have a lot of gaps starting for the target sequence. if you were using that as you said in your paper, how did you deal with the gaps? and do you have a code to postprocess the stockholm file to generate the MSA that needed for MSA transformer?
Thanks