mxnet random seed should not be fixed by default

[leezu]

mxnet seems to use a fixed seed for its random number generator. This is not good. To make sure research results are valid, experiments need to be repeated with different random initializations. To the best of my knowledge it is common practice that the seed for the random number generator is chosen by default randomly at startup. At least that is the behaviour of numpy.

As mxnet does not follow the expectated behaviour researchers may wrongly assume that their results are statistically significant, given that they'd expect a random seed.

Compare what happens when running the following two files repeatedly:

➜  ~ cat nprandom.py 
import numpy as np
print(np.random.normal(size=(2,2)))
➜  ~ cat mxrandom.py 
import mxnet as mx
print(mx.nd.random_normal(shape=(2,2)))

➜  ~ python3 nprandom.py
[[ 0.07005017 -0.10707515]
 [ 0.99219416  0.16670986]]
➜  ~ python3 nprandom.py
[[ 1.2185212  -1.56751472]
 [ 1.14549379 -1.90969064]]

➜  ~ python3 mxrandom.py

[[ 2.21220636  1.16307867]
 [ 0.7740038   0.48380461]]
<NDArray 2x2 @cpu(0)>
➜  ~ python3 mxrandom.py

[[ 2.21220636  1.16307867]
 [ 0.7740038   0.48380461]]
<NDArray 2x2 @cpu(0)>

I.e. mxnet random number generator uses the same seed..

[leezu]

@mli @piiswrong maybe worth fixing before the new release. Or at least explicitly warning about this.

[szha]

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!
Also, do please check out our forum (and Chinese version) for general "how-to" questions.

[larroy]

I think this is the appropriate behaviour. If you want different seeds, you should seed yourself using time or the systems random pool.

[imgemp]

I strongly disagree. Fixing the random seed to zero is not the default behavior in any package I've come across. It also goes against the purpose of a "random" library. By default, any outputs that are expected to exhibit randomness, should not be deterministic. When debugging or ensuring reproducibility, it is useful to be able to set the random seed, but otherwise it should be set as you say: based on time or system state. At the least, this should raise a warning. It's very difficult to discover this behavior on your own when other commonly used libraries like numpy add an additional source of true randomness on top of mxnet's deterministic default.

[wkcn]

I prefer the fixed random seed by default. It is similar to the random seed setting in C++. Users know what the random seed is and reproduce the experiment with the seed.

[eric-haibin-lin]

I suggest we fix it in the numpy compatibility project. @reminisce @haojin2

[chinakook]

As I'vs tested, pytorch behaved like numpy.random.

[wkcn]

I consider the following solution:
We should follow the random seed settings of different frontends.
For c++ front-end, the initial random seed is 0.
For Python front-end, the initial random seed is random. In addition, we provide a funcyion to get current random seed.

[junrushao]

What about going the JAX/scikit-learn way? that we explicitly require RandomSeed to be fed?

[leezu]

@haojin2 can you comment on if this will be fixed as part of the numpy compatibility effort?

[zburning]

I think it could be better to warn users if they do not feed a seed. I'm training a model but it cannot converge until I feed the seed manually. It took me much time to check my code...

[haojin2]

@leezu Have we all reached a consensus on what is the direction to go? If there's not even a clear goal yet then I cannot confirm that this will be changed in the 1.6.0 release.

[leezu]

Would the consensus be already reached based on meta-consenus on numpy compatibility #14253?
This may rule out the idea of following JAX/scikit-learn and requesting explict seed (@junrushao1994).

[junrushao]

@leezu Having an explicit random seed absolutely helps researchers in terms of reproducibility, as JAX did it, and scikit-learn allows you to provide a RandomState object. I didn't quite think deep on this yet, however. I think @szhengac and @sxjscience are much more familiar with this

[leezu]

@junrushao1994 I agree about the advantages. The problem is that it doesn't follow numpy API.
Would it make sense to add it as part of a npx.random?

[sxjscience]

I suggest fixing the seeding issue in 1.6.1. Currently, MXNet is not consistently even if we fix the seed: #16605 #16532 #16705 Also, we should try to accelerate the performance of our default random number generator.

…

________________________________ From: Leonard Lausen <[email protected]> Sent: Friday, November 15, 2019 12:07:06 AM To: apache/incubator-mxnet <[email protected]> Cc: Xingjian SHI <[email protected]>; Mention <[email protected]> Subject: Re: [apache/incubator-mxnet] mxnet random seed should not be fixed by default (#7410) @junrushao1994<https://github.com/junrushao1994> I agree about the advantages. The problem is that it doesn't follow numpy API. Would it make sense to add it as part of a npx.random? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://github.com/apache/incubator-mxnet/issues/7410?email_source=notifications&email_token=ABHQH3SM2COXIZSEUSKK5WLQTZKCVA5CNFSM4DWLSD62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEEURHQ#issuecomment-554256542>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHQH3X6XWJ23LSGTLKD4HLQTZKCVANCNFSM4DWLSD6Q>.

[leezu]

I suggest fixing the seeding issue in 1.6.1.

Which of above options would you recommend?
Also, changing the default seeding behavior in 1.6.1 point release will break our backwards compatibility promise. We may need to make this change in a proper release.

[sxjscience]

We can fix the dropout consistency issue in 1.6.1. In 1.7, we can support configuration files in MXNet and create some common config files, e.g, configuration that guarantees consistent random number generator.

…

________________________________ From: Leonard Lausen <[email protected]> Sent: Sunday, November 17, 2019 11:07:25 PM To: apache/incubator-mxnet <[email protected]> Cc: Xingjian SHI <[email protected]>; Mention <[email protected]> Subject: Re: [apache/incubator-mxnet] mxnet random seed should not be fixed by default (#7410) I suggest fixing the seeding issue in 1.6.1. Which of above options would you recommend? Also, changing the default seeding behavior in 1.6.1 point release will break our backwards compatibility promise. We may need to make this change in a proper release. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://github.com/apache/incubator-mxnet/issues/7410?email_source=notifications&email_token=ABHQH3RLZ3MTPQLVXLH5GXDQUI5K3A5CNFSM4DWLSD62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEJNCZI#issuecomment-554881381>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHQH3XX54BFQRAE6O2TWC3QUI5K3ANCNFSM4DWLSD6Q>.

[xidulu]

FYI:
JAX's RNG design aims for backend-independent reproducibility and random samplers without side-effect (for the puorpose of parallelization).
https://github.com/google/jax/blob/master/design_notes/prng.md

Also, JAX requires explicit RNG key for all operators with randomness.
For example, parameter initilization
https://github.com/google/jax/blob/9198656055d7d0d3cbb3ebfd6c0979572cc50924/examples/mnist_classifier.py#L82

I think it's better to resolve the behavior difference between DeepNumpy and Numpy through some kinds of global random seed configuration.

[leezu]

Marking as bug as the behaviour is not numpy compatible.

[karan6181]

I saw a different result on CPU and GPU by using the same seed value. Is this an expected behavior or am I missing something here ??

script

import mxnet as mx
mx.random.seed(99)
t_cpu = mx.nd.random.uniform(shape=(4), ctx=mx.cpu())
print("CPU: ", t_cpu)
mx.random.seed(99)
t_gpu = mx.nd.random.uniform(shape=(4), ctx=mx.gpu(0))
print("GPU: ", t_gpu)

Output

CPU:
[0.9509902  0.71019113 0.7611274  0.08560332]
<NDArray 4 @cpu(0)>
GPU:
[0.29560053 0.07938761 0.29997164 0.4891801 ]
<NDArray 4 @gpu(0)>

[szha]

I saw a different result on CPU and GPU by using the same seed value. Is this an expected behavior or am I missing something here ??

script

import mxnet as mx
mx.random.seed(99)
t_cpu = mx.nd.random.uniform(shape=(4), ctx=mx.cpu())
print("CPU: ", t_cpu)
mx.random.seed(99)
t_gpu = mx.nd.random.uniform(shape=(4), ctx=mx.gpu(0))
print("GPU: ", t_gpu)

Output

CPU:
[0.9509902  0.71019113 0.7611274  0.08560332]
<NDArray 4 @cpu(0)>
GPU:
[0.29560053 0.07938761 0.29997164 0.4891801 ]
<NDArray 4 @gpu(0)>

This is expected. Setting the seed on a device only guarantees reproducible results on the same device, and does not guarantee the same result on different devices.