By: Marcus (m.delete@this.bitsnbites.eu), March 25, 2022 12:57 am
Room: Moderated Discussions
Adrian (a.delete@this.acm.org) on March 24, 2022 2:34 pm wrote:
> rwessel (rwessel.delete@this.yahoo.com) on March 24, 2022 1:28 pm wrote:
> > Paul A. Clayton (paaronclayton.delete@this.gmail.com) on March 24, 2022 11:43 am wrote:
> > > Pseudo-random number generator would provide reproducible stochastic rounding
> >
> > Easier said than done if the code is multi-threaded. Which is something of a given if we're assuming GPUs.
>
>
> There are many good PRNG's that allow the deterministic splitting of the
> generated sequence of numbers in separate subsequences for each thread.
>
> For example the multiplicative Fibonacci generators or those
> which use a pseudo-random function in counter mode.
>
>
> So the problem of having a multi-threaded program with pseudo-random but reproducible behavior
> has been solved since about 1984 for the cases when very fast PRNGs are needed and since
> about 1970 for the cases when the speed of the PRNGs does not matter so much.
>
What if you change your program so that you do some extra FP computations before the main work that you want to be reproducible? Or if you move from one compute node to another node with a different number of HW threads so that you partition the work differently?
I can see lots of situations where it would be hard to guarantee the same PRN sequence for your operations. Not saying that it can't be done, but it feels like an inherently fragile solution.
> rwessel (rwessel.delete@this.yahoo.com) on March 24, 2022 1:28 pm wrote:
> > Paul A. Clayton (paaronclayton.delete@this.gmail.com) on March 24, 2022 11:43 am wrote:
> > > Pseudo-random number generator would provide reproducible stochastic rounding
> >
> > Easier said than done if the code is multi-threaded. Which is something of a given if we're assuming GPUs.
>
>
> There are many good PRNG's that allow the deterministic splitting of the
> generated sequence of numbers in separate subsequences for each thread.
>
> For example the multiplicative Fibonacci generators or those
> which use a pseudo-random function in counter mode.
>
>
> So the problem of having a multi-threaded program with pseudo-random but reproducible behavior
> has been solved since about 1984 for the cases when very fast PRNGs are needed and since
> about 1970 for the cases when the speed of the PRNGs does not matter so much.
>
What if you change your program so that you do some extra FP computations before the main work that you want to be reproducible? Or if you move from one compute node to another node with a different number of HW threads so that you partition the work differently?
I can see lots of situations where it would be hard to guarantee the same PRN sequence for your operations. Not saying that it can't be done, but it feels like an inherently fragile solution.
