Communication in parallel simulations 

In parallel simulations, communication between the cpus may be a challenge. In most cases, this is handled automatically by Asap, but sometimes you will need to communicate in your own modules. This is most likely to happen in on-the-fly data analysis.

This page falls in three parts. The first part explains how normal Asap communication works. The second part explains how you can use these pre-existing communication patterns in your own module. The third part explains how you can roll your own communication patterns from scratch, using the Message Passing Interface (MPI).

In the following, a task refers to the process running on each cpu/core (this is consistent with usual MPI terminology). Thus, if a calculation is running on eight cpus/cores, there will be eight tasks, each taking care of atoms belonging to one part of space. A neighboring task is a task responsible for an adjacent part of space.

Normal Asap communication 

When an Asap calculator calculates forces, energies or stresses in a parallel simulation, the following communication steps occur:

Global communication of changes: Each task determines if the atoms have been changed, and if so if any atom has been moved sufficiently to trigger a migration. This is then communicated globally, so that all tasks proceed to the following steps even if atoms have only been modifies in some tasks.
Migration (not every time): If any atom has moved sufficiently outside the part of space handled by its task, it is necessary to migrate atoms. All data about atoms that should be handled by another task is moved to that task.
Ghost atoms update: Ghost atoms are atoms that are not in the part of space handled by this task, but are sufficiently close that they interact with atoms handled by this task. Before a calculation can proceed, the positions and atomic numbers of all ghost atoms must be transmitted from the task responsible for them. The calculation can now proceed.
Additional ghost update in some calculators: Some calculators need an additional communication step halfway through the calculation, where intermediate results for the ghost atoms are communicated. This is for example the case for EMT, but not for LennardJones.

Using Asap’s communication patterns 

Migrating your own data 

If you have your own data about the atoms, you will need to migrate that data along with the atoms as they migrate. Example: During an MD simulation, you calculate the average position of all the atoms. This average value is associated with the atoms, and needs to remain associated with the right atoms when atoms are migrating between processors.

Fortunately, Asap can do this for you. If you store data on the atoms with atoms.set_data(name, array), that data is now associated with the atoms, and will migrate along with them. Access the data with atoms.get_data(name). Just be sure that all such data is stored on the atoms when the calculator is invoked (and migration may happen): after a call to atoms.get_forces(), atoms.get_potential_energy() etc any data in local variables of your script may not any longer be associated with the right atoms.

Piggybacking on the ghost atom mechanism 

If your on-the-fly analysis not only needs information about the atoms, but also about their neighbors, difficulties arise as the neighboring atoms may be on other processors. In some cases, Asap’s internal mechanisms may help.

If you need information about an atom’s neighbors at distances closer than the cutoff used in the Calculator, you can access the calculator’s neigbor list object. This object will return information about neighbors even if they are on other processors. The positions are returned directly by the XXXX method, it also returns the indices of the atoms. If the index is larger than the number of atoms on this processor, it refers to “ghost atoms” on other processors. Data for the ghost atoms can be found in the atoms.ghosts dictionary, it normally contains ‘positions’ and ‘numbers’ (atomic numbers). To access e.g. the atomic numbers of the ghost atoms, use atoms.ghosts['numbers'][i-len(atoms)], where i is the index returned by the neighbor list.

Other kinds of data may be added to the atoms.ghosts dictionary. Add an array of the same name and type as an array in atoms.arrays, with the appropriate shape (i.e. the first dimension should be the number of ghosts, the remaining dimensions should be the same as in atoms.arrays). After the next force/energy calculation, the ghost data will be updated. If you add the ghost data before the first calculation, the first dimension of the array should be 0.

Beware that calling e.g. atoms.get_forces() does not guarantee update of the ghost data, if any other kind of calculation has been performed since last time the atoms were moved. An update may be guaranteed by invalidating the neighbor list, that will also trigger a migration.

Communication using message passing 

The module asap3.mpi exposes the default MPI communicator, world, which can be used to perform your own communication. The interface is the same as the similar module in GPAW, any divergences that may arise should be regarded as bugs.

The MPI Communicator object 

class asap3.mpi.Communicator

MPI communicator object.

Wraps an MPI communicator. The default communicator, spanning every task, is available as asap3.mpi.world; sub-communicators are created with new_communicator. The interface follows the equivalent object in GPAW.

The size and rank attributes (documented below) are read-only.

abort(errcode)

Abort all MPI tasks. For emergency use only.

Parameters:: errcode (int) – Error code returned to the environment.

all_gather(a, b)

Gather an array from every task into a single array on every task.

Parameters:

a (numpy.ndarray) – This task’s contribution.
b (numpy.ndarray) – Array receiving the gathered data on every task. Its size must be the size of a times the number of tasks.

alltoallv(sbuffer, scounts, sdispls, rbuffer, rcounts, rdispls)

Send data from all tasks to all tasks, with a separate count and displacement for each task (a vectorised all-to-all exchange).

Parameters:

sbuffer (numpy.ndarray) – Contiguous send buffer.
scounts (numpy.ndarray) – Number of elements to send to each task.
sdispls (numpy.ndarray) – Offset into sbuffer of the data for each task.
rbuffer (numpy.ndarray) – Contiguous receive buffer.
rcounts (numpy.ndarray) – Number of elements to receive from each task.
rdispls (numpy.ndarray) – Offset into rbuffer of the data from each task.

Notes

Counts and displacements are given in array elements, not bytes.

barrier(): Block until all tasks in the communicator have called this method.

broadcast(data, root)

Broadcast an array in place from one task to all other tasks.

Parameters:

data (numpy.ndarray) – On task root, the array to send; on the other tasks, the array that receives it. It must have the same size and type on all tasks.
root (int) – Rank of the task holding the data to broadcast.

compare(other)

Compare this communicator with another one using MPI_Comm_compare.

Parameters:: other (Communicator) – The communicator to compare with.
Returns:: {‘ident’, ‘congruent’, ‘similar’, ‘unequal’} – The relationship between the two communicators.

gather(a, root, b=None)

Gather an array from every task into a single array on one task.

Like all_gather, but the result is collected only on task root.

Parameters:

a (numpy.ndarray) – This task’s contribution.
root (int) – Rank of the task collecting the data.
b (numpy.ndarray, optional) – On task root, the array receiving the gathered data; its size must be the size of a times the number of tasks. Not used on the other tasks.

get_c_object()

Return a pointer to the underlying MPI_Comm as a Python integer.

Returns:: int – The address of the communicator’s MPI_Comm handle, for interfacing with other C extensions that use MPI.

get_members()

Return the ranks of this communicator’s members in the parent communicator.

Returns:: numpy.ndarray

max(data, root=-1)

Reduce a scalar or array across all tasks with the maximum operation.

Behaves like sum but takes the element-wise maximum. Complex numbers are not supported.

Parameters:

data (int, float or numpy.ndarray) – The value(s) to reduce. An array is overwritten in place with the result.
root (int, optional) – If -1 (default), every task receives the result. Otherwise only task root receives it.

Returns:

int, float or None – The maximum if data is a scalar; None if data is an array (which is then modified in place).

max_scalar(a, root=-1): Maximum of a scalar across all tasks. Alias for max kept for GPAW compatibility.

min(data, root=-1)

Reduce a scalar or array across all tasks with the minimum operation.

Behaves like sum but takes the element-wise minimum. Complex numbers are not supported.

Parameters:

data (int, float or numpy.ndarray) – The value(s) to reduce. An array is overwritten in place with the result.
root (int, optional) – If -1 (default), every task receives the result. Otherwise only task root receives it.

Returns:

int, float or None – The minimum if data is a scalar; None if data is an array (which is then modified in place).

min_scalar(a, root=-1): Minimum of a scalar across all tasks. Alias for min kept for GPAW compatibility.

name()

Return the name of the processor node running this task.

Returns:: str

new_communicator(ranks)

Create a new communicator spanning a subset of the tasks.

Parameters:: ranks (numpy.ndarray or list of int) – The ranks, in this communicator, of the tasks that will be members of the new communicator.
Returns:: Communicator or None – The new communicator on tasks that are members of it, None on the other tasks.

probe(src=-1, tag=-1, block=False)

Check whether a message is available to be received.

Parameters:

src (int, optional) – Rank of the source task. The default, -1, matches any task.
tag (int, optional) – Message tag. The default, -1, matches any tag.
block (bool, optional) – If True, wait until a message is available instead of returning None. The default is False.

Returns:

tuple of int or None – (source, tag, count) for the available message, where count is its size in bytes; or None if no message is available.

product(data, root=-1)

Multiply a scalar or array across all tasks.

Parameters:

data (int, float or numpy.ndarray) – The value(s) to multiply. An array is multiplied element-wise and overwritten in place with the result. Complex numbers are not supported.
root (int, optional) – If -1 (default), every task receives the result. Otherwise only task root receives it.

Returns:

int, float or None – The product if data is a scalar; None if data is an array (which is then modified in place).

product_scalar(a, root=-1): Multiply a scalar across all tasks. Alias for product kept for GPAW compatibility.

rank: The rank of this task, an integer in the range 0 <= rank < size.

receive(a, src=-1, tag=123, block=True)

Receive an array from another MPI task.

Parameters:

a (numpy.ndarray) – Array to receive the data into. It must be contiguous and of the correct type and size; this is not checked. It should not be accessed until the communication has completed.
src (int, optional) – Rank of the source task. The default, -1, accepts data from any task.
tag (int, optional) – Message tag (default 123).
block (bool, optional) – If True (default), block until the data has been received. If False, return immediately with a Request object.

Returns:

int or Request – For a blocking receive, the rank of the task the data was received from. For a non-blocking receive, a Request object that can be used to query whether the communication has completed.

scatter(a, b, root)

Distribute contiguous chunks of an array from one task to all tasks.

Task N receives the N-th chunk of a into b.

Parameters:

a (numpy.ndarray) – On task root, the array to distribute; its size must be the size of b times the number of tasks. Not used on the other tasks.
b (numpy.ndarray) – Array to receive this task’s chunk.
root (int) – Rank of the task holding a.

send(a, dest, tag=123, block=True)

Send an array to another MPI task.

Parameters:

a (numpy.ndarray) – Contiguous array to send.
dest (int) – Rank of the destination task.
tag (int, optional) – Message tag (default 123).
block (bool, optional) – If True (default), use a blocking send. If False, use a non-blocking send; a must then not be modified until the communication has completed.

Returns:

None or Request – None for a blocking send. For a non-blocking send, a Request object that completes when a may be reused; it keeps a reference to a until then.

sendreceive(a, dest, b, src, sendtag=123, recvtag=123)

Send one array while simultaneously receiving another.

The same effect may be obtained by overlapping a non-blocking send with a blocking receive or vice versa, but sendreceive may give better performance with some MPI implementations.

Parameters:

a (numpy.ndarray) – Contiguous array to send to task dest.
dest (int) – Rank of the destination task.
b (numpy.ndarray) – Contiguous array to receive data from task src into.
src (int) – Rank of the source task.
sendtag (int, optional) – Tag of the sent message (default 123).
recvtag (int, optional) – Tag of the received message (default 123).

size: The number of MPI tasks in this communicator.

ssend(a, dest, tag=123)

Send an array using a synchronous blocking send.

Like send with block=True, but does not return until the receiving task has started to receive the message.

Parameters:

a (numpy.ndarray) – Contiguous array to send.
dest (int) – Rank of the destination task.
tag (int, optional) – Message tag (default 123).

sum(data, root=-1)

Sum a scalar or array across all tasks.

Parameters:

data (int, float, complex or numpy.ndarray) – The value(s) to sum. An array is summed element-wise and overwritten in place with the result.
root (int, optional) – If -1 (default), every task receives the result. Otherwise only task root receives it.

Returns:

int, float, complex or None – The sum if data is a scalar; None if data is an array (which is then modified in place).

Notes

Unlike product, min and max, this method also supports complex numbers.

sum_scalar(a, root=-1): Sum a scalar across all tasks. Alias for sum kept for GPAW compatibility.

test(request)

Test whether a non-blocking communication has completed.

Equivalent to request.test(). If the communication is complete the request is cleaned up, so calling wait afterwards is optional (but allowed).

Parameters:: request (Request) – The request object returned by a non-blocking send or receive.
Returns:: bool – True if the communication has completed, False otherwise.

testall(requests)

Test whether all of a sequence of non-blocking communications have completed.

Parameters:: requests (sequence of Request) – The request objects to test.
Returns:: int – Nonzero if all of the communications have completed, zero if at least one is still incomplete.

translate_ranks(other, ranks)

Translate task ranks from this communicator into another communicator.

Parameters:

other (Communicator) – The communicator to translate the ranks into.
ranks (numpy.ndarray or list of int) – Ranks valid in this communicator.

Returns:

numpy.ndarray – The corresponding ranks in other, with -1 where a rank has no counterpart. The result has the same dtype as ranks when that is an array.

wait(request)

Wait until a non-blocking communication has completed.

Equivalent to request.wait().

Parameters:: request (Request) – The request object returned by a non-blocking send or receive.

waitall(requests)

Wait until all of a sequence of non-blocking communications have completed.

Parameters:: requests (sequence of Request) – The request objects to wait for.

Request object 

The send and receive methods of the communicator object will return a Request object if called with block=False. The request object should be used to wait for the communication to complete by calling the wait and/or test method.

class asap3.mpi.Request

MPI request object.

Represents a pending non-blocking communication. Instances are created by the send and receive methods of the communicator object when called with block=False, and are not meant to be instantiated directly. Use the wait and/or test method to wait for the communication to complete.

Notes

It is safe to call wait and test multiple times on the same object. Once test has returned True or wait has been called, any subsequent call to test returns True.

You should not discard a request until wait has been called, or test has returned True, or you have by some other means ensured that the communication has completed. Failure to ensure this may cause the interpreter to block while garbage-collecting the request object, until the communication completes.

status: Nonzero while the communication is still pending, zero once it has completed.

test()

Test whether the communication has completed.

Returns:: bool – True if the communication has completed, False otherwise. Once True has been returned, the reference to the array used for the communication has been released.

wait(): Wait until the communication has completed, releasing the reference to the array used for the communication.