Basic domain decomposition based parallelisation, dividing the domain into #procs equal sized cuboids. More...

#include <DomainDecomposition.h>

Inheritance diagram for DomainDecomposition:

Public Member Functions
	DomainDecomposition ()
	The constructor has to determine the own rank and the number of neighbours and sets up the topology.

void	readXML (XMLfileUnits &xmlconfig) override
	Read in XML configuration for DomainDecomposition and all its included objects. More...

double	getBoundingBoxMin (int dimension, Domain *domain) override
	get the minimum of the bounding box of this process' domain in the given dimension (0,1,2) More...

double	getBoundingBoxMax (int dimension, Domain *domain) override
	get the maximum of the bounding box of this process' domain in the given dimension (0,1,2) More...

void	balanceAndExchange (double lastTraversalTime, bool forceRebalancing, ParticleContainer moleculeContainer, Domain domain) override
	balance the load (and optimize communication) and exchange boundary particles More...

void	initCommunicationPartners (double cutoffRadius, Domain domain, ParticleContainer moleculeContainer)

std::vector< int >	getNeighbourRanks () override

std::vector< int >	getNeighbourRanksFullShell () override

std::vector< std::vector< std::vector< int > > >	getAllRanks ()

void	prepareNonBlockingStage (bool forceRebalancing, ParticleContainer moleculeContainer, Domain domain, unsigned int stageNumber) override

void	finishNonBlockingStage (bool forceRebalancing, ParticleContainer moleculeContainer, Domain domain, unsigned int stageNumber) override

bool	queryBalanceAndExchangeNonBlocking (bool forceRebalancing, ParticleContainer moleculeContainer, Domain domain, double etime) override
	Checks whether the balance and exchange step can be performed non-blocking. More...

std::vector< CommunicationPartner >	getNeighboursFromHaloRegion (Domain *domain, const HaloRegion &haloRegion, double cutoff) override

Public Member Functions inherited from DomainDecompMPIBase
void	barrier () const override
	synchronizes all processes More...

unsigned	Ndistribution (unsigned localN, float minrnd, float maxrnd)
	returns total number of molecules More...

void	assertIntIdentity (int IX)
	checks identity of random number generators More...

void	assertDisjunctivity (ParticleContainer *moleculeContainer) const override

void	collCommInit (int numValues, int key=0) override
	has to call init method of a CollComm class More...

void	collCommFinalize () override
	has to call finalize method of a CollComm class More...

void	collCommAppendInt (int intValue) override
	has to call appendInt method of a CollComm class More...

void	collCommAppendUnsLong (unsigned long unsLongValue) override
	has to call appendUnsLong method of a CollComm class More...

void	collCommAppendFloat (float floatValue) override
	has to call appendFloat method of a CollComm class More...

void	collCommAppendDouble (double doubleValue) override
	has to call appendDouble method of a CollComm class More...

void	collCommAppendLongDouble (long double longDoubleValue) override
	has to call appendLongDouble method of a CollComm class More...

int	collCommGetInt () override
	has to call getInt method of a CollComm class More...

unsigned long	collCommGetUnsLong () override
	has to call getUnsLong method of a CollComm class More...

float	collCommGetFloat () override
	has to call getFloat method of a CollComm class More...

double	collCommGetDouble () override
	has to call getDouble method of a CollComm class More...

long double	collCommGetLongDouble () override
	has to call getLongDouble method of a CollComm class More...

void	collCommAllreduceSum () override
	has to call allreduceSum method of a CollComm class (none in sequential version) More...

void	collCommAllreduceSumAllowPrevious () override
	has to call allreduceSum method of a CollComm class (none in sequential version), allows for values of previous iteration. More...

void	collCommAllreduceCustom (ReduceType type) override
	has to call allreduceCustom method of a CollComm class (none in sequential version) More...

void	collCommScanSum () override
	has to call scanSum method of a CollComm class (none in sequential version) More...

void	collCommBroadcast (int root=0) override
	has to call broadcast method of a CollComm class (none in sequential version) More...

virtual void	balanceAndExchangeInitNonBlocking (bool forceRebalancing, ParticleContainer moleculeContainer, Domain domain)

void	exchangeMoleculesMPI (ParticleContainer moleculeContainer, Domain domain, MessageType msgType, bool doHaloPositionCheck=true, bool removeRecvDuplicates=false)
	exchange molecules between processes More...

void	exchangeForces (ParticleContainer moleculeContainer, Domain domain) override
	Exchanges forces at the domain boundaries if it's required by the cell container. More...

MPI_Datatype	getMPIParticleType ()

MPI_Datatype	getMPIParticleForceType ()

MPI_Comm	getCommunicator () override

virtual void	setCommunicationScheme (const std::string &scheme, const std::string &comScheme)

virtual int	getNonBlockingStageCount () override

virtual size_t	getTotalSize () override

virtual void	printSubInfo (int offset) override

virtual void	printDecomp (const std::string &filename, Domain domain, ParticleContainer particleContainer) override
	writes information about the current decomposition into the given file More...

virtual std::string	getName () override

void	printCommunicationPartners (std::string filename) const override

Public Member Functions inherited from DomainDecompBase
	DomainDecompBase ()
	The Constructor determines the own rank and the number of the neighbours *‍/.

	~DomainDecompBase () override
	The Destructor finalizes MPI.

void	exchangeMolecules (ParticleContainer moleculeContainer, Domain domain)
	exchange molecules between processes More...

virtual bool	procOwnsPos (double x, double y, double z, Domain *domain) final
	find out whether the given position belongs to the domain of this process More...

void	getBoundingBoxMinMax (Domain domain, double min, double *max)
	get the minimum and maximum coordinate of the bounding box of this process' domain More...

virtual int	getRank () const
	returns the own rank More...

virtual int	getNumProcs () const
	returns the number of processes More...

virtual double	getTime () const
	returns the time in seconds since some time in the past

double	getIOCutoffRadius (int dim, Domain domain, ParticleContainer moleculeContainer)
	returns an cutoff radius for a dimension for a global linked cells datastructure More...

void	writeMoleculesToMPIFileBinary (const std::string &filename, ParticleContainer *moleculeContainer) const
	appends molecule data to the file. The format is the same as that of the input file This version uses, MPI IO. More...

void	writeMoleculesToFile (const std::string &filename, ParticleContainer *moleculeContainer, bool binary=false) const
	appends molecule data to the file. The format is the same as that of the input file If MPI is enabled and binary files are supposed to be written this function will call writeMoleculesToMPIFileBinary(). Otherwise, to ensure that not more than one process writes to the file at any time, there is a loop over all processes with a barrier in between. More...

void	updateSendLeavingWithCopies (bool sendTogether)

bool	sendLeavingWithCopies () const

Additional Inherited Members
Protected Member Functions inherited from DomainDecompMPIBase
virtual void	prepareNonBlockingStageImpl (ParticleContainer moleculeContainer, Domain domain, unsigned int stageNumber, MessageType msgType, bool removeRecvDuplicates=false)

virtual void	finishNonBlockingStageImpl (ParticleContainer moleculeContainer, Domain domain, unsigned int stageNumber, MessageType msgType, bool removeRecvDuplicates=false)

Protected Member Functions inherited from DomainDecompBase
void	addLeavingMolecules (std::vector< Molecule > &&invalidMolecules, ParticleContainer *moleculeContainer)

void	handleDomainLeavingParticles (unsigned dim, ParticleContainer *moleculeContainer) const

void	handleDomainLeavingParticlesDirect (const HaloRegion &haloRegion, ParticleContainer *moleculeContainer, std::vector< Molecule > &invalidParticles) const

void	handleForceExchange (unsigned dim, ParticleContainer *moleculeContainer) const
	Does the force exchange for each dimension. Will be called for dim=0, 1 and 2. More...

virtual void	handleForceExchangeDirect (const HaloRegion &haloRegion, ParticleContainer *moleculeContainer) const

void	populateHaloLayerWithCopies (unsigned dim, ParticleContainer *moleculeContainer) const

void	populateHaloLayerWithCopiesDirect (const HaloRegion &haloRegion, ParticleContainer *moleculeContainer, bool positionCheck=true) const

Protected Attributes inherited from DomainDecompMPIBase
MPI_Datatype	_mpiParticleType

MPI_Datatype	_mpiParticleForceType

MPI_Comm	_comm

std::unique_ptr< NeighbourCommunicationScheme >	_neighbourCommunicationScheme

bool	_forceDirectPP {false}

Protected Attributes inherited from DomainDecompBase
int	_rank
	the id of the current process

int	_numProcs
	total number of processes in the simulation

Detailed Description

Basic domain decomposition based parallelisation, dividing the domain into #procs equal sized cuboids.

In a domain decomposition, each process gets part of the spacial domain. In this implementation, the whole domain has to be a cuboid which is decomposed into several, equally sized smaller cuboids. At the boundary, each process needs molecules from neighbouring domains to be able to calculate the forces on the own molecules. Molecules are moving across the boundaries of local domains. So methods are implemented to transfer those molecules.

[2]

Member Function Documentation

◆ balanceAndExchange()

void DomainDecomposition::balanceAndExchange	(	double	lastTraversalTime,
		bool	forceRebalancing,
		ParticleContainer *	moleculeContainer,
		Domain *	domain
	)

overridevirtual

balance the load (and optimize communication) and exchange boundary particles

This method is used to perform a new decomposition of the global domain with the goal of getting the equal load (and possibly additional goals) on each process. This method is then also responsible for redistributing all particles, so after the method was called, each process has a domain with all particles belonging to this domain (as if exchangeParticles was called after the new decomposition).

Parameters

balance	if true, a rebalancing is forced; otherwise automatic balancing of Decomposition is applied
moleculeContainer	needed for calculating load and to get the particles
domain	is e.g. needed to get the size of the local domain

Reimplemented from DomainDecompBase.

◆ finishNonBlockingStage()

void DomainDecomposition::finishNonBlockingStage	(	bool	forceRebalancing,
		ParticleContainer *	moleculeContainer,
		Domain *	domain,
		unsigned int	stageNumber
	)

overridevirtual

Finishes the stageNumber'th stage. This includes receiving the particles, that have to be received in that stage.

Parameters

forceRebalancing	true if rebalancing should be forced
moleculeContainer	pointer to the molecule container
domain	pointer to the domain
stageNumber	the number of the stage, the communication is in.

Implements DomainDecompMPIBase.

◆ getAllRanks()

std::vector< std::vector< std::vector< int > > > DomainDecomposition::getAllRanks ( )

virtual

Reimplemented from DomainDecompBase.

◆ getBoundingBoxMax()

double DomainDecomposition::getBoundingBoxMax	(	int	dimension,
		Domain *	domain
	)

overridevirtual

get the maximum of the bounding box of this process' domain in the given dimension (0,1,2)

Parameters

dimension	coordinate direction for which the maximum of the bounding box is returned
domain	here the bounding box is stored

Reimplemented from DomainDecompBase.

◆ getBoundingBoxMin()

double DomainDecomposition::getBoundingBoxMin	(	int	dimension,
		Domain *	domain
	)

overridevirtual

get the minimum of the bounding box of this process' domain in the given dimension (0,1,2)

Parameters

dimension	coordinate direction for which the minimum of the bounding box is returned
domain	here the bounding box is stored

Reimplemented from DomainDecompBase.

◆ getNeighbourRanks()

std::vector< int > DomainDecomposition::getNeighbourRanks ( )

overridevirtual

Implements DomainDecompMPIBase.

◆ getNeighbourRanksFullShell()

std::vector< int > DomainDecomposition::getNeighbourRanksFullShell ( )

overridevirtual

The key of this function is that opposite sites are always neighbouring each other in the array (i.e. leftAreaIndex = 0, righAreaIndex = 1, ...)

Implements DomainDecompMPIBase.

◆ getNeighboursFromHaloRegion()

std::vector< CommunicationPartner > DomainDecomposition::getNeighboursFromHaloRegion	(	Domain *	domain,
		const HaloRegion &	haloRegion,
		double	cutoff
	)

overridevirtual

Implements DomainDecompMPIBase.

◆ prepareNonBlockingStage()

void DomainDecomposition::prepareNonBlockingStage	(	bool	forceRebalancing,
		ParticleContainer *	moleculeContainer,
		Domain *	domain,
		unsigned int	stageNumber
	)

overridevirtual

Prepares the stageNumber'th stage. This includes sending particles, that have to be send in that stage.

Parameters

forceRebalancing	true if rebalancing should be forced
moleculeContainer	pointer to the molecule container
domain	pointer to the domain
stageNumber	the number of the stage, the communication is in.

Implements DomainDecompMPIBase.

◆ queryBalanceAndExchangeNonBlocking()

bool DomainDecomposition::queryBalanceAndExchangeNonBlocking	(	bool	forceRebalancing,
		ParticleContainer *	moleculeContainer,
		Domain *	domain,
		double	etime
	)

overridevirtual

Checks whether the balance and exchange step can be performed non-blocking.

A non-blocking behaviour is typically possible, as long as no rebalancing has to be done.

Parameters

forceRebalancing	if true, a rebalancing is forced; otherwise automatic balancing of Decomposition is applied
moleculeContainer	needed for calculating load and to get the particles
domain	is e.g. needed to get the size of the local domain

Reimplemented from DomainDecompBase.

◆ readXML()

void DomainDecomposition::readXML ( XMLfileUnits & xmlconfig )

overridevirtual

Read in XML configuration for DomainDecomposition and all its included objects.

The following xml object structure is handled by this method:

<parallelisation type="DomainDecomposition">
  <!-- structure handled by DomainDecompMPIBase -->
  <MPIGridDims> <x>INT</x> <y>INT</y> <z>INT</z> </MPIGridDims>
</parallelisation>

Reimplemented from DomainDecompMPIBase.

The documentation for this class was generated from the following files:

src/parallel/DomainDecomposition.h
src/parallel/DomainDecomposition.cpp

Public Member Functions

Additional Inherited Members

Detailed Description

Member Function Documentation

◆ balanceAndExchange()

◆ finishNonBlockingStage()

◆ getAllRanks()

◆ getBoundingBoxMax()

◆ getBoundingBoxMin()

◆ getNeighbourRanks()

◆ getNeighbourRanksFullShell()

◆ getNeighboursFromHaloRegion()

◆ prepareNonBlockingStage()

◆ queryBalanceAndExchangeNonBlocking()

◆ readXML()