Nektar::MultiRegions::AllToAll Class Reference

#include <AssemblyCommDG.h>

Inheritance diagram for Nektar::MultiRegions::AllToAll:

Public Member Functions
	AllToAll (const LibUtilities::CommSharedPtr &comm, const int &maxQuad, const int &nRanks, const std::map< int, std::vector< int >> &rankSharedEdges, const std::map< int, std::vector< int >> &edgeToTrace)
	Default constructor. More...
void	PerformExchange (const Array< OneD, NekDouble > &testFwd, Array< OneD, NekDouble > &testBwd) final
Public Member Functions inherited from Nektar::MultiRegions::ExchangeMethod
	ExchangeMethod ()=default
	Default constructor. More...
virtual	~ExchangeMethod ()=default
	Default destructor. More...

Private Attributes
LibUtilities::CommSharedPtr	m_comm
	Communicator. More...
int	m_maxQuad = 0
	Max number of quadrature points in an element. More...
int	m_nRanks = 0
	Number of ranks/processes/partitions. More...
std::vector< int >	m_allEdgeIndex
	List of trace map indices of the quad points to exchange. More...
int	m_maxCount = 0
	Largest shared partition edge. More...

Detailed Description

Uses the MPI_AllToAll collective operation to perform the exchange of quadrature values. This does not allow for varying exchange array sizes so padding is used to ensure all partitions send/receive the same length array. All ranks communicate full array sizes to all other ranks. One collective operation is posted on each rank which requires communication.

Definition at line 102 of file AssemblyCommDG.h.

Constructor & Destructor Documentation

AllToAll()

Nektar::MultiRegions::AllToAll::AllToAll	(	const LibUtilities::CommSharedPtr &	comm,
		const int &	maxQuad,
		const int &	nRanks,
		const std::map< int, std::vector< int >> &	rankSharedEdges,
		const std::map< int, std::vector< int >> &	edgeToTrace
	)

Default constructor.

Definition at line 46 of file AssemblyCommDG.cpp.

    : m_comm(comm), m_maxQuad(maxQuad), m_nRanks(nRanks)
{
    for (size_t i = 0; i < nRanks; ++i)
    {
        if (rankSharedEdges.find(i) != rankSharedEdges.end())
        {
            m_maxCount = (rankSharedEdges.at(i).size() > m_maxCount)
                             ? static_cast<int>(rankSharedEdges.at(i).size())
                             : m_maxCount;
        }
    }
 
    comm->AllReduce(m_maxCount, LibUtilities::ReduceMax);
 
    // Creates the edge index vector where value -1 indicates
    // padding of value 0 to be inserted instead of value from Fwd
    for (size_t i = 0; i < nRanks; ++i)
    {
        if (rankSharedEdges.find(i) != rankSharedEdges.end())
        {
            for (size_t j = 0; j < rankSharedEdges.at(i).size(); ++j)
            {
                std::vector<int> edgeIndex =
                    edgeToTrace.at(rankSharedEdges.at(i)[j]);
                if (edgeIndex.size() < maxQuad)
                {
                    std::vector<int> diff(maxQuad - edgeIndex.size(), -1);
                    edgeIndex.insert(edgeIndex.end(), diff.begin(), diff.end());
                }
 
                m_allEdgeIndex.insert(m_allEdgeIndex.end(), edgeIndex.begin(),
                                      edgeIndex.end());
            }
 
            if (rankSharedEdges.at(i).size() < m_maxCount)
            {
                std::vector<int> edgeIndex(
                    maxQuad * (m_maxCount - rankSharedEdges.at(i).size()), -1);
                m_allEdgeIndex.insert(m_allEdgeIndex.end(), edgeIndex.begin(),
                                      edgeIndex.end());
            }
        }
        else
        {
            std::vector<int> edgeIndex(maxQuad * m_maxCount, -1);
            m_allEdgeIndex.insert(m_allEdgeIndex.end(), edgeIndex.begin(),
                                  edgeIndex.end());
        }
    }
}

References m_allEdgeIndex, m_maxCount, and Nektar::LibUtilities::ReduceMax.

Member Function Documentation

PerformExchange()

void Nektar::MultiRegions::AllToAll::PerformExchange ( const Array< OneD, NekDouble > &  testFwd, Array< OneD, NekDouble > &  testBwd  )

finalvirtual

Perform MPI comm exchange taking the Fwd trace and sending partition edge trace values to the matching locations in the Bwd trace of corresponding adjacent partitions.

Parameters

[in]	testFwd	The values to send to adjacent partitions
[out]	testBwd	The values received from adjacent partitions

Implements Nektar::MultiRegions::ExchangeMethod.

Definition at line 232 of file AssemblyCommDG.cpp.

{
    int size = m_maxQuad * m_maxCount * m_nRanks;
    Array<OneD, NekDouble> sendBuff(size, -1);
    Array<OneD, NekDouble> recvBuff(size, -1);
 
    for (size_t j = 0; j < size; ++j)
    {
        if (m_allEdgeIndex[j] == -1)
        {
            sendBuff[j] = 0;
        }
        else
        {
            sendBuff[j] = testFwd[m_allEdgeIndex[j]];
        }
    }
 
    m_comm->AlltoAll(sendBuff, recvBuff);
 
    for (size_t j = 0; j < size; ++j)
    {
        if (m_allEdgeIndex[j] != -1)
        {
            testBwd[m_allEdgeIndex[j]] = recvBuff[j];
        }
    }
}

References m_allEdgeIndex, m_comm, m_maxCount, m_maxQuad, and m_nRanks.

Member Data Documentation

m_allEdgeIndex

std::vector<int> Nektar::MultiRegions::AllToAll::m_allEdgeIndex

private

List of trace map indices of the quad points to exchange.

Definition at line 124 of file AssemblyCommDG.h.

Referenced by AllToAll(), and PerformExchange().

m_comm

LibUtilities::CommSharedPtr Nektar::MultiRegions::AllToAll::m_comm

private

Communicator.

Definition at line 118 of file AssemblyCommDG.h.

Referenced by PerformExchange().

m_maxCount

int Nektar::MultiRegions::AllToAll::m_maxCount = 0

private

Largest shared partition edge.

Definition at line 126 of file AssemblyCommDG.h.

Referenced by AllToAll(), and PerformExchange().

m_maxQuad

int Nektar::MultiRegions::AllToAll::m_maxQuad = 0

private

Max number of quadrature points in an element.

Definition at line 120 of file AssemblyCommDG.h.

Referenced by PerformExchange().

m_nRanks

int Nektar::MultiRegions::AllToAll::m_nRanks = 0

private

Number of ranks/processes/partitions.

Definition at line 122 of file AssemblyCommDG.h.

Referenced by PerformExchange().