Nektar::MultiRegions::NeighborAllToAllV Class Reference

#include <AssemblyCommDG.h>

Inheritance diagram for Nektar::MultiRegions::NeighborAllToAllV:

Public Member Functions
	NeighborAllToAllV (const LibUtilities::CommSharedPtr &comm, const std::map< int, std::vector< int > > &rankSharedEdges, const std::map< int, std::vector< int > > &edgeToTrace)
	Default constructor. More...
void	PerformExchange (const Array< OneD, NekDouble > &testFwd, Array< OneD, NekDouble > &testBwd) final
Public Member Functions inherited from Nektar::MultiRegions::ExchangeMethod
	ExchangeMethod ()=default
	Default constructor. More...
virtual	~ExchangeMethod ()=default
	Default destructor. More...
virtual void	PerformExchange (const Array< OneD, NekDouble > &testFwd, Array< OneD, NekDouble > &testBwd)=0

Private Attributes
LibUtilities::CommSharedPtr	m_comm
	Communicator. More...
Array< OneD, int >	m_sendDisp
	List of displacements. More...
std::vector< int >	m_edgeTraceIndex
	List of trace map indices of the quad points to exchange. More...
Array< OneD, int >	m_sendCount
	List of counts. More...

Detailed Description

Uses the MPI_NeighborAllToAllV collective operation to perform the exchange of quadrature values. This allows for varying exchange array sizes to minimise communication data size. Ranks only communicate with ranks with which they need to exchange data, i.e. are adjacent in the mesh or share a periodic boundary condition, this further minimises unnecessary data transfer over just reducing array sizes to 0 such as in MPI_AllToAllV. One collective operation is posted on each rank which requires communication.

Definition at line 166 of file AssemblyCommDG.h.

Constructor & Destructor Documentation

NeighborAllToAllV()

Nektar::MultiRegions::NeighborAllToAllV::NeighborAllToAllV	(	const LibUtilities::CommSharedPtr &	comm,
		const std::map< int, std::vector< int > > &	rankSharedEdges,
		const std::map< int, std::vector< int > > &	edgeToTrace
	)

Default constructor.

Definition at line 132 of file AssemblyCommDG.cpp.

    : m_comm(comm)
{
    int nNeighbours = rankSharedEdges.size();
    Array<OneD, int> destinations(nNeighbours, 0);
    Array<OneD, int> weights(nNeighbours, 0);
    int cnt = 0;
    for (auto &rankEdgeVec : rankSharedEdges)
    {
        destinations[cnt] = rankEdgeVec.first;
        weights[cnt]      = rankEdgeVec.second.size();
        ++cnt;
    }
 
    comm->DistGraphCreateAdjacent(destinations, weights, 1);
 
    // Setting up indices
    m_sendCount = Array<OneD, int>(nNeighbours, 0);
    cnt         = 0;
 
    for (const auto &rankEdgeSet : rankSharedEdges)
    {
        for (size_t i : rankEdgeSet.second)
        {
            std::vector<int> edgeIndex = edgeToTrace.at(i);
            m_edgeTraceIndex.insert(m_edgeTraceIndex.end(), edgeIndex.begin(),
                                    edgeIndex.end());
            m_sendCount[cnt] += edgeIndex.size();
        }
 
        ++cnt;
    }
 
    m_sendDisp = Array<OneD, int>(nNeighbours, 0);
    for (size_t i = 1; i < nNeighbours; ++i)
    {
        m_sendDisp[i] = m_sendDisp[i - 1] + m_sendCount[i - 1];
    }
}

References m_edgeTraceIndex, m_sendCount, and m_sendDisp.

Member Function Documentation

PerformExchange()

void Nektar::MultiRegions::NeighborAllToAllV::PerformExchange ( const Array< OneD, NekDouble > &  testFwd, Array< OneD, NekDouble > &  testBwd  )

finalvirtual

Perform MPI comm exchange taking the Fwd trace and sending partition edge trace values to the matching locations in the Bwd trace of corresponding adjacent partitions.

Parameters

[in]	testFwd	The values to send to adjacent partitions
[out]	testBwd	The values received from adjacent partitions

Implements Nektar::MultiRegions::ExchangeMethod.

Definition at line 288 of file AssemblyCommDG.cpp.

{
    Array<OneD, NekDouble> sendBuff(m_edgeTraceIndex.size(), -1);
    Array<OneD, NekDouble> recvBuff(m_edgeTraceIndex.size(), -1);
    Vmath::Gathr(int(m_edgeTraceIndex.size()), testFwd.data(),
                 m_edgeTraceIndex.data(), sendBuff.data());
 
    m_comm->NeighborAlltoAllv(sendBuff, m_sendCount, m_sendDisp, recvBuff,
                              m_sendCount, m_sendDisp);
 
    Vmath::Scatr(int(m_edgeTraceIndex.size()), recvBuff.data(),
                 m_edgeTraceIndex.data(), testBwd.data());
}

References Vmath::Gathr(), m_comm, m_edgeTraceIndex, m_sendCount, m_sendDisp, and Vmath::Scatr().

Member Data Documentation

m_comm

LibUtilities::CommSharedPtr Nektar::MultiRegions::NeighborAllToAllV::m_comm

private

Communicator.

Definition at line 181 of file AssemblyCommDG.h.

Referenced by PerformExchange().

m_edgeTraceIndex

std::vector<int> Nektar::MultiRegions::NeighborAllToAllV::m_edgeTraceIndex

private

List of trace map indices of the quad points to exchange.

Definition at line 185 of file AssemblyCommDG.h.

Referenced by NeighborAllToAllV(), and PerformExchange().

m_sendCount

Array<OneD, int> Nektar::MultiRegions::NeighborAllToAllV::m_sendCount

private

List of counts.

Definition at line 187 of file AssemblyCommDG.h.

Referenced by NeighborAllToAllV(), and PerformExchange().

m_sendDisp

Array<OneD, int> Nektar::MultiRegions::NeighborAllToAllV::m_sendDisp

private

List of displacements.

Definition at line 183 of file AssemblyCommDG.h.

Referenced by NeighborAllToAllV(), and PerformExchange().