Nektar::MultiRegions::AllToAllV Class Reference

#include <AssemblyCommDG.h>

Inheritance diagram for Nektar::MultiRegions::AllToAllV:

Public Member Functions
	AllToAllV (const LibUtilities::CommSharedPtr &comm, const std::map< int, std::vector< int > > &rankSharedEdges, const std::map< int, std::vector< int > > &edgeToTrace, const int &nRanks)
	Default constructor. More...
void	PerformExchange (const Array< OneD, NekDouble > &testFwd, Array< OneD, NekDouble > &testBwd) final
Public Member Functions inherited from Nektar::MultiRegions::ExchangeMethod
	ExchangeMethod ()=default
	Default constructor. More...
virtual	~ExchangeMethod ()=default
	Default destructor. More...
virtual void	PerformExchange (const Array< OneD, NekDouble > &testFwd, Array< OneD, NekDouble > &testBwd)=0

Private Attributes
LibUtilities::CommSharedPtr	m_comm
	Communicator. More...
std::vector< int >	m_allVEdgeIndex
	List of trace map indices of the quad points to exchange. More...
Array< OneD, int >	m_allVSendCount
	List of counts for MPI_alltoallv. More...
Array< OneD, int >	m_allVSendDisp
	List of displacements for MPI_alltoallv. More...

Detailed Description

Uses the MPI_AllToAllV collective operation to perform the exchange of quadrature values. This allows for varying exchange array sizes to minimise communication data size. All ranks communicate to all other ranks, however the array size can be 0 to avoid unnecessary data transfer. One collective peration is posted on each rank which requires communication.

Definition at line 133 of file AssemblyCommDG.h.

Constructor & Destructor Documentation

AllToAllV()

Nektar::MultiRegions::AllToAllV::AllToAllV	(	const LibUtilities::CommSharedPtr &	comm,
		const std::map< int, std::vector< int > > &	rankSharedEdges,
		const std::map< int, std::vector< int > > &	edgeToTrace,
		const int &	nRanks
	)

Default constructor.

Definition at line 99 of file AssemblyCommDG.cpp.

    : m_comm(comm)
{
    m_allVSendCount = Array<OneD, int>(nRanks, 0);
    for (size_t i = 0; i < nRanks; ++i)
    {
        if (rankSharedEdges.find(i) != rankSharedEdges.end())
        {
            for (size_t j = 0; j < rankSharedEdges.at(i).size(); ++j)
            {
                std::vector<int> edgeIndex =
                    edgeToTrace.at(rankSharedEdges.at(i)[j]);
                m_allVEdgeIndex.insert(m_allVEdgeIndex.end(), edgeIndex.begin(),
                                       edgeIndex.end());
                m_allVSendCount[i] += edgeIndex.size();
            }
        }
        else
        {
            m_allVSendCount[i] = 0;
        }
    }
 
    m_allVSendDisp = Array<OneD, int>(nRanks, 0);
    for (size_t i = 1; i < nRanks; ++i)
    {
        m_allVSendDisp[i] = m_allVSendDisp[i - 1] + m_allVSendCount[i - 1];
    }
}

References m_allVEdgeIndex, m_allVSendCount, and m_allVSendDisp.

Member Function Documentation

PerformExchange()

void Nektar::MultiRegions::AllToAllV::PerformExchange ( const Array< OneD, NekDouble > &  testFwd, Array< OneD, NekDouble > &  testBwd  )

finalvirtual

Perform MPI comm exchange taking the Fwd trace and sending partition edge trace values to the matching locations in the Bwd trace of corresponding adjacent partitions.

Parameters

[in]	testFwd	The values to send to adjacent partitions
[out]	testBwd	The values received from adjacent partitions

Implements Nektar::MultiRegions::ExchangeMethod.

Definition at line 272 of file AssemblyCommDG.cpp.

{
    Array<OneD, NekDouble> sendBuff(m_allVEdgeIndex.size(), -1);
    Array<OneD, NekDouble> recvBuff(m_allVEdgeIndex.size(), -1);
 
    Vmath::Gathr(int(m_allVEdgeIndex.size()), testFwd.data(),
                 m_allVEdgeIndex.data(), sendBuff.data());
 
    m_comm->AlltoAllv(sendBuff, m_allVSendCount, m_allVSendDisp, recvBuff,
                      m_allVSendCount, m_allVSendDisp);
 
    Vmath::Scatr(int(m_allVEdgeIndex.size()), recvBuff.data(),
                 m_allVEdgeIndex.data(), testBwd.data());
}

References Vmath::Gathr(), m_allVEdgeIndex, m_allVSendCount, m_allVSendDisp, m_comm, and Vmath::Scatr().

Member Data Documentation

m_allVEdgeIndex

std::vector<int> Nektar::MultiRegions::AllToAllV::m_allVEdgeIndex

private

List of trace map indices of the quad points to exchange.

Definition at line 150 of file AssemblyCommDG.h.

Referenced by AllToAllV(), and PerformExchange().

m_allVSendCount

Array<OneD, int> Nektar::MultiRegions::AllToAllV::m_allVSendCount

private

List of counts for MPI_alltoallv.

Definition at line 152 of file AssemblyCommDG.h.

Referenced by AllToAllV(), and PerformExchange().

m_allVSendDisp

Array<OneD, int> Nektar::MultiRegions::AllToAllV::m_allVSendDisp

private

List of displacements for MPI_alltoallv.

Definition at line 154 of file AssemblyCommDG.h.

Referenced by AllToAllV(), and PerformExchange().

m_comm

LibUtilities::CommSharedPtr Nektar::MultiRegions::AllToAllV::m_comm

private

Communicator.

Definition at line 148 of file AssemblyCommDG.h.

Referenced by PerformExchange().