Nektar::MultiRegions::Pairwise Class Reference

#include <AssemblyCommDG.h>

Inheritance diagram for Nektar::MultiRegions::Pairwise:

Public Member Functions
	Pairwise (const LibUtilities::CommSharedPtr &comm, const std::map< int, std::vector< int >> &rankSharedEdges, const std::map< int, std::vector< int >> &edgeToTrace)
void	PerformExchange (const Array< OneD, NekDouble > &testFwd, Array< OneD, NekDouble > &testBwd) final
Public Member Functions inherited from Nektar::MultiRegions::ExchangeMethod
	ExchangeMethod ()=default
	Default constructor. More...
virtual	~ExchangeMethod ()=default
	Default destructor. More...

Private Attributes
LibUtilities::CommSharedPtr	m_comm
	Communicator. More...
std::vector< std::pair< int, std::vector< int > > >	m_vecPairPartitionTrace
	List of partition to trace map indices of the quad points to exchange. More...
Array< OneD, int >	m_sendDisp
	List of displacements. More...
Array< OneD, NekDouble >	m_recvBuff
	Receive buffer for exchange. More...
Array< OneD, NekDouble >	m_sendBuff
	Send buffer for exchange. More...
LibUtilities::CommRequestSharedPtr	m_recvRequest
	List of receive requests. More...
LibUtilities::CommRequestSharedPtr	m_sendRequest
	List of send requests. More...

Detailed Description

Uses persistent MPI_Irecv and MPI_Isend operations to perform the exchange of quadrature values. This allows for varying exchange array sizes to minimise communication data size. Ranks only communicate with ranks with which they need to exchange data, i.e. are adjacent in the mesh or share a periodic boundary condition. On each rank there are 'n' receives and 'n' sends posted where 'n' is the number of other ranks with which communication is needed. We use persistent communication methods to reduce overhead.

Definition at line 202 of file AssemblyCommDG.h.

Constructor & Destructor Documentation

Pairwise()

Nektar::MultiRegions::Pairwise::Pairwise	(	const LibUtilities::CommSharedPtr &	comm,
		const std::map< int, std::vector< int >> &	rankSharedEdges,
		const std::map< int, std::vector< int >> &	edgeToTrace
	)

Definition at line 177 of file AssemblyCommDG.cpp.

    : m_comm(comm)
{
    int cnt         = 0;
    int nNeighbours = rankSharedEdges.size();
    Array<OneD, int> sendCount(nNeighbours, -1);
 
    for (const auto &rankEdgeSet : rankSharedEdges)
    {
        std::vector<int> edgeTraceIndex;
        for (size_t i : rankEdgeSet.second)
        {
            std::vector<int> edgeIndex = edgeToTrace.at(i);
            edgeTraceIndex.insert(edgeTraceIndex.end(), edgeIndex.begin(),
                                  edgeIndex.end());
        }
 
        m_vecPairPartitionTrace.emplace_back(
            std::make_pair(rankEdgeSet.first, edgeTraceIndex));
 
        sendCount[cnt++] = edgeTraceIndex.size();
    }
 
    m_sendDisp = Array<OneD, int>(nNeighbours, 0);
 
    for (size_t i = 1; i < nNeighbours; ++i)
    {
        m_sendDisp[i] = m_sendDisp[i - 1] + sendCount[i - 1];
    }
 
    size_t totSends = std::accumulate(sendCount.begin(), sendCount.end(), 0);
 
    m_recvBuff = Array<OneD, NekDouble>(totSends, -1);
    m_sendBuff = Array<OneD, NekDouble>(totSends, -1);
 
    m_recvRequest = m_comm->CreateRequest(m_vecPairPartitionTrace.size());
    m_sendRequest = m_comm->CreateRequest(m_vecPairPartitionTrace.size());
 
    // Construct persistent requests
    for (size_t i = 0; i < m_vecPairPartitionTrace.size(); ++i)
    {
        size_t len = m_vecPairPartitionTrace[i].second.size();
 
        // Initialise receive requests
        m_comm->RecvInit(m_vecPairPartitionTrace[i].first,
                         m_recvBuff[m_sendDisp[i]], len, m_recvRequest, i);
 
        // Initialise send requests
        m_comm->SendInit(m_vecPairPartitionTrace[i].first,
                         m_sendBuff[m_sendDisp[i]], len, m_sendRequest, i);
    }
}

References m_comm, m_recvBuff, m_recvRequest, m_sendBuff, m_sendDisp, m_sendRequest, and m_vecPairPartitionTrace.

Member Function Documentation

PerformExchange()

void Nektar::MultiRegions::Pairwise::PerformExchange ( const Array< OneD, NekDouble > &  testFwd, Array< OneD, NekDouble > &  testBwd  )

finalvirtual

Perform MPI comm exchange taking the Fwd trace and sending partition edge trace values to the matching locations in the Bwd trace of corresponding adjacent partitions.

Parameters

[in]	testFwd	The values to send to adjacent partitions
[out]	testBwd	The values received from adjacent partitions

Implements Nektar::MultiRegions::ExchangeMethod.

Definition at line 301 of file AssemblyCommDG.cpp.

{
    // Perform receive posts
    m_comm->StartAll(m_recvRequest);
 
    // Fill send buffer from Fwd trace
    for (size_t i = 0; i < m_vecPairPartitionTrace.size(); ++i)
    {
        size_t len = m_vecPairPartitionTrace[i].second.size();
        for (size_t j = 0; j < len; ++j)
        {
            m_sendBuff[m_sendDisp[i] + j] =
                testFwd[m_vecPairPartitionTrace[i].second[j]];
        }
    }
 
    // Perform send posts
    m_comm->StartAll(m_sendRequest);
 
    // Wait for all send/recvs to complete
    m_comm->WaitAll(m_sendRequest);
    m_comm->WaitAll(m_recvRequest);
 
    // Fill Bwd trace from recv buffer
    for (size_t i = 0; i < m_vecPairPartitionTrace.size(); ++i)
    {
        size_t len = m_vecPairPartitionTrace[i].second.size();
        for (size_t j = 0; j < len; ++j)
        {
            testBwd[m_vecPairPartitionTrace[i].second[j]] =
                m_recvBuff[m_sendDisp[i] + j];
        }
    }
}

References m_comm, m_recvBuff, m_recvRequest, m_sendBuff, m_sendDisp, m_sendRequest, and m_vecPairPartitionTrace.

Member Data Documentation

m_comm

LibUtilities::CommSharedPtr Nektar::MultiRegions::Pairwise::m_comm

private

Communicator.

Definition at line 216 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_recvBuff

Array<OneD, NekDouble> Nektar::MultiRegions::Pairwise::m_recvBuff

private

Receive buffer for exchange.

Definition at line 222 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_recvRequest

LibUtilities::CommRequestSharedPtr Nektar::MultiRegions::Pairwise::m_recvRequest

private

List of receive requests.

Definition at line 226 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_sendBuff

Array<OneD, NekDouble> Nektar::MultiRegions::Pairwise::m_sendBuff

private

Send buffer for exchange.

Definition at line 224 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_sendDisp

Array<OneD, int> Nektar::MultiRegions::Pairwise::m_sendDisp

private

List of displacements.

Definition at line 220 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_sendRequest

LibUtilities::CommRequestSharedPtr Nektar::MultiRegions::Pairwise::m_sendRequest

private

List of send requests.

Definition at line 228 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_vecPairPartitionTrace

std::vector<std::pair<int, std::vector<int> > > Nektar::MultiRegions::Pairwise::m_vecPairPartitionTrace

private

List of partition to trace map indices of the quad points to exchange.

Definition at line 218 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().