Nektar::MultiRegions::Pairwise Class Reference

#include <AssemblyCommDG.h>

Inheritance diagram for Nektar::MultiRegions::Pairwise:

Public Member Functions
	Pairwise (const LibUtilities::CommSharedPtr &comm, const std::map< int, std::vector< int > > &rankSharedEdges, const std::map< int, std::vector< int > > &edgeToTrace)
void	PerformExchange (const Array< OneD, NekDouble > &testFwd, Array< OneD, NekDouble > &testBwd) final
Public Member Functions inherited from Nektar::MultiRegions::ExchangeMethod
	ExchangeMethod ()=default
	Default constructor. More...
virtual	~ExchangeMethod ()=default
	Default destructor. More...
virtual void	PerformExchange (const Array< OneD, NekDouble > &testFwd, Array< OneD, NekDouble > &testBwd)=0

Private Attributes
LibUtilities::CommSharedPtr	m_comm
	Communicator. More...
Array< OneD, int >	m_edgeTraceIndex
	List of trace index locations in recv/send buff. More...
Array< OneD, NekDouble >	m_recvBuff
	Receive buffer for exchange. More...
Array< OneD, NekDouble >	m_sendBuff
	Send buffer for exchange. More...
LibUtilities::CommRequestSharedPtr	m_recvRequest
	List of receive requests. More...
LibUtilities::CommRequestSharedPtr	m_sendRequest
	List of send requests. More...

Detailed Description

Uses persistent MPI_Irecv and MPI_Isend operations to perform the exchange of quadrature values. This allows for varying exchange array sizes to minimise communication data size. Ranks only communicate with ranks with which they need to exchange data, i.e. are adjacent in the mesh or share a periodic boundary condition. On each rank there are 'n' receives and 'n' sends posted where 'n' is the number of other ranks with which communication is needed. We use persistent communication methods to reduce overhead.

Definition at line 199 of file AssemblyCommDG.h.

Constructor & Destructor Documentation

Pairwise()

Nektar::MultiRegions::Pairwise::Pairwise	(	const LibUtilities::CommSharedPtr &	comm,
		const std::map< int, std::vector< int > > &	rankSharedEdges,
		const std::map< int, std::vector< int > > &	edgeToTrace
	)

Definition at line 175 of file AssemblyCommDG.cpp.

    : m_comm(comm)
{
    int cnt         = 0;
    int nNeighbours = rankSharedEdges.size();
    Array<OneD, int> sendCount(nNeighbours, -1);
 
    // List of partition to trace map indices of the quad points to exchange
    std::vector<std::pair<int, std::vector<int>>> vecPairPartitionTrace;
    for (const auto &rankEdgeSet : rankSharedEdges)
    {
        std::vector<int> edgeTraceIndex;
        for (size_t i : rankEdgeSet.second)
        {
            std::vector<int> edgeIndex = edgeToTrace.at(i);
            edgeTraceIndex.insert(edgeTraceIndex.end(), edgeIndex.begin(),
                                  edgeIndex.end());
        }
 
        vecPairPartitionTrace.emplace_back(
            std::make_pair(rankEdgeSet.first, edgeTraceIndex));
 
        sendCount[cnt++] = edgeTraceIndex.size();
    }
 
    Array<OneD, int> sendDisp(nNeighbours, 0);
    for (size_t i = 1; i < nNeighbours; ++i)
    {
        sendDisp[i] = sendDisp[i - 1] + sendCount[i - 1];
    }
 
    size_t totSends = std::accumulate(sendCount.begin(), sendCount.end(), 0);
 
    m_recvBuff       = Array<OneD, NekDouble>(totSends, -1);
    m_sendBuff       = Array<OneD, NekDouble>(totSends, -1);
    m_edgeTraceIndex = Array<OneD, int>(totSends, -1);
 
    // Set up m_edgeTraceIndex to reduce complexity when performing exchange
    cnt = 0;
    for (auto &i : vecPairPartitionTrace)
    {
        for (auto j : i.second)
        {
            m_edgeTraceIndex[cnt++] = j;
        }
    }
 
    m_recvRequest = m_comm->CreateRequest(vecPairPartitionTrace.size());
    m_sendRequest = m_comm->CreateRequest(vecPairPartitionTrace.size());
 
    // Construct persistent requests
    for (size_t i = 0; i < vecPairPartitionTrace.size(); ++i)
    {
        size_t len = vecPairPartitionTrace[i].second.size();
 
        // Initialise receive requests
        m_comm->RecvInit(vecPairPartitionTrace[i].first,
                         m_recvBuff[sendDisp[i]], len, m_recvRequest, i);
 
        // Initialise send requests
        m_comm->SendInit(vecPairPartitionTrace[i].first,
                         m_sendBuff[sendDisp[i]], len, m_sendRequest, i);
    }
}

References m_comm, m_edgeTraceIndex, m_recvBuff, m_recvRequest, m_sendBuff, and m_sendRequest.

Member Function Documentation

PerformExchange()

void Nektar::MultiRegions::Pairwise::PerformExchange ( const Array< OneD, NekDouble > &  testFwd, Array< OneD, NekDouble > &  testBwd  )

finalvirtual

Perform MPI comm exchange taking the Fwd trace and sending partition edge trace values to the matching locations in the Bwd trace of corresponding adjacent partitions.

Parameters

[in]	testFwd	The values to send to adjacent partitions
[out]	testBwd	The values received from adjacent partitions

Implements Nektar::MultiRegions::ExchangeMethod.

Definition at line 303 of file AssemblyCommDG.cpp.

{
    // Perform receive posts
    m_comm->StartAll(m_recvRequest);
 
    // Fill send buffer from Fwd trace
    Vmath::Gathr(int(m_edgeTraceIndex.size()), testFwd.data(),
                 m_edgeTraceIndex.data(), m_sendBuff.data());
 
    // Perform send posts
    m_comm->StartAll(m_sendRequest);
 
    // Wait for all send/recvs to complete
    m_comm->WaitAll(m_sendRequest);
    m_comm->WaitAll(m_recvRequest);
 
    // Fill Bwd trace from recv buffer
    Vmath::Scatr(int(m_edgeTraceIndex.size()), m_recvBuff.data(),
                 m_edgeTraceIndex.data(), testBwd.data());
}

References Vmath::Gathr(), m_comm, m_edgeTraceIndex, m_recvBuff, m_recvRequest, m_sendBuff, m_sendRequest, and Vmath::Scatr().

Member Data Documentation

m_comm

LibUtilities::CommSharedPtr Nektar::MultiRegions::Pairwise::m_comm

private

Communicator.

Definition at line 213 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_edgeTraceIndex

Array<OneD, int> Nektar::MultiRegions::Pairwise::m_edgeTraceIndex

private

List of trace index locations in recv/send buff.

Definition at line 215 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_recvBuff

Array<OneD, NekDouble> Nektar::MultiRegions::Pairwise::m_recvBuff

private

Receive buffer for exchange.

Definition at line 217 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_recvRequest

LibUtilities::CommRequestSharedPtr Nektar::MultiRegions::Pairwise::m_recvRequest

private

List of receive requests.

Definition at line 221 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_sendBuff

Array<OneD, NekDouble> Nektar::MultiRegions::Pairwise::m_sendBuff

private

Send buffer for exchange.

Definition at line 219 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().

m_sendRequest

LibUtilities::CommRequestSharedPtr Nektar::MultiRegions::Pairwise::m_sendRequest

private

List of send requests.

Definition at line 223 of file AssemblyCommDG.h.

Referenced by Pairwise(), and PerformExchange().