Nektar::MultiRegions::InterfaceMapDG Class Reference

#include <InterfaceMapDG.h>

Public Member Functions
	~InterfaceMapDG ()=default
	Default destructor. More...
	InterfaceMapDG (const SpatialDomains::MeshGraphSharedPtr &graph, const ExpListSharedPtr &trace)
void	ExchangeTrace (Array< OneD, NekDouble > &Fwd, Array< OneD, NekDouble > &Bwd)
	Perform the trace exchange between processors, given the forwards and backwards spaces. More...
void	ExchangeCoords ()
	Perform the coordinate exchange between processors. This is where the missing coordinates on the interface are found and sent to all other processors on the other side of that interface so the matching ranks can be found. More...

Private Attributes
SpatialDomains::MeshGraphSharedPtr	m_graph
	Mesh associated with this expansion list. More...
SpatialDomains::MovementSharedPtr	m_movement
	Movement object associated with the non-conformal interfaces. More...
std::vector< InterfaceTraceSharedPtr >	m_localInterfaces
	Interface sides present on current process. More...
const ExpListSharedPtr	m_trace
	Trace expansion list. More...
std::vector< InterfaceExchangeSharedPtr >	m_exchange
	Vector of interface exchanges, i.e. every rank-to-rank comm needed. More...

Detailed Description

Implements the communication patterns to allow for exchange of information across non-conformal interfaces and across different partitions. Holds all the InterfaceExchange objects in m_exchange.

Definition at line 226 of file InterfaceMapDG.h.

Constructor & Destructor Documentation

~InterfaceMapDG()

Nektar::MultiRegions::InterfaceMapDG::~InterfaceMapDG ( )

default

Default destructor.

InterfaceMapDG()

Nektar::MultiRegions::InterfaceMapDG::InterfaceMapDG	(	const SpatialDomains::MeshGraphSharedPtr &	graph,
		const ExpListSharedPtr &	trace
	)

Sets up the InterfaceExchange objects stored in m_exchange, each object is rank -> rank and contains a vector of InterfaceTrace objects corresponding to shared interfaces between those ranks.

Definition at line 53 of file InterfaceMapDG.cpp.

    : m_graph(meshGraph), m_movement(meshGraph->GetMovement()), m_trace(trace)
{
    auto comm                = m_trace->GetComm()->GetSpaceComm();
    auto interfaceCollection = m_movement->GetInterfaces();
 
    // myIndxLR contains the info about what interface edges are present on
    // current rank with each interface no, i,  consisting of:
    // [i] = indx
    // [i + 1] = 0 (non), = 1 (left only), = 2 (right only), = 3 (both)
    std::map<int, int> myIndxLRMap;
 
    // localInterfaces contains a map of interface ID to a pair of interface
    // traces, this pair is 'left' and 'right' interface and is used to
    // construct and store the traces for interfaces present on the current rank
    std::map<int, std::pair<InterfaceTraceSharedPtr, InterfaceTraceSharedPtr>>
        localInterfaces;
 
    // Map of m_localInterfaces vector to interface ID
    Array<OneD, int> indxToInterfaceID(interfaceCollection.size());
 
    // Loops over all interfaces and check if either side, 'left' or 'right'
    // are empty on the current rank, if not empty then populate the local
    // interface data structures and keep track using myIndxLRMap
    size_t cnt = 0;
    for (const auto &interface : interfaceCollection)
    {
        indxToInterfaceID[cnt]             = interface.first.first;
        myIndxLRMap[interface.first.first] = 0;
 
        if (!interface.second->GetLeftInterface()->IsEmpty())
        {
            myIndxLRMap[interface.first.first] += 1;
 
            localInterfaces[interface.first.first].first =
                MemoryManager<InterfaceTrace>::AllocateSharedPtr(
                    trace, interface.second->GetLeftInterface());
            m_localInterfaces.emplace_back(
                localInterfaces[interface.first.first].first);
        }
 
        if (!interface.second->GetRightInterface()->IsEmpty())
        {
            myIndxLRMap[interface.first.first] += 2;
 
            localInterfaces[interface.first.first].second =
                MemoryManager<InterfaceTrace>::AllocateSharedPtr(
                    trace, interface.second->GetRightInterface());
            m_localInterfaces.emplace_back(
                localInterfaces[interface.first.first].second);
        }
 
        cnt++;
    }
 
    // Send num of interfaces size so all partitions can prepare buffers
    int nRanks = comm->GetSize();
 
    // Send all interface edges present to all partitions
    Array<OneD, int> interfaceEdges(myIndxLRMap.size());
    cnt = 0;
    for (auto pres : myIndxLRMap)
    {
        interfaceEdges[cnt++] = pres.second;
    }
    Array<OneD, int> rankLocalInterfaceIds(myIndxLRMap.size() * nRanks, 0);
    comm->AllGather(interfaceEdges, rankLocalInterfaceIds);
 
    // Map of rank to vector of interface traces
    std::map<int, std::vector<InterfaceTraceSharedPtr>> oppRankSharedInterface;
 
    // Find what interface Ids match with other ranks, then check if opposite
    // edge
    size_t myRank        = comm->GetRank();
    size_t numInterfaces = interfaceCollection.size();
    for (int i = 0; i < nRanks; ++i)
    {
        for (size_t j = 0; j < numInterfaces; ++j)
        {
            int otherId = indxToInterfaceID[j];
 
            // otherCode represents for a specific interface ID what sides are
            // present on rank i, 0=non, 1=left, 2=right, 3=both
            int otherCode = rankLocalInterfaceIds[i * numInterfaces + j];
 
            // myCode represents for a specific interface ID what sides are
            // present on this rank/process 0=non, 1=left, 2=right, 3=both
            int myCode = myIndxLRMap[otherId];
 
            // Special case if checking current rank (this process)
            if (i == myRank)
            {
                // If contains both edges locally then set check local to true
                if (myCode == 3)
                {
                    localInterfaces[otherId].first->SetCheckLocal(true);
                    localInterfaces[otherId].second->SetCheckLocal(true);
                }
 
                continue;
            }
 
            // Checks if this ranks 'left' matches any 'right' on other rank
            if ((myCode == 1 && otherCode == 2) ||
                (myCode == 1 && otherCode == 3) ||
                (myCode == 3 && otherCode == 2))
            {
                oppRankSharedInterface[i].emplace_back(
                    localInterfaces[otherId].first);
            }
            // Checks if this ranks 'right' matches any 'left' on other rank
            else if ((myCode == 2 && otherCode == 1) ||
                     (myCode == 2 && otherCode == 3) ||
                     (myCode == 3 && otherCode == 1))
            {
                oppRankSharedInterface[i].emplace_back(
                    localInterfaces[otherId].second);
            }
            // Checks if this ranks 'both' matches any 'both' on other rank
            else if (myCode == 3 && otherCode == 3)
            {
                oppRankSharedInterface[i].emplace_back(
                    localInterfaces[otherId].first);
                oppRankSharedInterface[i].emplace_back(
                    localInterfaces[otherId].second);
            }
        }
    }
 
    // Create individual interface exchange objects (each object is rank ->
    // rank) and contains a vector of interfaceTrace objects
    for (auto &rank : oppRankSharedInterface)
    {
        m_exchange.emplace_back(
            MemoryManager<InterfaceExchange>::AllocateSharedPtr(
                m_movement, m_trace, comm, rank));
    }
 
    // Find missing coordinates on interface from other side
    ExchangeCoords();
}

References Nektar::MemoryManager< DataType >::AllocateSharedPtr(), ExchangeCoords(), m_exchange, m_localInterfaces, m_movement, and m_trace.

Member Function Documentation

ExchangeCoords()

void Nektar::MultiRegions::InterfaceMapDG::ExchangeCoords

(

)

Perform the coordinate exchange between processors. This is where the missing coordinates on the interface are found and sent to all other processors on the other side of that interface so the matching ranks can be found.

Definition at line 197 of file InterfaceMapDG.cpp.

{
    LibUtilities::Timer timer;
    timer.Start();
 
    auto comm  = m_trace->GetComm();
    auto zones = m_movement->GetZones();
 
    LibUtilities::Timer timer2;
    timer2.Start();
    for (auto &interfaceTrace : m_localInterfaces)
    {
        interfaceTrace->CalcLocalMissing(m_movement);
    }
    timer2.Stop();
    timer2.AccumulateRegion("InterfaceMapDG::ExchangeCoords local", 1);
 
    // If parallel communication is needed
    if (!m_exchange.empty())
    {
        LibUtilities::Timer timer3;
        timer3.Start();
 
        auto requestSend = comm->CreateRequest(m_exchange.size());
        auto requestRecv = comm->CreateRequest(m_exchange.size());
 
        for (int i = 0; i < m_exchange.size(); ++i)
        {
            m_exchange[i]->RankFillSizes(requestSend, requestRecv, i);
        }
        comm->WaitAll(requestSend);
        comm->WaitAll(requestRecv);
 
        for (int i = 0; i < m_exchange.size(); ++i)
        {
            m_exchange[i]->SendMissing(requestSend, requestRecv, i);
        }
        comm->WaitAll(requestSend);
        comm->WaitAll(requestRecv);
 
        for (auto &i : m_exchange)
        {
            i->CalcRankDistances();
        }
 
        timer3.Stop();
        timer3.AccumulateRegion("InterfaceMapDG::ExchangeCoords parallel", 1);
    }
 
    m_movement->GetCoordExchangeFlag() = false;
 
    timer.Stop();
    timer.AccumulateRegion("InterfaceMapDG::ExchangeCoords");
}

References Nektar::LibUtilities::Timer::AccumulateRegion(), m_exchange, m_localInterfaces, m_movement, m_trace, Nektar::LibUtilities::Timer::Start(), and Nektar::LibUtilities::Timer::Stop().

Referenced by ExchangeTrace(), and InterfaceMapDG().

ExchangeTrace()

void Nektar::MultiRegions::InterfaceMapDG::ExchangeTrace	(	Array< OneD, NekDouble > &	Fwd,
		Array< OneD, NekDouble > &	Bwd
	)

Perform the trace exchange between processors, given the forwards and backwards spaces.

Parameters

Fwd	Local forwards space of the trace (which will be sent)
Bwd	Local backwards space of the trace (which will receive contributions)

Performs the trace exchange across interfaces 1) Calculate and send the Fwd trace to other ranks with pairwise comms 2) While waiting for the send/recv to complete we fill the local interfaces Bwd trace 3) Fill the remaining Bwd trace with the received trace data from other ranks

Definition at line 487 of file InterfaceMapDG.cpp.

{
    if (m_movement->GetCoordExchangeFlag())
    {
        ExchangeCoords();
    }
    auto comm = m_trace->GetComm();
    // If no parallel exchange needed we only fill the local traces
    if (m_exchange.empty())
    {
 
        // Fill local interface traces
        for (auto &m_localInterface : m_localInterfaces)
        {
            m_localInterface->FillLocalBwdTrace(Fwd, Bwd);
        }
    }
    else
    {
        auto requestSend = comm->CreateRequest(m_exchange.size());
        auto requestRecv = comm->CreateRequest(m_exchange.size());
        for (int i = 0; i < m_exchange.size(); ++i)
        {
            m_exchange[i]->SendFwdTrace(requestSend, requestRecv, i, Fwd);
        }
 
        // Fill local interface traces
        for (auto &m_localInterface : m_localInterfaces)
        {
            m_localInterface->FillLocalBwdTrace(Fwd, Bwd);
        }
 
        comm->WaitAll(requestSend);
        comm->WaitAll(requestRecv);
 
        // Fill communicated interface traces
        for (auto &i : m_exchange)
        {
            i->FillRankBwdTraceExchange(Bwd);
        }
    }
}

References ExchangeCoords(), m_exchange, m_localInterfaces, m_movement, and m_trace.

Member Data Documentation

m_exchange

std::vector<InterfaceExchangeSharedPtr> Nektar::MultiRegions::InterfaceMapDG::m_exchange

private

Vector of interface exchanges, i.e. every rank-to-rank comm needed.

Definition at line 269 of file InterfaceMapDG.h.

Referenced by ExchangeCoords(), ExchangeTrace(), and InterfaceMapDG().

m_graph

SpatialDomains::MeshGraphSharedPtr Nektar::MultiRegions::InterfaceMapDG::m_graph

private

Mesh associated with this expansion list.

Definition at line 261 of file InterfaceMapDG.h.

m_localInterfaces

std::vector<InterfaceTraceSharedPtr> Nektar::MultiRegions::InterfaceMapDG::m_localInterfaces

private

Interface sides present on current process.

Definition at line 265 of file InterfaceMapDG.h.

Referenced by ExchangeCoords(), ExchangeTrace(), and InterfaceMapDG().

m_movement

SpatialDomains::MovementSharedPtr Nektar::MultiRegions::InterfaceMapDG::m_movement

private

Movement object associated with the non-conformal interfaces.

Definition at line 263 of file InterfaceMapDG.h.

Referenced by ExchangeCoords(), ExchangeTrace(), and InterfaceMapDG().

m_trace

const ExpListSharedPtr Nektar::MultiRegions::InterfaceMapDG::m_trace

private

Trace expansion list.

Definition at line 267 of file InterfaceMapDG.h.

Referenced by ExchangeCoords(), ExchangeTrace(), and InterfaceMapDG().