BCType::ext_dir_cc (AMReX-Codes/amrex#3581)

Add a new BC type `BCType::ext_dir_cc` to address the issue in #3580. For cell conservation interpolation, the boundary location for `BCType::ext_dir` is on the domain face. Applications assuming boundary location at cell centers can now use `BCType::ext_dir_cc`.

## Checklist

The proposed changes:
- [x] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX users
- [ ] include documentation in the code and/or rst files, if appropriate

Location of ext_dir

I just looked more carefully into this. When we need to fill a fine cell from spatial interpolation and it is near the physical boundary, we may need to use the physical boundary conditions stored in coarse level ghost cells to compute slopes. The BC type bug does not affect how ghost cells outside physical boundary are filled, because it calls a user provided function that does not use the BC type we set. For Flash-X, this function is `gr_fillPhysicalBC`. What the BC type bug affects is how slopes are computed. See https://github.com/AMReX-Codes/amrex/blob/87556458d867cb9cb2f148b4bcfc920c67bc918a/Src/AmrCore/AMReX_MFInterp_C.H#L13 for an example. You may wonder whether we treat `ext_dir` and `hoextrap` differently. This is because historically BoxLib codes assumed that the boundary location of `ext_dir` and `hoextrap` is exactly on the face, not half cell outside. So this is actually a known defect for codes that assume the location is at half cell outside. Maybe we should introduce a new type called `ext_dir_cc`. So I assume some Flash-X tests have `ext_dir` BC. If so, in the interpolation function, it was treated as being at cell centered before that bug fix and now as being on the domain face.

_Originally posted by @WeiqunZhang in https://github.com/AMReX-Codes/amrex/discussions/3578#discussioncomment-7190999_

diff --git a/Docs/sphinx_documentation/source/Basics.rst b/Docs/sphinx_documentation/source/Basics.rst
index fd2d12ee0b8..1227b5f7b71 100644
--- a/Docs/sphinx_documentation/source/Basics.rst
+++ b/Docs/sphinx_documentation/source/Basics.rst
@@ -2537,12 +2537,27 @@ The basic idea behind physical boundary conditions is as follows:

        ext_dir
            "External Dirichlet". It is the user's responsibility to write a routine
-           to fill ghost cells (more details below).
+           to fill ghost cells (more details below). The boundary location
+           is on the domain face even when the data inside the domain are
+           cell-centered.
+
+       ext_dir_cc
+           "External Dirichlet". It is the user's responsibility to write a routine
+           to fill ghost cells (more details below). The boundary location
+           is at the cell center of ghost cells outside the domain.

        foextrap
            "First Order Extrapolation"
            First order extrapolation from last cell in interior.

+       hoextrap
+           "High Order Extrapolation". The boundary location is on the domain
+           face even when the data inside the domain are cell-centered.
+
+       hoextrapcc
+           "High Order Extrapolation" The boundary location is at the cell
+           center of ghost cells outside the domain.
+
        reflect_even
            Reflection from interior cells with sign
            unchanged, :math:`q(-i) = q(i)`.
diff --git a/Src/Amr/AMReX_StateDescriptor.cpp b/Src/Amr/AMReX_StateDescriptor.cpp
index 161090d1a7f..4f1e3c3fe1b 100644
--- a/Src/Amr/AMReX_StateDescriptor.cpp
+++ b/Src/Amr/AMReX_StateDescriptor.cpp
@@ -19,7 +19,7 @@ StateDescriptor::bf_thread_safety (const int* /*lo*/,const int* /*hi*/,
     if (!bf_ext_dir_threadsafe) {
         bool has_ext_dir = false;
         for (int i=0; i<2*AMREX_SPACEDIM*ng && !has_ext_dir; ++i) {
-            has_ext_dir = bc[i]==BCType::ext_dir;
+            has_ext_dir = ((bc[i]==BCType::ext_dir) || (bc[i]==BCType::ext_dir_cc));
         }
         if (has_ext_dir) { thread_safe = false; }
     }
diff --git a/Src/Base/AMReX_BCUtil.H b/Src/Base/AMReX_BCUtil.H
index dd4e814610c..3ebc53a0576 100644
--- a/Src/Base/AMReX_BCUtil.H
+++ b/Src/Base/AMReX_BCUtil.H
@@ -9,11 +9,12 @@
 namespace amrex
 {
     // This is for filling cell-centered data outside physical domain
-    // (excluding periodic boundaries).  It only fills
-    // BCType::foextrap, BCType::hoextrap, BCType::reflect_even, and
-    // BCType::reflect::odd.  It does not fill BCType::ext_dir (i.e.,
-    // external Dirichlet).  If you have BCType::ext_dir, you can
-    // copy, paste and rename this function, and then modify it.
+    // (excluding periodic boundaries).  It only fills BCType::foextrap,
+    // BCType::hoextrap, BCType::hoextrapcc, BCType::reflect_even, and
+    // BCType::reflect::odd.  It does not fill BCType::ext_dir and
+    // BCType::ext_dir_cc (i.e., external Dirichlet).  If you have
+    // BCType::ext_dir or BCType::ext_dir_cc, you can copy, paste and rename
+    // this function, and then modify it.
     void FillDomainBoundary (MultiFab& phi, const Geometry& geom, const Vector<BCRec>& bc);
 }

diff --git a/Src/Base/AMReX_BCUtil.cpp b/Src/Base/AMReX_BCUtil.cpp
index c0a645ba05c..dc5b4f4f10c 100644
--- a/Src/Base/AMReX_BCUtil.cpp
+++ b/Src/Base/AMReX_BCUtil.cpp
@@ -15,7 +15,7 @@ void dummy_cpu_fill_extdir (Box const& /*bx*/, Array4<Real> const& /*dest*/,
                             const BCRec* /*bcr*/, const int /*bcomp*/,
                             const int /*orig_comp*/)
 {
-    // do something for external Dirichlet (BCType::ext_dir) if there are
+    // do something for external Dirichlet (BCType::ext_dir or BCType::ext_dir_cc) if there are
 }

 struct dummy_gpu_fill_extdir
@@ -27,7 +27,7 @@ struct dummy_gpu_fill_extdir
                      const BCRec* /*bcr*/, const int /*bcomp*/,
                      const int /*orig_comp*/) const
         {
-            // do something for external Dirichlet (BCType::ext_dir) if there are
+            // do something for external Dirichlet (BCType::ext_dir or BCType::ext_dir_cc) if there are
         }
 };

diff --git a/Src/Base/AMReX_BC_TYPES.H b/Src/Base/AMReX_BC_TYPES.H
index 872ac1eff5d..00fadd5e6d0 100644
--- a/Src/Base/AMReX_BC_TYPES.H
+++ b/Src/Base/AMReX_BC_TYPES.H
@@ -44,10 +44,8 @@ SYMMETRY       |  Un REFLECT_ODD |  REFLECT_EVEN    | REFLECT_EVEN   |

 INT_DIR      : data taken from other grids or interpolated
-EXT_DIR      : data specified on EDGE (FACE) of bndry for the linear solvers
-               for application codes, the location of the EXT_DIR data will
-               depend on how they do reconstruction, and may be edge or
-               cell-centered.
+EXT_DIR      : data specified on EDGE (FACE)
+EXT_DIR_CC   : data specified at cell center
 HOEXTRAP     : higher order extrapolation to EDGE of bndry
 HOEXTRAPCC   : linear extrapolation to CELL of bndry
 FOEXTRAP     : first order extrapolation from last cell in interior
@@ -74,6 +72,7 @@ enum mathematicalBndryTypes : int {
     ext_dir      =  3,
     hoextrap     =  4,
     hoextrapcc   =  5,
+    ext_dir_cc   =  6,
     user_1       = 1001,
     user_2       = 1002,
     user_3       = 1003
@@ -94,6 +93,7 @@ enum mathematicalBndryTypes : int {
 #define EXT_DIR 3
 #define HOEXTRAP 4
 #define HOEXTRAPCC 5
+#define EXT_DIR_CC 6

 #define Interior 0
 #define Inflow 1
diff --git a/Src/Base/AMReX_FILCC_1D.F90 b/Src/Base/AMReX_FILCC_1D.F90
index 873f67f5941..76e90a8d7a1 100644
--- a/Src/Base/AMReX_FILCC_1D.F90
+++ b/Src/Base/AMReX_FILCC_1D.F90
@@ -14,7 +14,7 @@
 !!                        corner of q array
 !! \param bc           => array of boundary flags bc(SPACEDIM,lo:hi)
 !!
-!! NOTE: all corner as well as edge data is filled if not EXT_DIR
+!! NOTE: all corner as well as edge data is filled if not EXT_DIR/EXT_DIR_CC
 ! -----------------------------------------------------------

 #ifndef AMREX_XSDK
diff --git a/Src/Base/AMReX_FILCC_2D.F90 b/Src/Base/AMReX_FILCC_2D.F90
index 89ef77d8384..c2ca8f7ddaf 100644
--- a/Src/Base/AMReX_FILCC_2D.F90
+++ b/Src/Base/AMReX_FILCC_2D.F90
@@ -16,7 +16,7 @@
 !!                        corner of q array
 !! \param bc           => array of boundary flags bc(SPACEDIM,lo:hi)
 !!
-!! NOTE: all corner as well as edge data is filled if not EXT_DIR
+!! NOTE: all corner as well as edge data is filled if not EXT_DIR/EXT_DIR_CC
 ! -----------------------------------------------------------

 subroutine filcc(q,q_l1,q_l2,q_h1,q_h2,domlo,domhi,dx,xlo,bc)
diff --git a/Src/Base/AMReX_FILCC_3D.F90 b/Src/Base/AMReX_FILCC_3D.F90
index aa3fec74ab0..59ce83d469d 100644
--- a/Src/Base/AMReX_FILCC_3D.F90
+++ b/Src/Base/AMReX_FILCC_3D.F90
@@ -6,7 +6,7 @@
 !> This routine is intended to be a generic fill function
 !! for cell centered data.  It knows how to exrapolate,
 !! and reflect data and can be used to supplement problem
-!! specific fill functions (ie. EXT_DIR).
+!! specific fill functions (ie. EXT_DIR/EXT_DIR_CC).
 !!
 !! \param q        <=  array to fill
 !! \param q_l1,q_l2,q_l3,q_h1,q_h2,q_h3   => index extent of q array
diff --git a/Src/Base/AMReX_FilCC_C.cpp b/Src/Base/AMReX_FilCC_C.cpp
index 7cdba486957..e2d8c6129fd 100644
--- a/Src/Base/AMReX_FilCC_C.cpp
+++ b/Src/Base/AMReX_FilCC_C.cpp
@@ -41,7 +41,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (lo.x < ilo) {
            const int imin = lo.x;
            const int imax = ilo-1;
-           if (bc.lo(0) == BCType::ext_dir) {
+           if (bc.lo(0) == BCType::ext_dir || bc.lo(0) == BCType::ext_dir_cc) {
                // Do nothing.
            } else if (bc.lo(0) == BCType::foextrap) {
                for (int k = lo.z; k <= hi.z; ++k) {
@@ -88,7 +88,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
             const int imin = ihi+1;
             const int imax = hi.x;

-            if (bc.hi(0) == BCType::ext_dir) {
+            if (bc.hi(0) == BCType::ext_dir || bc.hi(0) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.hi(0) == BCType::foextrap) {
                 for (int k = lo.z; k <= hi.z; ++k) {
@@ -136,7 +136,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (lo.y < jlo) {
             const int jmin = lo.y;
             const int jmax = jlo-1;
-            if (bc.lo(1) == BCType::ext_dir) {
+            if (bc.lo(1) == BCType::ext_dir || bc.lo(1) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.lo(1) == BCType::foextrap) {
                 for (int k = lo.z; k <= hi.z; ++k) {
@@ -182,7 +182,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (hi.y > jhi) {
             const int jmin = jhi+1;
             const int jmax = hi.y;
-            if (bc.hi(1) == BCType::ext_dir) {
+            if (bc.hi(1) == BCType::ext_dir || bc.hi(1) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.hi(1) == BCType::foextrap) {
                 for (int k = lo.z; k <= hi.z; ++k) {
@@ -231,7 +231,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (lo.z < klo) {
             const int kmin = lo.z;
             const int kmax = klo-1;
-            if (bc.lo(2) == BCType::ext_dir) {
+            if (bc.lo(2) == BCType::ext_dir || bc.lo(2) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.lo(2) == BCType::foextrap) {
                 for (int k = kmin; k <= kmax; ++k) {
@@ -277,7 +277,7 @@ void fab_filcc (Box const& bx, Array4<Real> const& qn, int ncomp,
         if (hi.z > khi) {
             const int kmin = khi+1;
             const int kmax = hi.z;
-            if (bc.hi(2) == BCType::ext_dir) {
+            if (bc.hi(2) == BCType::ext_dir || bc.hi(2) == BCType::ext_dir_cc) {
                 // Do nothing.
             } else if (bc.hi(2) == BCType::foextrap) {
                 for (int k = kmin; k <= kmax; ++k) {
diff --git a/Src/Base/AMReX_bc_types.fi b/Src/Base/AMReX_bc_types.fi
index 2f3340ccfc4..d929b589b3d 100644
--- a/Src/Base/AMReX_bc_types.fi
+++ b/Src/Base/AMReX_bc_types.fi
@@ -17,3 +17,5 @@
       PARAMETER (HOEXTRAP=4)
       INTEGER HOEXTRAPCC
       PARAMETER (HOEXTRAPCC=5)
+      INTEGER EXT_DIR_CC
+      PARAMETER (EXT_DIR_CC=6)
diff --git a/Src/Base/AMReX_bc_types_mod.F90 b/Src/Base/AMReX_bc_types_mod.F90
index c1c6f237ba8..5b8c58c255a 100644
--- a/Src/Base/AMReX_bc_types_mod.F90
+++ b/Src/Base/AMReX_bc_types_mod.F90
@@ -15,6 +15,7 @@ module amrex_bc_types_module
   integer, parameter, public :: amrex_bc_ext_dir      =  3
   integer, parameter, public :: amrex_bc_hoextrap     =  4
   integer, parameter, public :: amrex_bc_hoextrapcc   =  5
+  integer, parameter, public :: amrex_bc_ext_dir_cc   =  6
   integer, parameter, public :: amrex_bc_user_1       =  1001
   integer, parameter, public :: amrex_bc_user_2       =  1002
   integer, parameter, public :: amrex_bc_user_3       =  1003
diff --git a/Src/Base/AMReX_filcc_mod.F90 b/Src/Base/AMReX_filcc_mod.F90
index c1a2b2951f4..1cca9b84090 100644
--- a/Src/Base/AMReX_filcc_mod.F90
+++ b/Src/Base/AMReX_filcc_mod.F90
@@ -165,7 +165,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           imin = lo(1)
           imax = ilo-1

-          if (bc(1,1,n) .eq. amrex_bc_ext_dir) then
+          if (bc(1,1,n) .eq. amrex_bc_ext_dir .or. bc(1,1,n) .eq. amrex_bc_ext_dir_cc) then

              ! Do nothing.

@@ -237,7 +237,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           imin = ihi+1
           imax = hi(1)

-          if (bc(1,2,n) .eq. amrex_bc_ext_dir) then
+          if (bc(1,2,n) .eq. amrex_bc_ext_dir .or. bc(1,2,n) .eq. amrex_bc_ext_dir_cc) then

              ! Do nothing.

@@ -311,7 +311,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           jmin = lo(2)
           jmax = jlo-1

-          if (bc(2,1,n) .eq. amrex_bc_ext_dir) then
+          if (bc(2,1,n) .eq. amrex_bc_ext_dir .or. bc(2,1,n) .eq. amrex_bc_ext_dir_cc) then

              ! Do nothing.

@@ -383,7 +383,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           jmin = jhi+1
           jmax = hi(2)

-          if (bc(2,2,n) .eq. amrex_bc_ext_dir) then
+          if (bc(2,2,n) .eq. amrex_bc_ext_dir .or. bc(2,2,n) .eq. amrex_bc_ext_dir_cc) then

              ! Do nothing.

@@ -461,7 +461,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           kmin = lo(3)
           kmax = klo-1

-          if (bc(3,1,n) .eq. amrex_bc_ext_dir) then
+          if (bc(3,1,n) .eq. amrex_bc_ext_dir .or. bc(3,1,n) .eq. amrex_bc_ext_dir_cc) then

              ! Do nothing.

@@ -533,7 +533,7 @@ subroutine amrex_filccn(lo, hi, q, q_lo, q_hi, ncomp, domlo, domhi, dx, xlo, bc)
           kmin = khi+1
           kmax = hi(3)

-          if (bc(3,2,n) .eq. amrex_bc_ext_dir) then
+          if (bc(3,2,n) .eq. amrex_bc_ext_dir .or. bc(3,2,n) .eq. amrex_bc_ext_dir_cc) then

              ! Do nothing.

diff --git a/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90 b/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90
index f337de6e7ff..c79f2aa7b9a 100644
--- a/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90
+++ b/Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90
@@ -90,7 +90,7 @@ subroutine fill_physbc (pmf, scomp, ncomp, time, pgeom) bind(c)
                   geom%get_physical_location(plo), & ! physical location of lower left corner
                   lo_bc, hi_bc)                      ! bc types for each component

-             ! amrex_filcc doesn't fill EXT_DIR (see amrex_bc_types_module for a list of bc types
+             ! amrex_filcc doesn't fill EXT_DIR/EXT_DIR_CC (see amrex_bc_types_module for a list of bc types
              ! In that case, the user needs to fill it.
           end if
        end do
diff --git a/Tests/GPU/CNS/Source/CNS_bcfill.cpp b/Tests/GPU/CNS/Source/CNS_bcfill.cpp
index 1c787e29a96..a75c7ebce88 100644
--- a/Tests/GPU/CNS/Source/CNS_bcfill.cpp
+++ b/Tests/GPU/CNS/Source/CNS_bcfill.cpp
@@ -14,7 +14,7 @@ struct CnsFillExtDir
                      const BCRec* /*bcr*/, const int /*bcomp*/,
                      const int /*orig_comp*/) const
         {
-            // do something for external Dirichlet (BCType::ext_dir)
+            // do something for external Dirichlet (BCType::ext_dir/BCType::ext_dir_cc)
         }
 };


Pure SoA Particle: Separate Array for IdCPU (AMReX-Codes/amrex#3585)

## Summary

This addresses a regression we see when moving to pure SoA particles:
- slightly slower read/write to Ids when needed, e.g., for sorting
- issues going up to the full 64bit range

## Additional background

Once finished, this will close #3569.

## Checklist

The proposed changes:
- [ ] fix a bug or incorrect behavior in AMReX
- [x] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX users
- [ ] include documentation in the code and/or rst files, if appropriate

Pure SoA Particles: cpuid as Separate Array

For pure SoA particles, we noticed slight regressions in kernels that depend on ID operatins, e.g., for sorting, etc.
There reason for that lies in the now required double-load of two int32 arrays (cpu and ids) instead of a single load of a int64 array (cpuid) in AoS.

As a discussed solution to this with @atmyers, we should add a third default array to SoA particles:
- 64bit cpuid (new)
- Real (default: SPACEDIM members for positions)
- int (default: no members)

cc @AlexanderSinn

diff --git a/Src/Particle/AMReX_ParticleContainerI.H b/Src/Particle/AMReX_ParticleContainerI.H
index b845d130d5c..9dd1d39b953 100644
--- a/Src/Particle/AMReX_ParticleContainerI.H
+++ b/Src/Particle/AMReX_ParticleContainerI.H
@@ -18,10 +18,10 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
         if (h_redistribute_int_comp[i]) {++num_int_comm_comps;}
     }

-    if constexpr(!ParticleType::is_soa_particle) {
-        particle_size = sizeof(ParticleType);
+    if constexpr (ParticleType::is_soa_particle) {
+        particle_size = sizeof(uint64_t);  // idcpu
     } else {
-        particle_size = 0;
+        particle_size = sizeof(ParticleType);
     }
     superparticle_size = particle_size +
         num_real_comm_comps*sizeof(ParticleReal) + num_int_comm_comps*sizeof(int);
@@ -1095,7 +1095,7 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
     const size_t np_total = np + ptile.numNeighborParticles();

     if (memEfficientSort) {
-        if constexpr(!ParticleType::is_soa_particle) {
+        if constexpr (!ParticleType::is_soa_particle) {
             static_assert(sizeof(ParticleType)%4 == 0 && sizeof(uint32_t) == 4);
             using tmp_t = std::conditional_t<sizeof(ParticleType)%8 == 0,
                                              uint64_t, uint32_t>;
@@ -1530,7 +1530,7 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
             unsigned npart = ptile_ptrs[pmap_it]->numParticles();
             ParticleLocData pld;

-            if constexpr(!ParticleType::is_soa_particle){
+            if constexpr (!ParticleType::is_soa_particle){

                 if (npart != 0) {
                     Long last = npart - 1;
@@ -1647,7 +1647,7 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                     }
                 }

-            } else{ // soa particle
+            } else { // soa particle

                 auto particle_tile = ptile_ptrs[pmap_it];
                 if (npart != 0) {
@@ -1663,6 +1663,7 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                         }

                         if (p.id() < 0){
+                            soa.GetIdCPUData()[pindex] = soa.GetIdCPUData()[last];
                             for (int comp = 0; comp < NumRealComps(); comp++) {
                                 soa.GetRealData(comp)[pindex] = soa.GetRealData(comp)[last];
                             }
@@ -1679,6 +1680,7 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                         particlePostLocate(p, pld, lev);

                         if (p.id() < 0) {
+                            soa.GetIdCPUData()[pindex] = soa.GetIdCPUData()[last];
                             for (int comp = 0; comp < NumRealComps(); comp++) {
                                 soa.GetRealData(comp)[pindex] = soa.GetRealData(comp)[last];
                             }
@@ -1696,6 +1698,10 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                                 // We own it but must shift it to another place.
                                 auto index = std::make_pair(pld.m_grid, pld.m_tile);
                                 AMREX_ASSERT(soa_local[pld.m_lev][index].size() == num_threads);
+                                {
+                                    auto& arr = soa_local[pld.m_lev][index][thread_num].GetIdCPUData();
+                                    arr.push_back(soa.GetIdCPUData()[pindex]);
+                                }
                                 for (int comp = 0; comp < NumRealComps(); ++comp) {
                                     RealVector& arr = soa_local[pld.m_lev][index][thread_num].GetRealData(comp);
                                     arr.push_back(soa.GetRealData(comp)[pindex]);
@@ -1715,6 +1721,10 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                             particles_to_send.resize(new_size);

                             char* dst = &particles_to_send[old_size];
+                            {
+                                std::memcpy(dst, &soa.GetIdCPUData()[pindex], sizeof(uint64_t));
+                                dst += sizeof(uint64_t);
+                            }
                             int array_comp_start = AMREX_SPACEDIM + NStructReal;
                             for (int comp = 0; comp < NumRealComps(); comp++) {
                                 if (h_redistribute_real_comp[array_comp_start + comp]) {
@@ -1733,6 +1743,7 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                         }

                         if (p.id() < 0){
+                            soa.GetIdCPUData()[pindex] = soa.GetIdCPUData()[last];
                             for (int comp = 0; comp < NumRealComps(); comp++) {
                                 soa.GetRealData(comp)[pindex] = soa.GetRealData(comp)[last];
                             }
@@ -1747,6 +1758,10 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                         ++pindex;
                     }

+                    {
+                        auto& iddata = soa.GetIdCPUData();
+                        iddata.erase(iddata.begin() + last + 1, iddata.begin() + npart);
+                    }
                     for (int comp = 0; comp < NumRealComps(); comp++) {
                         RealVector& rdata = soa.GetRealData(comp);
                         rdata.erase(rdata.begin() + last + 1, rdata.begin() + npart);
@@ -1828,6 +1843,12 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                 auto& soa = ptile.GetStructOfArrays();
                 auto& soa_tmp = soa_local[lev][index];
                 for (int i = 0; i < num_threads; ++i) {
+                    {
+                        auto& arr = soa.GetIdCPUData();
+                        auto& tmp = soa_tmp[i].GetIdCPUData();
+                        arr.insert(arr.end(), tmp.begin(), tmp.end());
+                        tmp.erase(tmp.begin(), tmp.end());
+                    }
                     for (int comp = 0; comp < NumRealComps(); ++comp) {
                         RealVector& arr = soa.GetRealData(comp);
                         RealVector& tmp = soa_tmp[i].GetRealData(comp);
@@ -2045,20 +2066,16 @@ RedistributeMPI (std::map<int, Vector<char> >& not_ours,

                 Particle<NStructReal, NStructInt> p;

-                if constexpr (!ParticleType::is_soa_particle) {
-                   std::memcpy(&p, pbuf, sizeof(ParticleType));
-                } else {
+                if constexpr (ParticleType::is_soa_particle) {
+                    std::memcpy(&p.m_idcpu, pbuf, sizeof(uint64_t));
+
                     ParticleReal pos[AMREX_SPACEDIM];
-                    std::memcpy(&pos[0], pbuf, AMREX_SPACEDIM*sizeof(ParticleReal));
+                    std::memcpy(&pos[0], pbuf + sizeof(uint64_t), AMREX_SPACEDIM*sizeof(ParticleReal));
                     AMREX_D_TERM(p.pos(0) = pos[0];,
                                  p.pos(1) = pos[1];,
                                  p.pos(2) = pos[2]);
-
-                    int idcpu[2];
-                    std::memcpy(&idcpu[0], pbuf + NumRealComps()*sizeof(ParticleReal), 2*sizeof(int));
-
-                    p.id() = idcpu[0];
-                    p.cpu() = idcpu[1];
+                } else {
+                   std::memcpy(&p, pbuf, sizeof(ParticleType));
                 }

                 bool success = Where(p, pld, lev_min, lev_max, 0);
@@ -2097,7 +2114,12 @@ RedistributeMPI (std::map<int, Vector<char> >& not_ours,
                                                                           rcv_tile[ipart])];
                 char* pbuf = ((char*) &recvdata[offset]) + j*superparticle_size;

-                if constexpr(! ParticleType::is_soa_particle) {
+                if constexpr (ParticleType::is_soa_particle) {
+                    uint64_t idcpudata;
+                    std::memcpy(&idcpudata, pbuf, sizeof(uint64_t));
+                    pbuf += sizeof(uint64_t);
+                    ptile.GetStructOfArrays().GetIdCPUData().push_back(idcpudata);
+                } else {
                     ParticleType p;
                     std::memcpy(&p, pbuf, sizeof(ParticleType));
                     pbuf += sizeof(ParticleType);
@@ -2146,6 +2168,10 @@ RedistributeMPI (std::map<int, Vector<char> >& not_ours,
         host_int_attribs.reserve(15);
         host_int_attribs.resize(finestLevel()+1);

+        Vector<std::map<std::pair<int, int>, Gpu::HostVector<uint64_t> > > host_idcpu;
+        host_idcpu.reserve(15);
+        host_idcpu.resize(finestLevel()+1);
+
         ipart = 0;
         for (int i = 0; i < nrcvs; ++i)
         {
@@ -2159,7 +2185,15 @@ RedistributeMPI (std::map<int, Vector<char> >& not_ours,

                 char* pbuf = ((char*) &recvdata[offset]) + j*superparticle_size;

-                if constexpr(! ParticleType::is_soa_particle) {
+                host_real_attribs[lev][ind].resize(NumRealComps());
+                host_int_attribs[lev][ind].resize(NumIntComps());
+
+                if constexpr (ParticleType::is_soa_particle) {
+                    uint64_t idcpudata;
+                    std::memcpy(&idcpudata, pbuf, sizeof(uint64_t));
+                    pbuf += sizeof(uint64_t);
+                    host_idcpu[lev][ind].push_back(idcpudata);
+                } else {
                     ParticleType p;
                     std::memcpy(&p, pbuf, sizeof(ParticleType));
                     pbuf += sizeof(ParticleType);
@@ -2210,7 +2244,12 @@ RedistributeMPI (std::map<int, Vector<char> >& not_ours,
               auto new_size = old_size + src_tile.size();
               dst_tile.resize(new_size);

-              if constexpr(! ParticleType::is_soa_particle) {
+              if constexpr (ParticleType::is_soa_particle) {
+                  Gpu::copyAsync(Gpu::hostToDevice,
+                                 host_idcpu[host_lev][std::make_pair(grid,tile)].begin(),
+                                 host_idcpu[host_lev][std::make_pair(grid,tile)].end(),
+                                 dst_tile.GetStructOfArrays().GetIdCPUData().begin() + old_size);
+              } else {
                   Gpu::copyAsync(Gpu::hostToDevice,
                                  src_tile.begin(), src_tile.end(),
                                  dst_tile.GetArrayOfStructs().begin() + old_size);
diff --git a/Src/Particle/AMReX_ParticleIO.H b/Src/Particle/AMReX_ParticleIO.H
index ea24b1e5f81..8b2c7389c89 100644
--- a/Src/Particle/AMReX_ParticleIO.H
+++ b/Src/Particle/AMReX_ParticleIO.H
@@ -964,6 +964,10 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
     host_int_attribs.reserve(15);
     host_int_attribs.resize(finest_level_in_file+1);

+    Vector<std::map<std::pair<int, int>, Gpu::HostVector<uint64_t> > > host_idcpu;
+    host_idcpu.reserve(15);
+    host_idcpu.resize(finestLevel()+1);
+
     for (int i = 0; i < cnt; i++) {
         // note: for pure SoA particle layouts, we do write the id, cpu and positions as a struct
         //       for backwards compatibility with readers
@@ -1031,8 +1035,7 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                 host_real_attribs[pld.m_lev][ind][j].push_back(ptemp.pos(j));
             }

-            host_int_attribs[pld.m_lev][ind][0].push_back(ptemp.id());
-            host_int_attribs[pld.m_lev][ind][1].push_back(ptemp.cpu());
+            host_idcpu[pld.m_lev][ind].push_back(ptemp.m_idcpu);

             // read all other SoA
             // add the real...
@@ -1042,7 +1045,7 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
             }

             // ... and int array data
-            for (int icomp = 2; icomp < NumIntComps(); icomp++) {
+            for (int icomp = 0; icomp < NumIntComps(); icomp++) {
                 host_int_attribs[lev][ind][icomp].push_back(*iptr);
                 ++iptr;
             }
@@ -1071,6 +1074,11 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
             {
                 Gpu::copyAsync(Gpu::hostToDevice, src_tile.begin(), src_tile.end(),
                                dst_tile.GetArrayOfStructs().begin() + old_size);
+            } else {
+                Gpu::copyAsync(Gpu::hostToDevice,
+                               host_idcpu[host_lev][std::make_pair(grid,tile)].begin(),
+                               host_idcpu[host_lev][std::make_pair(grid,tile)].end(),
+                               dst_tile.GetStructOfArrays().GetIdCPUData().begin() + old_size);
             }

             for (int i = 0; i < NumRealComps(); ++i) { // NOLINT(readability-misleading-indentation)
diff --git a/Src/Particle/AMReX_ParticleInit.H b/Src/Particle/AMReX_ParticleInit.H
index d2568d5d70e..75316dec746 100644
--- a/Src/Particle/AMReX_ParticleInit.H
+++ b/Src/Particle/AMReX_ParticleInit.H
@@ -1062,6 +1062,10 @@ InitRandom (Long                    icount,
         host_int_attribs.reserve(15);
         host_int_attribs.resize(finestLevel()+1);

+        Vector<std::map<std::pair<int, int>, Gpu::HostVector<uint64_t> > > host_idcpu;
+        host_idcpu.reserve(15);
+        host_idcpu.resize(finestLevel()+1);
+
         for (Long j = 0; j < icount; j++)
         {
             Particle<0, 0> ptest;
@@ -1117,8 +1121,9 @@ InitRandom (Long                    icount,
                         host_real_attribs[pld.m_lev][ind][i].push_back(pos[j*AMREX_SPACEDIM+i]);
                     }

-                    host_int_attribs[pld.m_lev][ind][0].push_back(ParticleType::NextID());
-                    host_int_attribs[pld.m_lev][ind][1].push_back(MyProc);
+                    host_idcpu[pld.m_lev][ind].push_back(0);
+                    ParticleIDWrapper(host_idcpu[pld.m_lev][ind].back()) = ParticleType::NextID();
+                    ParticleCPUWrapper(host_idcpu[pld.m_lev][ind].back()) = ParallelDescriptor::MyProc();

                     host_particles[pld.m_lev][ind];

@@ -1157,6 +1162,11 @@ InitRandom (Long                    icount,
                 {
                     Gpu::copyAsync(Gpu::hostToDevice, src_tile.begin(), src_tile.end(),
                                    dst_tile.GetArrayOfStructs().begin() + old_size);
+                } else {
+                    Gpu::copyAsync(Gpu::hostToDevice,
+                                   host_idcpu[host_lev][std::make_pair(grid,tile)].begin(),
+                                   host_idcpu[host_lev][std::make_pair(grid,tile)].end(),
+                                   dst_tile.GetStructOfArrays().GetIdCPUData().begin() + old_size);
                 }

                 for (int i = 0; i < NArrayReal; ++i) { // NOLINT(readability-misleading-indentation)
@@ -1201,6 +1211,10 @@ InitRandom (Long                    icount,
         host_int_attribs.reserve(15);
         host_int_attribs.resize(finestLevel()+1);

+        Vector<std::map<std::pair<int, int>, Gpu::HostVector<uint64_t> > > host_idcpu;
+        host_idcpu.reserve(15);
+        host_idcpu.resize(finestLevel()+1);
+
         for (Long icnt = 0; icnt < M; icnt++) {
             Particle<0, 0> ptest;
             for (int i = 0; i < AMREX_SPACEDIM; i++) {
@@ -1261,8 +1275,9 @@ InitRandom (Long                    icount,
                     host_real_attribs[pld.m_lev][ind][i].push_back(ptest.pos(i));
                 }

-                host_int_attribs[pld.m_lev][ind][0].push_back(ptest.id());
-                host_int_attribs[pld.m_lev][ind][1].push_back(ptest.cpu());
+                host_idcpu[pld.m_lev][ind].push_back(0);
+                ParticleIDWrapper(host_idcpu[pld.m_lev][ind].back()) = ParticleType::NextID();
+                ParticleCPUWrapper(host_idcpu[pld.m_lev][ind].back()) = ParallelDescriptor::MyProc();

                 host_particles[pld.m_lev][ind];

@@ -1300,6 +1315,11 @@ InitRandom (Long                    icount,
                 {
                     Gpu::copyAsync(Gpu::hostToDevice, src_tile.begin(), src_tile.end(),
                                    dst_tile.GetArrayOfStructs().begin() + old_size);
+                } else {
+                    Gpu::copyAsync(Gpu::hostToDevice,
+                                   host_idcpu[host_lev][std::make_pair(grid,tile)].begin(),
+                                   host_idcpu[host_lev][std::make_pair(grid,tile)].end(),
+                                   dst_tile.GetStructOfArrays().GetIdCPUData().begin() + old_size);
                 }

                 for (int i = 0; i < NArrayReal; ++i) { // NOLINT(readability-misleading-indentation)
diff --git a/Src/Particle/AMReX_ParticleTile.H b/Src/Particle/AMReX_ParticleTile.H
index e35af847ec4..2b60b37d304 100644
--- a/Src/Particle/AMReX_ParticleTile.H
+++ b/Src/Particle/AMReX_ParticleTile.H
@@ -43,6 +43,7 @@ struct ParticleTileData

     ParticleType* AMREX_RESTRICT m_aos;

+    uint64_t* m_idcpu;
     GpuArray<ParticleReal*, NAR> m_rdata;
     GpuArray<int*, NAI> m_idata;

@@ -67,7 +68,7 @@ struct ParticleTileData
         if constexpr(!ParticleType::is_soa_particle) {
             return this->m_aos[index].id();
         } else {
-            return this->m_idata[0][index];
+            return ParticleIDWrapper(this->m_idcpu[index]);
         }
     }

@@ -77,7 +78,17 @@ struct ParticleTileData
         if constexpr(!ParticleType::is_soa_particle) {
             return this->m_aos[index].cpu();
         } else {
-            return this->m_idata[1][index];
+            return ParticleCPUWrapper(this->m_idcpu[index]);
+        }
+    }
+
+    [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
+    decltype(auto) idcpu (const int index) const &
+    {
+        if constexpr(ParticleType::is_soa_particle) {
+            return this->m_idcpu[index];
+        } else {
+            amrex::Abort("not implemented");
         }
     }

@@ -112,6 +123,9 @@ struct ParticleTileData
         if constexpr (!ParticleType::is_soa_particle) {
             memcpy(dst, m_aos + src_index, sizeof(ParticleType));
             dst += sizeof(ParticleType);
+        } else {
+            memcpy(dst, m_idcpu + src_index, sizeof(uint64_t));
+            dst += sizeof(uint64_t);
         }
         int array_start_index  = AMREX_SPACEDIM + NStructReal;
         for (int i = 0; i < NAR; ++i)
@@ -160,6 +174,9 @@ struct ParticleTileData
         if constexpr (!ParticleType::is_soa_particle) {
             memcpy(m_aos + dst_index, src, sizeof(ParticleType));
             src += sizeof(ParticleType);
+        } else {
+            memcpy(m_idcpu + dst_index, src, sizeof(uint64_t));
+            src += sizeof(uint64_t);
         }
         int array_start_index  = AMREX_SPACEDIM + NStructReal;
         for (int i = 0; i < NAR; ++i)
@@ -231,9 +248,8 @@ struct ParticleTileData
     {
         AMREX_ASSERT(index < m_size);
         SuperParticleType sp;
+        sp.m_idcpu = m_idcpu[index];
         for (int i = 0; i < AMREX_SPACEDIM; ++i) {sp.pos(i) = m_rdata[i][index];}
-        sp.id()  = m_idata[0][index];
-        sp.cpu() = m_idata[1][index];
         for (int i = 0; i < NAR; ++i) {
             sp.rdata(i) = m_rdata[i][index];
         }
@@ -270,6 +286,7 @@ struct ParticleTileData
     AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
     void setSuperParticle (const SuperParticleType& sp, int index) const noexcept
     {
+        m_idcpu[index] = sp.m_idcpu;
         for (int i = 0; i < NAR; ++i) {
             m_rdata[i][index] = sp.rdata(i);
         }
@@ -303,10 +320,10 @@ struct ConstSoAParticle : SoAParticleBase
     //functions to get id and cpu in the SOA data

     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
-    int cpu () const { return this->m_constparticle_tile_data.m_idata[1][m_index]; }
+    ConstParticleCPUWrapper cpu () const { return this->m_constparticle_tile_data.m_idcpu[m_index]; }

     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
-    int id () const { return this->m_constparticle_tile_data.m_idata[0][m_index]; }
+    ConstParticleIDWrapper id () const { return this->m_constparticle_tile_data.m_idcpu[m_index]; }

     //functions to get positions of the particle in the SOA data

@@ -366,16 +383,22 @@ struct SoAParticle : SoAParticleBase
     //functions to get id and cpu in the SOA data

     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
-    int& cpu () & { return this->m_particle_tile_data.m_idata[1][m_index]; }
+    ParticleCPUWrapper cpu () & { return this->m_particle_tile_data.m_idcpu[m_index]; }
+
+    [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
+    ParticleIDWrapper id () & { return this->m_particle_tile_data.m_idcpu[m_index]; }

     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
-    int& id () & { return this->m_particle_tile_data.m_idata[0][m_index]; }
+    uint64_t& idcpu () & { return this->m_particle_tile_data.m_idcpu[m_index]; }

     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
-    const int& cpu () const & { return this->m_particle_tile_data.m_idata[1][m_index]; }
+    ConstParticleCPUWrapper cpu () const & { return this->m_particle_tile_data.m_idcpu[m_index]; }

     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
-    const int& id () const & { return this->m_particle_tile_data.m_idata[0][m_index]; }
+    ConstParticleIDWrapper id () const & { return this->m_particle_tile_data.m_idcpu[m_index]; }
+
+    [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
+    const uint64_t& idcpu () const & { return this->m_particle_tile_data.m_idcpu[m_index]; }

     //functions to get positions of the particle in the SOA data

@@ -477,6 +500,7 @@ struct ConstParticleTileData
     Long m_size;
     const ParticleType* AMREX_RESTRICT m_aos;

+    const uint64_t* m_idcpu;
     GpuArray<const ParticleReal*, NArrayReal> m_rdata;
     GpuArray<const int*, NArrayInt > m_idata;

@@ -496,7 +520,7 @@ struct ConstParticleTileData
         if constexpr(!ParticleType::is_soa_particle) {
             return this->m_aos[index].id();
         } else {
-            return this->m_idata[0][index];
+            return ConstParticleIDWrapper(this->m_idcpu[index]);
         }
     }

@@ -506,7 +530,17 @@ struct ConstParticleTileData
         if constexpr(!ParticleType::is_soa_particle) {
             return this->m_aos[index].cpu();
         } else {
-            return this->m_idata[1][index];
+            return ConstParticleCPUWrapper(this->m_idcpu[index]);
+        }
+    }
+
+    [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
+    decltype(auto) idcpu (const int index) const &
+    {
+        if constexpr(ParticleType::is_soa_particle) {
+            return this->m_idcpu[index];
+        } else {
+            amrex::Abort("not implemented");
         }
     }

@@ -546,6 +580,9 @@ struct ConstParticleTileData
         if constexpr (!ParticleType::is_soa_particle) {
             memcpy(dst, m_aos + src_index, sizeof(ParticleType));
             dst += sizeof(ParticleType);
+        } else {
+            memcpy(dst, m_idcpu + src_index, sizeof(uint64_t));
+            dst += sizeof(uint64_t);
         }
         int array_start_index  = AMREX_SPACEDIM + NStructReal;
         for (int i = 0; i < NArrayReal; ++i)
@@ -622,8 +659,7 @@ struct ConstParticleTileData
         AMREX_ASSERT(index < m_size);
         SuperParticleType sp;
         for (int i = 0; i < AMREX_SPACEDIM; ++i) {sp.pos(i) = m_rdata[i][index];}
-        sp.id()  = m_idata[0][index];
-        sp.cpu() = m_idata[1][index];
+        sp.m_idcpu = m_idcpu[index];
         for (int i = 0; i < NAR; ++i) {
             sp.rdata(i) = m_rdata[i][index];
         }
@@ -663,7 +699,10 @@ struct ParticleTile
         ArrayOfStructs<ParticleType, Allocator>>::type;
     //using ParticleVector = typename AoS::ParticleVector;

-    using SoA = StructOfArrays<NArrayReal, NArrayInt, Allocator>;
+    using SoA = typename std::conditional<
+        ParticleType::is_soa_particle,
+        StructOfArrays<NArrayReal, NArrayInt, Allocator, true>,
+        StructOfArrays<NArrayReal, NArrayInt, Allocator, false>>::type;
     using RealVector = typename SoA::RealVector;
     using IntVector = typename SoA::IntVector;
     using StorageParticleType = typename ParticleType::StorageParticleType;
@@ -688,7 +727,7 @@ struct ParticleTile
         if constexpr (!ParticleType::is_soa_particle) {
             return m_aos_tile[index].id();
         } else {
-            return m_soa_tile.GetIntData(0)[index];
+            return ParticleIDWrapper(m_soa_tile.GetIdCPUData()[index]);
         }
     }

@@ -697,7 +736,7 @@ struct ParticleTile
         if constexpr (!ParticleType::is_soa_particle) {
             return m_aos_tile[index].id();
         } else {
-            return m_soa_tile.GetIntData(0)[index];
+            return ConstParticleIDWrapper(m_soa_tile.GetIdCPUData()[index]);
         }
     }

@@ -706,7 +745,7 @@ struct ParticleTile
         if constexpr (!ParticleType::is_soa_particle) {
             return m_aos_tile[index].cpu();
         } else {
-            return m_soa_tile.GetIntData(1)[index];
+            return ParticleCPUWrapper(m_soa_tile.GetIdCPUData()[index]);
         }
     }

@@ -715,7 +754,7 @@ struct ParticleTile
         if constexpr (!ParticleType::is_soa_particle) {
             return m_aos_tile[index].cpu();
         } else {
-            return m_soa_tile.GetIntData(1)[index];
+            return ConstParticleCPUWrapper(m_soa_tile.GetIdCPUData()[index]);
         }
     }

@@ -873,7 +912,9 @@ struct ParticleTile
         }

         m_soa_tile.resize(np+1);
-
+        if constexpr (!ParticleType::is_soa_particle) {
+            m_soa_tile.GetIdCPUData()[np] = sp.m_idcpu;
+        }
         auto& arr_rdata = m_soa_tile.GetRealData();
         auto& arr_idata = m_soa_tile.GetIntData();
         for (int i = 0; i < NArrayReal; ++i) {
@@ -1092,6 +1133,11 @@ struct ParticleTile
         } else {
             ptd.m_aos = nullptr;
         }
+        if constexpr (ParticleType::is_soa_particle) {
+            ptd.m_idcpu = m_soa_tile.GetIdCPUData().dataPtr();
+        } else {
+            ptd.m_idcpu = nullptr;
+        }
         if constexpr(NArrayReal > 0) {
             for (int i = 0; i < NArrayReal; ++i) {
                 ptd.m_rdata[i] = m_soa_tile.GetRealData(i).dataPtr();
@@ -1157,6 +1203,11 @@ struct ParticleTile
         } else {
             ptd.m_aos = nullptr;
         }
+        if constexpr (ParticleType::is_soa_particle) {
+            ptd.m_idcpu = m_soa_tile.GetIdCPUData().dataPtr();
+        } else {
+            ptd.m_idcpu = nullptr;
+        }
         if constexpr(NArrayReal > 0) {
             for (int i = 0; i < NArrayReal; ++i) {
                 ptd.m_rdata[i] = m_soa_tile.GetRealData(i).dataPtr();
diff --git a/Src/Particle/AMReX_ParticleTransformation.H b/Src/Particle/AMReX_ParticleTransformation.H
index 28ccfa84a97..aa737455ce6 100644
--- a/Src/Particle/AMReX_ParticleTransformation.H
+++ b/Src/Particle/AMReX_ParticleTransformation.H
@@ -35,7 +35,11 @@ void copyParticle (const      ParticleTileData<T_ParticleType, NAR, NAI>& dst,
     AMREX_ASSERT(dst.m_num_runtime_real == src.m_num_runtime_real);
     AMREX_ASSERT(dst.m_num_runtime_int  == src.m_num_runtime_int );

-    dst.m_aos[dst_i] = src.m_aos[src_i];
+    if constexpr(!T_ParticleType::is_soa_particle) {
+        dst.m_aos[dst_i] = src.m_aos[src_i];
+    } else {
+        dst.m_idcpu[dst_i] = src.m_idcpu[src_i];
+    }
     if constexpr(NAR > 0) {
         for (int j = 0; j < NAR; ++j) {
             dst.m_rdata[j][dst_i] = src.m_rdata[j][src_i];
diff --git a/Src/Particle/AMReX_StructOfArrays.H b/Src/Particle/AMReX_StructOfArrays.H
index 0ef3b8ae869..6cd498e20a2 100644
--- a/Src/Particle/AMReX_StructOfArrays.H
+++ b/Src/Particle/AMReX_StructOfArrays.H
@@ -11,9 +11,11 @@
 namespace amrex {

 template <int NReal, int NInt,
-          template<class> class Allocator=DefaultAllocator>
+          template<class> class Allocator=DefaultAllocator,
+          bool use64BitIdCpu=false>
 struct StructOfArrays {

+    using IdCPU = amrex::PODVector<uint64_t, Allocator<uint64_t> >;
     using RealVector = amrex::PODVector<ParticleReal, Allocator<ParticleReal> >;
     using IntVector = amrex::PODVector<int, Allocator<int> >;

@@ -28,9 +30,12 @@ struct StructOfArrays {

     [[nodiscard]] int NumIntComps () const noexcept { return NInt + m_runtime_idata.size(); }

+    [[nodiscard]] IdCPU& GetIdCPUData () { return m_idcpu; }
     [[nodiscard]] std::array<RealVector, NReal>& GetRealData () { return m_rdata; }
     [[nodiscard]] std::array< IntVector,  NInt>& GetIntData  () { return m_idata; }

+    /** Get access to the particle id/cpu Arrays */
+    [[nodiscard]] const IdCPU& GetIdCPUData () const { return m_idcpu; }
     /** Get access to the particle Real Arrays (only compile-time components) */
     [[nodiscard]] const std::array<RealVector, NReal>& GetRealData () const { return m_rdata; }
     /** Get access to the particle Int Arrays (only compile-time components) */
@@ -119,7 +124,9 @@ struct StructOfArrays {
     */
     [[nodiscard]] std::size_t size () const
     {
-        if constexpr (NReal > 0) {
+        if constexpr (use64BitIdCpu == true) {
+            return m_idcpu.size();
+        } else if constexpr (NReal > 0) {
             return m_rdata[0].size();
         } else if constexpr (NInt > 0) {
             return m_idata[0].size();
@@ -175,6 +182,9 @@ struct StructOfArrays {

     void resize (size_t count)
     {
+        if constexpr (use64BitIdCpu == true) {
+            m_idcpu.resize(count);
+        }
         if constexpr (NReal > 0) {
             for (int i = 0; i < NReal; ++i) { m_rdata[i].resize(count); }
         }
@@ -185,6 +195,15 @@ struct StructOfArrays {
         for (int i = 0; i < int(m_runtime_idata.size()); ++i) { m_runtime_idata[i].resize(count); }
     }

+    [[nodiscard]] IdCPU* idcpuarray () {
+        if constexpr (use64BitIdCpu == true) {
+            return m_idcpu.dataPtr();
+        } else {
+            return nullptr;
+        }
+
+    }
+
     [[nodiscard]] GpuArray<ParticleReal*, NReal> realarray ()
     {
         GpuArray<Real*, NReal> arr;
@@ -208,6 +227,7 @@ struct StructOfArrays {
     int m_num_neighbor_particles{0};

 private:
+    IdCPU m_idcpu;
     std::array<RealVector, NReal> m_rdata;
     std::array< IntVector,  NInt> m_idata;

diff --git a/Src/Particle/AMReX_WriteBinaryParticleData.H b/Src/Particle/AMReX_WriteBinaryParticleData.H
index 31ca7f8df6a..d3cafc5be58 100644
--- a/Src/Particle/AMReX_WriteBinaryParticleData.H
+++ b/Src/Particle/AMReX_WriteBinaryParticleData.H
@@ -338,9 +338,10 @@ packIOData (Vector<int>& idata, Vector<ParticleReal>& rdata, const PC& pc, int l
                 else {
                     amrex::ignore_unused(is_checkpoint);
                     // Int: id, cpu
-                    *iptr = soa.GetIntData(0)[pindex];
+                    uint64_t idcpu = soa.GetIdCPUData()[pindex];
+                    *iptr = (int) ParticleIDWrapper(idcpu);
                     iptr += 1;
-                    *iptr = soa.GetIntData(1)[pindex];
+                    *iptr = (int) ParticleCPUWrapper(idcpu);
                     iptr += 1;

                     // Real: position
@@ -348,8 +349,8 @@ packIOData (Vector<int>& idata, Vector<ParticleReal>& rdata, const PC& pc, int l
                     rptr += AMREX_SPACEDIM;
                 }

-                // extra SoA int data
-                const int int_start_offset = PC::ParticleType::is_soa_particle ? 2 : 0; // pure SoA: skip id, cpu
+                // SoA int data
+                const int int_start_offset = 0;
                 for (int j = int_start_offset; j < pc.NumIntComps(); j++) {
                     if (write_int_comp[PC::NStructInt+j]) {
                         *iptr = soa.GetIntData(j)[pindex];
@@ -1021,14 +1022,15 @@ void WriteBinaryParticleDataAsync (PC const& pc,
                         }
                         else {
                             // Ints: id, cpu
-                            *iptr = soa.GetIntData(0)[pindex];
+                            uint64_t idcpu = soa.GetIdCPUData()[pindex];
+                            *iptr = (int) ParticleIDWrapper(idcpu);
                             iptr += 1;
-                            *iptr = soa.GetIntData(1)[pindex];
+                            *iptr = (int) ParticleCPUWrapper(idcpu);
                             iptr += 1;
                         }

                         // extra SoA Ints
-                        const int int_start_offset = PC::ParticleType::is_soa_particle ? 2 : 0; // pure SoA: skip id, cpu
+                        const int int_start_offset = 0;
                         for (int j = int_start_offset; j < nic; j++)
                         {
                             if (write_int_comp[NStructInt+j])

diff --git a/Tests/Particles/RedistributeSOA/main.cpp b/Tests/Particles/RedistributeSOA/main.cpp
index 94715b7d6aa..62da81def81 100644
--- a/Tests/Particles/RedistributeSOA/main.cpp
+++ b/Tests/Particles/RedistributeSOA/main.cpp
@@ -92,6 +92,7 @@ class TestParticleContainer
         {
             const Box& tile_box  = mfi.tilebox();

+            Gpu::HostVector<uint64_t> host_idcpu;
             std::array<Gpu::HostVector<ParticleReal>, NR> host_real;
             std::array<Gpu::HostVector<int>, NI> host_int;

@@ -106,6 +107,10 @@ class TestParticleContainer

                     amrex::Long id = ParticleType::NextID();

+                    host_idcpu.push_back(0);
+                    ParticleIDWrapper(host_idcpu.back()) = id;
+                    ParticleCPUWrapper(host_idcpu.back()) = ParallelDescriptor::MyProc();
+
                     host_int[0].push_back(static_cast<int>(id));
                     host_int[1].push_back(ParallelDescriptor::MyProc());
                     host_real[0].push_back(static_cast<ParticleReal> (plo[0] + (iv[0] + r[0])*dx[0]));
@@ -137,6 +142,13 @@ class TestParticleContainer
             particle_tile.resize(new_size);

             auto& soa = particle_tile.GetStructOfArrays();
+            {
+                Gpu::copyAsync(Gpu::hostToDevice,
+                               host_idcpu.begin(),
+                               host_idcpu.end(),
+                               soa.GetIdCPUData().begin() + old_size);
+
+            }
             for (int i = 0; i < NR; ++i)
             {
                 Gpu::copyAsync(Gpu::hostToDevice,