Reduce use of SuperParticle (AMReX-Codes/amrex#4526)

## Summary

In preparation of #4404, this PR reduces the reliance of SuperParticle. The issue with getSuperParticle is that the compiler is instructed to read in all the data of the particle into registers even if most of it is not needed. Additionally, runtime data is not accessible through SuperParticle.

## Additional background

The previous version of packIOData with RunOnGpu had a bug when some particles were invalid.

## Checklist

The proposed changes:
- [x] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX users
- [ ] include documentation in the code and/or rst files, if appropriate

[WIP] Add simpler version of ParticleTile using 2D array

## Summary

This PR aims to explore adding a simpler version of ParticleTile #3570

It already assumes that a tile only has runtime components and always has a polymorphic arena allocator as described in #4380. To avoid the array of pointer style of the current particle tile, here a 2D array over particle ids and component ids is used. As a simplification both AoS and SoA members have been removed as well as all the push_back functions.

Performance test with HiPACE++:

dev, ParticleContainerPureSoA<11, 1>
```
TinyProfiler total time across processes [min...avg...max]: 21.44 ... 21.44 ... 21.44
TinyProfiler total time across processes [min...avg...max]: 21.42 ... 21.42 ... 21.42
TinyProfiler total time across processes [min...avg...max]: 21.4 ... 21.4 ... 21.4
TinyProfiler total time across processes [min...avg...max]: 21.43 ... 21.43 ... 21.43
TinyProfiler total time across processes [min...avg...max]: 21.39 ... 21.39 ... 21.39
```
ParticleContainerPureSoA<0, 0> using only runtime components
```
TinyProfiler total time across processes [min...avg...max]: 21.93 ... 21.93 ... 21.93
TinyProfiler total time across processes [min...avg...max]: 22.05 ... 22.05 ... 22.05
TinyProfiler total time across processes [min...avg...max]: 21.95 ... 21.95 ... 21.95
TinyProfiler total time across processes [min...avg...max]: 21.96 ... 21.96 ... 21.96
TinyProfiler total time across processes [min...avg...max]: 21.94 ... 21.94 ... 21.94

```
this PR, ParticleContainerPureSoA2<amrex::Real, int>
```
TinyProfiler total time across processes [min...avg...max]: 21.4 ... 21.4 ... 21.4
TinyProfiler total time across processes [min...avg...max]: 21.38 ... 21.38 ... 21.38
TinyProfiler total time across processes [min...avg...max]: 21.4 ... 21.4 ... 21.4
TinyProfiler total time across processes [min...avg...max]: 21.44 ... 21.44 ... 21.44
TinyProfiler total time across processes [min...avg...max]: 21.38 ... 21.38 ... 21.38
```

## Additional background

ParticleReal and int are replaced with template types to make implementing the const versions easier and to allow for more flexibility in the future (use double and float in the same application, use double2 or float4, use 64 bit int etc).

## Checklist

The proposed changes:
- [ ] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX users
- [ ] include documentation in the code and/or rst files, if appropriate

diff --git a/.github/workflows/gcc.yml b/.github/workflows/gcc.yml
index 3249c6eacd5..0e19117ded1 100644
--- a/.github/workflows/gcc.yml
+++ b/.github/workflows/gcc.yml
@@ -376,7 +376,7 @@ jobs:
       # /home/runner/work/amrex/amrex/Src/Base/AMReX_IntVect.H:194:92: error: array subscript -1 is below array bounds of ‘int [3]’ [-Werror=array-bounds]
       # int& operator[] (int i) noexcept { BL_ASSERT(i>=0 && i < AMREX_SPACEDIM); return vect[i]; }
       #
-      env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wnull-dereference -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wunreachable-code -Wnon-virtual-dtor -Wlogical-op -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches -Wmissing-include-dirs -Wno-array-bounds"}
+      env: {CXXFLAGS: "-fno-operator-names -Werror -Wall -Wextra -Wpedantic -Wfloat-conversion -Wshadow -Woverloaded-virtual -Wunreachable-code -Wnon-virtual-dtor -Wlogical-op -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches -Wmissing-include-dirs -Wno-array-bounds"}
       run: |
         export CCACHE_COMPRESS=1
         export CCACHE_COMPRESSLEVEL=10
diff --git a/Src/Extern/HDF5/AMReX_ParticleHDF5.H b/Src/Extern/HDF5/AMReX_ParticleHDF5.H
index 29747ea9d76..b51e3638fb5 100644
--- a/Src/Extern/HDF5/AMReX_ParticleHDF5.H
+++ b/Src/Extern/HDF5/AMReX_ParticleHDF5.H
@@ -66,9 +66,9 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
     WriteHDF5ParticleData(dir, name, write_real_comp, write_int_comp,
                           tmp_real_comp_names, tmp_int_comp_names,
                           compression,
-                          [=] AMREX_GPU_HOST_DEVICE (const SuperParticleType& p) -> int
+                          [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, int i) -> int
                           {
-                              return p.id() > 0;
+                              return ptd.id(i) > 0;
                           }, true);
 }

@@ -102,9 +102,9 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
     WriteHDF5ParticleData(dir, name, write_real_comp, write_int_comp,
                           real_comp_names, int_comp_names,
                           compression,
-                          [=] AMREX_GPU_HOST_DEVICE (const SuperParticleType& p) -> int
+                          [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, int i) -> int
                           {
-                              return p.id() > 0;
+                              return ptd.id(i) > 0;
                           });
 }

@@ -138,9 +138,9 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
     WriteHDF5ParticleData(dir, name, write_real_comp, write_int_comp,
                           real_comp_names, int_comp_names,
                           compression,
-                          [=] AMREX_GPU_HOST_DEVICE (const SuperParticleType& p)
+                          [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, int i)
                           {
-                              return p.id() > 0;
+                              return ptd.id(i) > 0;
                           });
 }

@@ -166,9 +166,9 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                           write_real_comp, write_int_comp,
                           real_comp_names, int_comp_names,
                           compression,
-                          [=] AMREX_GPU_HOST_DEVICE (const SuperParticleType& p)
+                          [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, int i)
                           {
-                              return p.id() > 0;
+                              return ptd.id(i) > 0;
                           });
 }

@@ -200,9 +200,9 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
                           write_real_comp, write_int_comp,
                           real_comp_names, int_comp_names,
                           compression,
-                          [=] AMREX_GPU_HOST_DEVICE (const ParticleType& p)
+                          [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, int i)
                           {
-                              return p.id() > 0;
+                              return ptd.id(i) > 0;
                           });
 }

@@ -238,9 +238,9 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
     WriteHDF5ParticleData(dir, name, write_real_comp, write_int_comp,
                           real_comp_names, int_comp_names,
                           compression,
-                          [=] AMREX_GPU_HOST_DEVICE (const SuperParticleType& p)
+                          [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, int i)
                           {
-                              return p.id() > 0;
+                              return ptd.id(i) > 0;
                           });
 }

@@ -261,9 +261,9 @@ WritePlotFileHDF5 (const std::string& dir, const std::string& name,
                           write_real_comp, write_int_comp,
                           real_comp_names, int_comp_names,
                           compression,
-                          [=] AMREX_GPU_HOST_DEVICE (const SuperParticleType& p)
+                          [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, int i)
                           {
-                              return p.id() > 0;
+                              return ptd.id(i) > 0;
                           });
 }

diff --git a/Src/Particle/AMReX_Particle.H b/Src/Particle/AMReX_Particle.H
index 51303ae4161..098fdfb7b7a 100644
--- a/Src/Particle/AMReX_Particle.H
+++ b/Src/Particle/AMReX_Particle.H
@@ -337,6 +337,7 @@ struct alignas(sizeof(double)) Particle

     //! The floating point type used for the particles.
     using RealType = ParticleReal;
+    using IntType = int;

     static Long the_next_id;

@@ -366,6 +367,12 @@ struct alignas(sizeof(double)) Particle
 #endif
     }

+    [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
+    uint64_t& idcpu () & { return this->m_idcpu; }
+
+    [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
+    const uint64_t& idcpu () const & { return this->m_idcpu; }
+
     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
     RealVect pos () const & {return RealVect(AMREX_D_DECL(this->m_pos[0], this->m_pos[1], this->m_pos[2]));}

diff --git a/Src/Particle/AMReX_ParticleContainer.H b/Src/Particle/AMReX_ParticleContainer.H
index 449b1c85409..3baafe1d34e 100644
--- a/Src/Particle/AMReX_ParticleContainer.H
+++ b/Src/Particle/AMReX_ParticleContainer.H
@@ -185,6 +185,8 @@ public:
     //! A single level worth of particles is indexed (grid id, tile id)
     //! for both SoA and AoS data.
     using ParticleLevel = std::map<std::pair<int, int>, ParticleTileType>;
+    using PTDType = typename ParticleTileType::ParticleTileDataType;
+    using ConstPTDType = typename ParticleTileType::ConstParticleTileDataType;
     using AoS = typename ParticleTileType::AoS;
     using SoA = typename ParticleTileType::SoA;

diff --git a/Src/Particle/AMReX_ParticleContainerI.H b/Src/Particle/AMReX_ParticleContainerI.H
index 7b200bedb75..1d90fde2c76 100644
--- a/Src/Particle/AMReX_ParticleContainerI.H
+++ b/Src/Particle/AMReX_ParticleContainerI.H
@@ -954,42 +954,47 @@ ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssig
         auto dst = virts.getParticleTileData();
         ParallelFor(bx, [=] AMREX_GPU_DEVICE(int i, int j, int k)
         {
-             if(imf_arr(i,j,k,0)!=0)
-             {
-                 SuperParticleType p;
-                 p.cpu() = 0;
-                 p.id() = LongParticleIds::VirtualParticleID;
-
-                 //Set rdata(0) first so we can normalize the weighted fields
-                 p.rdata(0) = static_cast<ParticleReal>(partData(i,j,k,AMREX_SPACEDIM));
-                 //Set pos with the normalized weighted field
-                 for (int n = 0; n < AMREX_SPACEDIM; ++n)
-                 {
-                     p.pos(n) = static_cast<ParticleReal>(partData(i,j,k,n) / p.rdata(0));
-                 }
-                 //Set rdata(n>0) with the normalized weighted field for NStructReal
-                 for (int n = 1; n < NStructReal; ++n)
-                 {
-                     p.rdata(n) = static_cast<ParticleReal>(partData(i,j,k,AMREX_SPACEDIM+n) / p.rdata(0));
-                 }
-                 //Set rdata(n>0) with the normalized weighted field for NArrayReal
-                 for (int n = 0; n < NArrayReal; ++n)
-                 {
-                     p.rdata(NStructReal+n) = static_cast<ParticleReal>(partData(i,j,k,AMREX_SPACEDIM+NStructReal+n));
-                 }
-                 //Set idata with the "first" particles idata field for NStructInt
-                 for (int n = 0; n < NStructInt; ++n)
-                 {
-                     p.idata(n) = imf_arr(i,j,k,1+n);
-                 }
-                 //Set idata with the "first" particles idata field for NArrayInt
-                 for (int n = 0; n < NArrayInt; ++n)
-                 {
-                     p.idata(NStructInt+n) = imf_arr(i,j,k,1+NStructInt+n);
-                 }
-                 //Set the new particle in dst tile
-                 dst.setSuperParticle(p, last_offset+offsets_ptr[((i-imf_arr.begin.x)+(j-imf_arr.begin.y)*imf_arr.jstride+(k-imf_arr.begin.z)*imf_arr.kstride)]);
-             }
+            if(imf_arr(i,j,k,0)!=0)
+            {
+                const auto idx = last_offset + offsets_ptr[
+                    ((i-imf_arr.begin.x)
+                    +(j-imf_arr.begin.y)*imf_arr.jstride
+                    +(k-imf_arr.begin.z)*imf_arr.kstride)
+                ];
+
+                dst.cpu(idx) = 0;
+                dst.id(idx) = LongParticleIds::VirtualParticleID;
+
+                auto& p = dst[idx];
+                //Set rdata(0) first so we can normalize the weighted fields
+                //Note that this does not work for soa PC
+                p.rdata(0) = static_cast<ParticleReal>(partData(i,j,k,AMREX_SPACEDIM));;
+                //Set pos with the normalized weighted field
+                for (int n = 0; n < AMREX_SPACEDIM; ++n)
+                {
+                    p.pos(n) = static_cast<ParticleReal>(partData(i,j,k,n) / p.rdata(0));
+                }
+                //Set rdata(n>0) with the normalized weighted field for NStructReal
+                for (int n = 1; n < NStructReal; ++n)
+                {
+                    p.rdata(n) = static_cast<ParticleReal>(partData(i,j,k,AMREX_SPACEDIM+n) / p.rdata(0));
+                }
+                //Set rdata(n>0) with the normalized weighted field for NArrayReal
+                for (int n = 0; n < NArrayReal; ++n)
+                {
+                    dst.rdata(n)[idx] = static_cast<ParticleReal>(partData(i,j,k,AMREX_SPACEDIM+NStructReal+n));
+                }
+                //Set idata with the "first" particles idata field for NStructInt
+                for (int n = 0; n < NStructInt; ++n)
+                {
+                    p.idata(n) = imf_arr(i,j,k,1+n);
+                }
+                //Set idata with the "first" particles idata field for NArrayInt
+                for (int n = 0; n < NArrayInt; ++n)
+                {
+                    dst.idata(n)[idx] = imf_arr(i,j,k,1+NStructInt+n);
+                }
+            }

         });
         last_offset+=next_offset;
diff --git a/Src/Particle/AMReX_ParticleReduce.H b/Src/Particle/AMReX_ParticleReduce.H
index db2bb6ea3e2..50002e29327 100644
--- a/Src/Particle/AMReX_ParticleReduce.H
+++ b/Src/Particle/AMReX_ParticleReduce.H
@@ -66,18 +66,18 @@ auto call_f (F const& f,
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto sm = amrex::ReduceSum(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceSum (PC const& pc, F&& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
     return ReduceSum(pc, 0, pc.finestLevel(), std::forward<F>(f));
 }
@@ -114,18 +114,18 @@ ReduceSum (PC const& pc, F&& f)
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto sm = amrex::ReduceSum(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceSum (PC const& pc, int lev, F&& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
     return ReduceSum(pc, lev, lev, std::forward<F>(f));
 }
@@ -163,20 +163,20 @@ ReduceSum (PC const& pc, int lev, F&& f)
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto sm = amrex::ReduceSum(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceSum (PC const& pc, int lev_min, int lev_max, F const& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
-    using value_type = decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()));
+    using value_type = decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()));
     value_type sm = 0;

 #ifdef AMREX_USE_GPU
@@ -266,18 +266,18 @@ ReduceSum (PC const& pc, int lev_min, int lev_max, F const& f)
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto mx = amrex::ReduceMax(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceMax (PC const& pc, F&& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
     return ReduceMax(pc, 0, pc.finestLevel(), std::forward<F>(f));
 }
@@ -314,18 +314,18 @@ ReduceMax (PC const& pc, F&& f)
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto mx = amrex::ReduceMax(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceMax (PC const& pc, int lev, F&& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
     return ReduceMax(pc, lev, lev, std::forward<F>(f));
 }
@@ -363,20 +363,20 @@ ReduceMax (PC const& pc, int lev, F&& f)
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto mx = amrex::ReduceMax(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceMax (PC const& pc, int lev_min, int lev_max, F const& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
-    using value_type = decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()));
+    using value_type = decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()));
     constexpr value_type value_lowest = std::numeric_limits<value_type>::lowest();
     value_type r = value_lowest;

@@ -467,18 +467,18 @@ ReduceMax (PC const& pc, int lev_min, int lev_max, F const& f)
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto mn = amrex::ReduceMin(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceMin (PC const& pc, F&& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
     return ReduceMin(pc, 0, pc.finestLevel(), std::forward<F>(f));
 }
@@ -515,18 +515,18 @@ ReduceMin (PC const& pc, F&& f)
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto mn = amrex::ReduceMin(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceMin (PC const& pc, int lev, F&& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
     return ReduceMin(pc, lev, lev, std::forward<F>(f));
 }
@@ -564,20 +564,20 @@ ReduceMin (PC const& pc, int lev, F&& f)
  *                      return p.idata(0);
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto mn = amrex::ReduceMin(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> ParticleReal
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> ParticleReal
  *                  {
- *                      return ptd.m_aos[i].rdata(0);
+ *                      return ptd.rdata(0)[i];
  *                  });
  *
  */
 template <class PC, class F, std::enable_if_t<IsParticleContainer<PC>::value, int> foo = 0>
 auto
 ReduceMin (PC const& pc, int lev_min, int lev_max, F const& f)
-    -> decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()))
+    -> decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()))
 {
-    using value_type = decltype(particle_detail::call_f(f, typename PC::ParticleTileType::ConstParticleTileDataType(), int()));
+    using value_type = decltype(particle_detail::call_f(f, typename PC::ConstPTDType(), int()));
     constexpr value_type value_max = std::numeric_limits<value_type>::max();
     value_type r = value_max;

@@ -668,11 +668,11 @@ ReduceMin (PC const& pc, int lev_min, int lev_max, F const& f)
  *                      return p.id() > 0;
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto rv = amrex::ReduceLogicalAnd(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> bool
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> bool
  *                  {
- *                      return ptd.m_aos[i].id() > 0;
+ *                      return ptd.id(i) > 0;
  *                  });
  *
  */
@@ -715,11 +715,11 @@ ReduceLogicalAnd (PC const& pc, F&& f)
  *                      return p.id() > 0;
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto rv = amrex::ReduceLogicalAnd(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> bool
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> bool
  *                  {
- *                      return ptd.m_aos[i].id() > 0;
+ *                      return ptd.id(i) > 0;
  *                  });
  *
  */
@@ -763,11 +763,11 @@ ReduceLogicalAnd (PC const& pc, int lev, F&& f)
  *                      return p.id() > 0;
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto rv = amrex::ReduceLogicalAnd(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> bool
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> bool
  *                  {
- *                      return ptd.m_aos[i].id() > 0;
+ *                      return ptd.id(i) > 0;
  *                  });
  *
  */
@@ -864,11 +864,11 @@ ReduceLogicalAnd (PC const& pc, int lev_min, int lev_max, F const& f)
  *                      return p.id() < 1;
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto rv = amrex::ReduceLogicalOr(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> bool
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> bool
  *                  {
- *                      return ptd.m_aos[i].id() < 1;
+ *                      return ptd.id(i) < 1;
  *                  });
  *
  */
@@ -911,11 +911,11 @@ ReduceLogicalOr (PC const& pc, F&& f)
  *                      return p.id() < 1;
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto rv = amrex::ReduceLogicalOr(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> bool
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> bool
  *                  {
- *                      return ptd.m_aos[i].id() < 1;
+ *                      return ptd.id(i) < 1;
  *                  });
  *
  */
@@ -959,11 +959,11 @@ ReduceLogicalOr (PC const& pc, int lev, F&& f)
  *                      return p.id() < 1;
  *                  });
  *
- *        using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *        using ConstPTDType = typename PC::ConstPTDType;
  *        auto rv = amrex::ReduceLogicalOr(pc,
- *                  [=] AMREX_GPU_HOST_DEVICE (const PTDType& ptd, const int i) -> bool
+ *                  [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> bool
  *                  {
- *                      return ptd.m_aos[i].id() < 1;
+ *                      return ptd.id(i) < 1;
  *                  });
  *
  */
@@ -1078,15 +1078,15 @@ ReduceLogicalOr (PC const& pc, int lev_min, int lev_max, F const& f)
  *                 return {a, b, c};
  *             }, reduce_ops);
  *
- *    using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *    using ConstPTDType = typename PC::ConstPTDType;
  *    amrex::ReduceOps<ReduceOpSum, ReduceOpMin, ReduceOpMax> reduce_ops;
  *    auto r = amrex::ParticleReduce<ReduceData<amrex::Real, amrex::Real,int>> (
- *                 pc, [=] AMREX_GPU_DEVICE (const PTDType& ptd, const int i) noexcept
+ *                 pc, [=] AMREX_GPU_DEVICE (const ConstPTDType& ptd, const int i) noexcept
  *                               -> amrex::GpuTuple<amrex::Real,amrex::Real,int>
  *             {
- *                 const amrex::Real a = ptd.m_aos[i].rdata(1);
- *                 const amrex::Real b = ptd.m_aos[i].rdata(2);
- *                 const int c = ptd.m_aos[i].idata(1);
+ *                 const amrex::Real a = ptd.rdata(1)[i];
+ *                 const amrex::Real b = ptd.rdata(2)[i];
+ *                 const int c = ptd.idata(1)[i];
  *                 return {a, b, c};
  *             }, reduce_ops);
  *
@@ -1148,15 +1148,15 @@ ParticleReduce (PC const& pc, F&& f, ReduceOps& reduce_ops)
  *                 return {a, b, c};
  *             }, reduce_ops);
  *
- *    using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *    using ConstPTDType = typename PC::ConstPTDType;
  *    amrex::ReduceOps<ReduceOpSum, ReduceOpMin, ReduceOpMax> reduce_ops;
  *    auto r = amrex::ParticleReduce<ReduceData<amrex::Real, amrex::Real,int>> (
- *                 pc, [=] AMREX_GPU_DEVICE (const PTDType& ptd, const int i) noexcept
+ *                 pc, [=] AMREX_GPU_DEVICE (const ConstPTDType& ptd, const int i) noexcept
  *                               -> amrex::GpuTuple<amrex::Real,amrex::Real,int>
  *             {
- *                 const amrex::Real a = ptd.m_aos[i].rdata(1);
- *                 const amrex::Real b = ptd.m_aos[i].rdata(2);
- *                 const int c = ptd.m_aos[i].idata(1);
+ *                 const amrex::Real a = ptd.rdata(1)[i];
+ *                 const amrex::Real b = ptd.rdata(2)[i];
+ *                 const int c = ptd.idata(1)[i];
  *                 return {a, b, c};
  *             }, reduce_ops);
  *
@@ -1219,15 +1219,15 @@ ParticleReduce (PC const& pc, int lev, F&& f, ReduceOps& reduce_ops)
  *                 return {a, b, c};
  *             }, reduce_ops);
  *
- *    using PTDType = typename PC::ParticleTileType::ConstParticleTileDataType;
+ *    using ConstPTDType = typename PC::ConstPTDType;
  *    amrex::ReduceOps<ReduceOpSum, ReduceOpMin, ReduceOpMax> reduce_ops;
  *    auto r = amrex::ParticleReduce<ReduceData<amrex::Real, amrex::Real,int>> (
- *                 pc, [=] AMREX_GPU_DEVICE (const PTDType& ptd, const int i) noexcept
+ *                 pc, [=] AMREX_GPU_DEVICE (const ConstPTDType& ptd, const int i) noexcept
  *                               -> amrex::GpuTuple<amrex::Real,amrex::Real,int>
  *             {
- *                 const amrex::Real a = ptd.m_aos[i].rdata(1);
- *                 const amrex::Real b = ptd.m_aos[i].rdata(2);
- *                 const int c = ptd.m_aos[i].idata(1);
+ *                 const amrex::Real a = ptd.rdata(1)[i];
+ *                 const amrex::Real b = ptd.rdata(2)[i];
+ *                 const int c = ptd.idata(1)[i];
  *                 return {a, b, c};
  *             }, reduce_ops);
  *
diff --git a/Src/Particle/AMReX_ParticleTile.H b/Src/Particle/AMReX_ParticleTile.H
index 80f7705e266..d7ba7a1f136 100644
--- a/Src/Particle/AMReX_ParticleTile.H
+++ b/Src/Particle/AMReX_ParticleTile.H
@@ -36,6 +36,8 @@ struct ParticleTileData
     using ParticleType = T_ParticleType;
     using ParticleRefType = T_ParticleType&;
     using Self = ParticleTileData<ParticleType, NAR, NAI>;
+    using RealType = ParticleReal;
+    using IntType = int;

     static constexpr int NStructReal = ParticleType::NReal;
     static constexpr int NStructInt = ParticleType::NInt;
@@ -60,7 +62,7 @@ struct ParticleTileData
     int* AMREX_RESTRICT * AMREX_RESTRICT m_runtime_idata;

     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
-    ParticleReal& pos (const int dir, const int index) const &
+    decltype(auto) pos (const int dir, const int index) const &
     {
         if constexpr(!ParticleType::is_soa_particle) {
             return this->m_aos[index].pos(dir);
@@ -95,7 +97,7 @@ struct ParticleTileData
         if constexpr(ParticleType::is_soa_particle) {
             return this->m_idcpu[index];
         } else {
-            amrex::Abort("not implemented");
+            return this->m_aos[index].idcpu();
         }
     }

@@ -329,6 +331,7 @@ struct alignas(sizeof(double)) ConstSoAParticle : SoAParticleBase
     static constexpr bool is_constsoa_particle = true;

     using RealType = ParticleReal;
+    using IntType = int;

     AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
     ConstSoAParticle (ConstPTD const& ptd, long i) : // Note: should this be int instead?
@@ -374,7 +377,7 @@ struct alignas(sizeof(double)) ConstSoAParticle : SoAParticleBase

     private :

-    static_assert(std::is_trivially_copyable<ParticleTileData<SoAParticleBase, NArrayReal, NArrayInt>>(), "ParticleTileData is not trivially copyable");
+    static_assert(std::is_trivially_copyable<ConstPTD>(), "ParticleTileData is not trivially copyable");

     ConstPTD m_constparticle_tile_data;
     int m_index;
@@ -392,6 +395,7 @@ struct alignas(sizeof(double)) SoAParticle : SoAParticleBase

     using ConstType = ConstSoAParticle<T_NArrayReal, T_NArrayInt>;
     using RealType = ParticleReal;
+    using IntType = int;

     AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
     SoAParticle (PTD const& ptd, long i) : // Note: should this be int instead?
@@ -456,7 +460,7 @@ struct alignas(sizeof(double)) SoAParticle : SoAParticleBase

 private :

-    static_assert(std::is_trivially_copyable<ParticleTileData<SoAParticleBase, NArrayReal, NArrayInt>>(), "ParticleTileData is not trivially copyable");
+    static_assert(std::is_trivially_copyable<PTD>(), "ParticleTileData is not trivially copyable");

     PTD m_particle_tile_data;
     int m_index;
@@ -511,6 +515,8 @@ struct ConstParticleTileData
     static constexpr int NAI = NArrayInt;
     using ParticleType = T_ParticleType;
     using ParticleRefType = T_ParticleType const&;
+    using RealType = ParticleReal;
+    using IntType = int;

     static constexpr int NStructReal = ParticleType::NReal;
     static constexpr int NStructInt = ParticleType::NInt;
@@ -530,7 +536,7 @@ struct ConstParticleTileData
     GpuArray<const int*, NArrayInt > m_idata;

     [[nodiscard]] AMREX_GPU_HOST_DEVICE AMREX_FORCE_INLINE
-    const ParticleReal& pos (const int dir, const int index) const &
+    decltype(auto) pos (const int dir, const int index) const &
     {
         if constexpr(!ParticleType::is_soa_particle) {
             return this->m_aos[index].pos(dir);
@@ -565,7 +571,7 @@ struct ConstParticleTileData
         if constexpr(ParticleType::is_soa_particle) {
             return this->m_idcpu[index];
         } else {
-            amrex::Abort("not implemented");
+            return this->m_aos[index].idcpu();
         }
     }

diff --git a/Src/Particle/AMReX_WriteBinaryParticleData.H b/Src/Particle/AMReX_WriteBinaryParticleData.H
index 3c03efe076b..4a5117a1d29 100644
--- a/Src/Particle/AMReX_WriteBinaryParticleData.H
+++ b/Src/Particle/AMReX_WriteBinaryParticleData.H
@@ -47,8 +47,12 @@ fillFlags (Container<int, Allocator>& pflags, const PTile& ptile, F const& f)
             amrex::ignore_unused(flag_ptr, f, engine);
             if constexpr (IsCallable<F,decltype(p),RandomEngine>::value) {
                 flag_ptr[k] = f(p,engine);
-            } else {
+            } else if constexpr (IsCallable<F,decltype(p)>::value) {
                 flag_ptr[k] = f(p);
+            } else if constexpr (IsCallable<F,decltype(ptd),int,RandomEngine>::value) {
+                flag_ptr[k] = f(ptd,k,engine);
+            } else {
+                flag_ptr[k] = f(ptd,k);
             }
         });
 }
@@ -68,8 +72,12 @@ fillFlags (Container<int, Allocator>& pflags, const PTile& ptile, F const& f)
         const auto p = ptd.getSuperParticle(k);
         if constexpr (IsCallable<F,decltype(p),RandomEngine>::value) {
             flag_ptr[k] = f(p,getInvalidRandomEngine());
-        } else {
+        } else if constexpr (IsCallable<F,decltype(p)>::value) {
             flag_ptr[k] = f(p);
+        } else if constexpr (IsCallable<F,decltype(ptd),int,RandomEngine>::value) {
+            flag_ptr[k] = f(ptd,k,getInvalidRandomEngine());
+        } else {
+            flag_ptr[k] = f(ptd,k);
         }
     }
 }
@@ -150,22 +158,98 @@ countFlags (const Container<int,Allocator>& pflags)
     return nparticles;
 }

-template <typename P, typename I>
+template <typename I>
 AMREX_GPU_HOST_DEVICE
-void packParticleIDs (I* idata, const P& p, bool is_checkpoint) noexcept
+void packParticleIDs (I* idata, const std::uint64_t idcpu, bool is_checkpoint) noexcept
 {
     if (is_checkpoint) {
         std::int32_t  xi, yi;
         std::uint32_t xu, yu;
-        xu = (std::uint32_t)((p.m_idcpu & 0xFFFFFFFF00000000LL) >> 32);
-        yu = (std::uint32_t)( p.m_idcpu & 0xFFFFFFFFLL);
+        xu = (std::uint32_t)((idcpu & 0xFFFFFFFF00000000LL) >> 32);
+        yu = (std::uint32_t)( idcpu & 0xFFFFFFFFLL);
         amrex::Gpu::memcpy(&xi, &xu, sizeof(xu));
         amrex::Gpu::memcpy(&yi, &yu, sizeof(yu));
         idata[0] = xi;
         idata[1] = yi;
     } else {
-        idata[0] = p.id();
-        idata[1] = p.cpu();
+        idata[0] = ConstParticleIDWrapper{idcpu};
+        idata[1] = ConstParticleCPUWrapper{idcpu};
+    }
+}
+
+template<class PTD>
+AMREX_GPU_HOST_DEVICE void
+rPackParticleData (const PTD& ptd, int idx, typename PTD::RealType * rdata_ptr,
+                   const int * write_real_comp)
+{
+    std::size_t rout_index = 0;
+
+    for (int j = 0; j < AMREX_SPACEDIM; ++j) {
+        rdata_ptr[rout_index] = ptd.pos(j, idx);
+        rout_index++;
+    }
+
+    if constexpr (!PTD::ParticleType::is_soa_particle) {
+        const auto& p = ptd[idx];
+
+        for (int j = 0; j < PTD::ParticleType::NReal; ++j) {
+            if (write_real_comp[j]) {
+                rdata_ptr[rout_index] = p.rdata(j);
+                rout_index++;
+            }
+        }
+    }
+
+    constexpr int real_start_offset = PTD::ParticleType::is_soa_particle ? AMREX_SPACEDIM : 0;
+
+    for (int j = real_start_offset; j < PTD::NAR; ++j) {
+        if (write_real_comp[PTD::ParticleType::NReal + j - real_start_offset]) {
+            rdata_ptr[rout_index] = ptd.rdata(j)[idx];
+            rout_index++;
+        }
+    }
+
+    for (int j = 0; j < ptd.m_num_runtime_real; ++j) {
+        if (write_real_comp[PTD::ParticleType::NReal + PTD::NAR + j - real_start_offset]) {
+            rdata_ptr[rout_index] = ptd.m_runtime_rdata[j][idx];
+            rout_index++;
+        }
+    }
+}
+
+template<class PTD>
+AMREX_GPU_HOST_DEVICE void
+iPackParticleData (const PTD& ptd, int idx, typename PTD::IntType * idata_ptr,
+                   const int * write_int_comp, bool is_checkpoint)
+{
+    std::size_t iout_index = 0;
+
+    packParticleIDs(&idata_ptr[iout_index], ptd.idcpu(idx), is_checkpoint);
+    iout_index += 2;
+
+    if constexpr (!PTD::ParticleType::is_soa_particle) {
+        const auto& p = ptd[idx];
+
+        for (int j = 0; j < PTD::ParticleType::NInt; ++j) {
+            if (write_int_comp[j]) {
+                idata_ptr[iout_index] = p.idata(j);
+                iout_index++;
+            }
+        }
+    }
+
+    for (int j = 0; j < PTD::NAI; ++j) {
+        if (write_int_comp[PTD::ParticleType::NInt + j]) {
+            idata_ptr[iout_index] = ptd.idata(j)[idx];
+            iout_index++;
+        }
+    }
+
+    for (int j = 0; j < ptd.m_num_runtime_int; ++j) {
+        if (write_int_comp[PTD::ParticleType::NInt + PTD::NAI + j]) {
+            idata_ptr[iout_index] = ptd.m_runtime_idata[j][idx];
+            iout_index++;
+        }
     }
 }

@@ -214,58 +298,22 @@ packIOData (Vector<int>& idata, Vector<ParticleReal>& rdata, const PC& pc, int l
         typename PC::RealVector rdata_d(num_copies*rChunkSize);

         const auto flag_ptr = pflags.data();
+        const auto offset_ptr = offsets.data();

         auto idata_d_ptr = idata_d.data();
         auto rdata_d_ptr = rdata_d.data();

         const auto& ptd = ptile.getConstParticleTileData();
-        amrex::ParallelFor(num_copies,
+        amrex::ParallelFor(ptile.numParticles(),
         [=] AMREX_GPU_DEVICE (int pindex) noexcept
         {
-            // might be worth using shared memory here
-            const auto p = ptd.getSuperParticle(pindex);
-
             if (flag_ptr[pindex]) {
-                std::size_t iout_index = pindex*iChunkSize;
-                packParticleIDs(&idata_d_ptr[iout_index], p, is_checkpoint);
-                iout_index += 2;
-
-                std::size_t rout_index = pindex*rChunkSize;
-                for (int j = 0; j < AMREX_SPACEDIM; j++) {
-                  rdata_d_ptr[rout_index] = p.pos(j);
-                  rout_index++;
-                }
-
-                for (int j = 0; j < PC::SuperParticleType::NInt; j++) {
-                    if (write_int_comp_d_ptr[j]) {
-                        idata_d_ptr[iout_index] = p.idata(j);
-                        iout_index++;
-                    }
-                }
+                const int out_indx = offset_ptr[pindex];
+                iPackParticleData(ptd, pindex, idata_d_ptr + out_indx * iChunkSize,
+                                  write_int_comp_d_ptr, is_checkpoint);

-                for (int j = 0; j < ptd.m_num_runtime_int; j++) {
-                    if (write_int_comp_d_ptr[PC::SuperParticleType::NInt + j]) {
-                        idata_d_ptr[iout_index] = ptd.m_runtime_idata[j][pindex];
-                        iout_index++;
-                    }
-                }
-
-                // extra SoA Real components
-                const int real_start_offset = PC::ParticleType::is_soa_particle ? AMREX_SPACEDIM : 0; // pure SoA: skip positions
-                for (int j = real_start_offset; j < PC::SuperParticleType::NReal; j++) {
-                    const int write_comp_index = j-real_start_offset;
-                    if (write_real_comp_d_ptr[write_comp_index]) {
-                        rdata_d_ptr[rout_index] = p.rdata(j);
-                        rout_index++;
-                    }
-                }
-
-                for (int j = 0; j < ptd.m_num_runtime_real; j++) {
-                    if (write_real_comp_d_ptr[PC::SuperParticleType::NReal+j-real_start_offset]) {
-                        rdata_d_ptr[rout_index] = ptd.m_runtime_rdata[j][pindex];
-                        rout_index++;
-                    }
-                }
+                rPackParticleData(ptd, pindex, rdata_d_ptr + out_indx * rChunkSize,
+                                  write_real_comp_d_ptr);
             }
         });

@@ -307,84 +355,17 @@ packIOData (Vector<int>& idata, Vector<ParticleReal>& rdata, const PC& pc, int l
     for (int tile : tiles) {
         const auto& ptile = pc.ParticlesAt(lev, grid, tile);
         const auto& pflags = particle_io_flags[lev].at(std::make_pair(grid, tile));
+        const auto& ptd = ptile.getConstParticleTileData();
+
         for (int pindex = 0; pindex < ptile.numParticles(); ++pindex) {
             if (pflags[pindex]) {
-                const auto& soa  = ptile.GetStructOfArrays();
-
-                // note: for pure SoA particle layouts, we do write the id, cpu and positions as a struct
-                //       for backwards compatibility with readers
-                if constexpr(!PC::ParticleType::is_soa_particle)
-                {
-                    const auto& aos = ptile.GetArrayOfStructs();
-                    const auto& p = aos[pindex];
-
-                    // Int: id, cpu
-                    packParticleIDs(iptr, p, is_checkpoint);
-                    iptr += 2;
-
-                    // Real: positions
-                    for (int j = 0; j < AMREX_SPACEDIM; j++) { rptr[j] = p.pos(j); }
-                    rptr += AMREX_SPACEDIM;
-
-                    // extra AoS Int components
-                    for (int j = 0; j < PC::NStructInt; j++) {
-                        if (write_int_comp[j]) {
-                            *iptr = p.idata(j);
-                            ++iptr;
-                        }
-                    }
-                    // extra AoS Real components
-                    for (int j = 0; j < PC::NStructReal; j++) {
-                        if (write_real_comp[j]) {
-                            *rptr = p.rdata(j);
-                            ++rptr;
-                        }
-                    }
-                }
-                else {
-                    uint64_t idcpu = soa.GetIdCPUData()[pindex];
-                    if (is_checkpoint) {
-                        std::int32_t  xi, yi;
-                        std::uint32_t xu, yu;
-                        xu = (std::uint32_t)((idcpu & 0xFFFFFFFF00000000LL) >> 32);
-                        yu = (std::uint32_t)( idcpu & 0xFFFFFFFFLL);
-                        std::memcpy(&xi, &xu, sizeof(xu));
-                        std::memcpy(&yi, &yu, sizeof(yu));
-                        *iptr = xi;
-                        iptr += 1;
-                        *iptr = yi;
-                        iptr += 1;
-                    } else {
-                        // Int: id, cpu
-                        *iptr = (int) ParticleIDWrapper(idcpu);
-                        iptr += 1;
-                        *iptr = (int) ParticleCPUWrapper(idcpu);
-                        iptr += 1;
-                    }
-
-                    // Real: position
-                    for (int j = 0; j < AMREX_SPACEDIM; j++) { rptr[j] = soa.GetRealData(j)[pindex]; }
-                    rptr += AMREX_SPACEDIM;
-                }
+                iPackParticleData(ptd, pindex, iptr,
+                                  write_int_comp.dataPtr(), is_checkpoint);
+                iptr += iChunkSize;

-                // SoA int data
-                const int int_start_offset = 0;
-                for (int j = int_start_offset; j < pc.NumIntComps(); j++) {
-                    if (write_int_comp[PC::NStructInt+j]) {
-                        *iptr = soa.GetIntData(j)[pindex];
-                        ++iptr;
-                    }
-                }
-
-                // extra SoA Real components
-                const int real_start_offset = PC::ParticleType::is_soa_particle ? AMREX_SPACEDIM : 0; // pure SoA: skip positions
-                for (int j = real_start_offset; j < pc.NumRealComps(); j++) {
-                    const int write_comp_index = PC::NStructReal+j-real_start_offset;
-                    if (write_real_comp[write_comp_index]) {
-                        *rptr = (ParticleReal) soa.GetRealData(j)[pindex];
-                        ++rptr;
-                    }
-                }
+                rPackParticleData(ptd, pindex, rptr,
+                                  write_real_comp.dataPtr());
+                rptr += rChunkSize;
             }
         }
     }
@@ -875,7 +856,6 @@ void WriteBinaryParticleDataAsync (PC const& pc,
         dms.push_back(pc.ParticleDistributionMap(lev));
     }

-    int nrc = pc.NumRealComps();
     int nic = pc.NumIntComps();
     int rnames_size = (int) real_comp_names.size();

@@ -1028,62 +1008,11 @@ void WriteBinaryParticleDataAsync (PC const& pc,
                     const auto& ptd = pbox.getConstParticleTileData();
                     for (int pindex = 0; pindex < pbox.numParticles(); ++pindex)
                     {
-                        const auto& soa  = pbox.GetStructOfArrays();
-
-                        make_particle<typename PC::ConstParticleType> mp{};
-                        const auto& p = mp(ptd, pindex);
-                        if (p.id() <= 0) { continue; }
-
-                        // note: for pure SoA particle layouts, we do write the id, cpu and positions as a struct
-                        //       for backwards compatibility with readers
-                        if constexpr(!PC::ParticleType::is_soa_particle)
-                        {
-                            // Ints: id, cpu
-                            particle_detail::packParticleIDs(iptr, p, is_checkpoint);
-                            iptr += 2;
-
-                            // extra AoS Int components
-                            for (int j = 0; j < NStructInt; j++)
-                            {
-                                if (write_int_comp[j])
-                                {
-                                    *iptr = p.idata(j);
-                                    ++iptr;
-                                }
-                            }
-                        }
-                        else {
-                            uint64_t idcpu = soa.GetIdCPUData()[pindex];
-                            if (is_checkpoint) {
-                                std::int32_t  xi, yi;
-                                std::uint32_t xu, yu;
-                                xu = (std::uint32_t)((idcpu & 0xFFFFFFFF00000000LL) >> 32);
-                                yu = (std::uint32_t)( idcpu & 0xFFFFFFFFLL);
-                                std::memcpy(&xi, &xu, sizeof(xu));
-                                std::memcpy(&yi, &yu, sizeof(yu));
-                                *iptr = xi;
-                                iptr += 1;
-                                *iptr = yi;
-                                iptr += 1;
-                            } else {
-                                // Int: id, cpu
-                                *iptr = (int) ParticleIDWrapper(idcpu);
-                                iptr += 1;
-                                *iptr = (int) ParticleCPUWrapper(idcpu);
-                                iptr += 1;
-                            }
-                        }
+                        if (ptd.id(pindex) <= 0) { continue; }

-                        // extra SoA Ints
-                        const int int_start_offset = 0;
-                        for (int j = int_start_offset; j < nic; j++)
-                        {
-                            if (write_int_comp[NStructInt+j])
-                            {
-                                *iptr = soa.GetIntData(j)[pindex];
-                                ++iptr;
-                            }
-                        }
+                        particle_detail::iPackParticleData(ptd, pindex, iptr,
+                            write_int_comp.dataPtr(), is_checkpoint);
+                        iptr += iChunkSize;
                     }
                 }

@@ -1104,49 +1033,13 @@ void WriteBinaryParticleDataAsync (PC const& pc,
                     auto ptile_index = std::make_pair(grid, tile_map[grid][i]);
                     const auto& pbox = (*myptiles)[lev][ptile_index];
                     const auto& ptd = pbox.getConstParticleTileData();
-                    for (int pindex = 0;
-                         pindex < pbox.numParticles(); ++pindex)
+                    for (int pindex = 0; pindex < pbox.numParticles(); ++pindex)
                     {
-                        const auto& soa  = pbox.GetStructOfArrays();
-                        make_particle<typename PC::ConstParticleType> mp{};
-                        const auto& p = mp(ptd, pindex);
-
-                        if (p.id() <= 0) { continue; }
-
-                        if constexpr(!PC::ParticleType::is_soa_particle)
-                        {
-                            // Real: position
-                            for (int j = 0; j < AMREX_SPACEDIM; j++) { rptr[j] = p.pos(j); }
-                            rptr += AMREX_SPACEDIM;
-
-                            // extra AoS real
-                            for (int j = 0; j < NStructReal; j++)
-                            {
-                                if (write_real_comp[j])
-                                {
-                                    *rptr = p.rdata(j);
-                                    ++rptr;
-                                }
-                            }
-                        }
-                        else {
-                            // Real: position
-                            for (int j = 0; j < AMREX_SPACEDIM; j++) { rptr[j] = soa.GetRealData(j)[pindex]; }
-                            rptr += AMREX_SPACEDIM;
-                        }
+                        if (ptd.id(pindex) <= 0) { continue; }

-                        // extra SoA real
-                        const int real_start_offset = PC::ParticleType::is_soa_particle ? AMREX_SPACEDIM : 0; // pure SoA: positions
-                        for (int j = real_start_offset; j < nrc; j++)
-                        {
-                    const int write_comp_offset = PC::ParticleType::is_soa_particle ? AMREX_SPACEDIM : 0; // pure SoA: skip positions
-                    const int write_comp_index = PC::NStructReal+j-write_comp_offset;
-                            if (write_real_comp[write_comp_index])
-                            {
-                                *rptr = (typename PC::ParticleType::RealType) soa.GetRealData(j)[pindex];
-                                ++rptr;
-                            }
-                        }
+                        particle_detail::rPackParticleData(ptd, pindex, rptr,
+                            write_real_comp.dataPtr());
+                        rptr += rChunkSize;
                     }
                 }

diff --git a/Tests/Particles/CheckpointRestartSOA/main.cpp b/Tests/Particles/CheckpointRestartSOA/main.cpp
index e3c544516e2..af2ce0f7a0b 100644
--- a/Tests/Particles/CheckpointRestartSOA/main.cpp
+++ b/Tests/Particles/CheckpointRestartSOA/main.cpp
@@ -150,21 +150,21 @@ void test ()
         std::snprintf(directory_path, sizeof directory_path, "%s%s", directory.c_str(), "plt00000");
         newPC.Restart(directory_path, "particle0");

-        using PType = typename MyPC::SuperParticleType;
+        using ConstPTDType = typename MyPC::ConstPTDType;

         for (int icomp=0; icomp<NReal; ++icomp)
         {
             amrex::Print() << "working on comp " << icomp << "\n";
             auto sm_new = amrex::ReduceSum(newPC,
-                [=] AMREX_GPU_HOST_DEVICE (const PType& p) -> Real
+                [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> Real
                 {
-                    return p.rdata(icomp);
+                    return ptd.rdata(icomp)[i];
                 });

             auto sm_old = amrex::ReduceSum(myPC,
-                [=] AMREX_GPU_HOST_DEVICE (const PType& p) -> Real
+                [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> Real
                 {
-                    return p.rdata(icomp);
+                    return ptd.rdata(icomp)[i];
                 });

             ParallelDescriptor::ReduceRealSum(sm_new);
@@ -178,15 +178,15 @@ void test ()
         {
             amrex::Print() << "working on comp " << icomp << "\n";
             auto sm_new = amrex::ReduceSum(newPC,
-                [=] AMREX_GPU_HOST_DEVICE (const PType& p) -> Real
+                [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> Real
                 {
-                    return p.idata(icomp);
+                    return ptd.idata(icomp)[i];
                 });

             auto sm_old = amrex::ReduceSum(myPC,
-                [=] AMREX_GPU_HOST_DEVICE (const PType& p) -> Real
+                [=] AMREX_GPU_HOST_DEVICE (const ConstPTDType& ptd, const int i) -> Real
                 {
-                    return p.idata(icomp);
+                    return ptd.idata(icomp)[i];
                 });

             ParallelDescriptor::ReduceRealSum(sm_new);


ParmParse: Add parser support for bool (AMReX-Codes/amrex#4552)

`bool` support in `queryAddWithParser`

Currently, `ParmParse::queryAddWithParser` does not support `bool` types. It would be very useful and consistent if added / completed for this.

diff --git a/Src/Base/AMReX_ParmParse.H b/Src/Base/AMReX_ParmParse.H
index 0e2aa7d37f3..c7ec4728b6d 100644
--- a/Src/Base/AMReX_ParmParse.H
+++ b/Src/Base/AMReX_ParmParse.H
@@ -1108,6 +1108,7 @@ public:
      * scalar. The return value indicates whether it's found. Note that
      * queryWithParser will be used recursively for unresolved symbols.
      */
+    int queryWithParser (const char* name, bool& ref) const;
     int queryWithParser (const char* name, int& ref) const;
     int queryWithParser (const char* name, long& ref) const;
     int queryWithParser (const char* name, long long& ref) const;
@@ -1120,12 +1121,14 @@ public:
      * symbols. If the number of elements in the input does not equal to
      * `nvals`, it's a runtime error.
      */
+    int queryarrWithParser (const char* name, int nvals, bool* ref) const;
     int queryarrWithParser (const char* name, int nvals, int* ref) const;
     int queryarrWithParser (const char* name, int nvals, long* ref) const;
     int queryarrWithParser (const char* name, int nvals, long long* ref) const;
     int queryarrWithParser (const char* name, int nvals, float* ref) const;
     int queryarrWithParser (const char* name, int nvals, double* ref) const;
-    template <typename T, std::enable_if_t<std::is_same_v<T,int> ||
+    template <typename T, std::enable_if_t<std::is_same_v<T,bool> ||
+                                           std::is_same_v<T,int> ||
                                            std::is_same_v<T,long> ||
                                            std::is_same_v<T,long long> ||
                                            std::is_same_v<T,float> ||
@@ -1146,7 +1149,8 @@ public:
      * scalar. If not, the value in `ref` will be added to the ParmParse
      * database. The return value indicates whether it's found.
      */
-    template <typename T, std::enable_if_t<std::is_same_v<T,int> ||
+    template <typename T, std::enable_if_t<std::is_same_v<T,bool> ||
+                                           std::is_same_v<T,int> ||
                                            std::is_same_v<T,long> ||
                                            std::is_same_v<T,long long> ||
                                            std::is_same_v<T,float> ||
@@ -1165,7 +1169,8 @@ public:
      * to parse the entire list of empty space separated values as a single
      * scalar. If not, it's a runtime error.
      */
-    template <typename T, std::enable_if_t<std::is_same_v<T,int> ||
+    template <typename T, std::enable_if_t<std::is_same_v<T,bool> ||
+                                           std::is_same_v<T,int> ||
                                            std::is_same_v<T,long> ||
                                            std::is_same_v<T,long long> ||
                                            std::is_same_v<T,float> ||
@@ -1183,7 +1188,8 @@ public:
      * error. If the number of elements does not equal to `nvals`, it's also
      * a runtime error.
      */
-    template <typename T, std::enable_if_t<std::is_same_v<T,int> ||
+    template <typename T, std::enable_if_t<std::is_same_v<T,bool> ||
+                                           std::is_same_v<T,int> ||
                                            std::is_same_v<T,long> ||
                                            std::is_same_v<T,long long> ||
                                            std::is_same_v<T,float> ||
@@ -1201,7 +1207,8 @@ public:
      * error. If the number of elements does not equal to `nvals`, it's also
      * a runtime error.
      */
-    template <typename T, std::enable_if_t<std::is_same_v<T,int> ||
+    template <typename T, std::enable_if_t<std::is_same_v<T,bool> ||
+                                           std::is_same_v<T,int> ||
                                            std::is_same_v<T,long> ||
                                            std::is_same_v<T,long long> ||
                                            std::is_same_v<T,float> ||
@@ -1219,7 +1226,8 @@ public:
      *
      * For unknown symbols, ParmParse database will be queried.
      */
-    template <typename T, std::enable_if_t<std::is_same_v<T,int> ||
+    template <typename T, std::enable_if_t<std::is_same_v<T,bool> ||
+                                           std::is_same_v<T,int> ||
                                            std::is_same_v<T,long> ||
                                            std::is_same_v<T,long long> ||
                                            std::is_same_v<T,float> ||
diff --git a/Src/Base/AMReX_ParmParse.cpp b/Src/Base/AMReX_ParmParse.cpp
index 7a41f187bc2..098ad89c9a7 100644
--- a/Src/Base/AMReX_ParmParse.cpp
+++ b/Src/Base/AMReX_ParmParse.cpp
@@ -718,7 +718,8 @@ squeryval (const ParmParse::Table& table,
     bool ok = is(valname, ref);
     if ( !ok )
     {
-        if constexpr (std::is_same_v<T,int> ||
+        if constexpr (std::is_same_v<T,bool> ||
+                      std::is_same_v<T,int> ||
                       std::is_same_v<T,long> ||
                       std::is_same_v<T,long long> ||
                       std::is_same_v<T,float> ||
@@ -1011,14 +1012,16 @@ bool squeryWithParser (const ParmParse::Table& table,
                        const std::string&      name,
                        T&                      ref);

-template <typename T, typename PARSER_t = std::conditional_t<std::is_integral_v<T>,
+template <typename T, typename PARSER_t = std::conditional_t<std::is_integral_v<T>
+                                                             && !std::is_same_v<bool,T>,
                                                              IParser, Parser>>
 PARSER_t
 pp_make_parser (std::string const& func, Vector<std::string> const& vars,
                 ParmParse::Table const& table, std::string const& parser_prefix,
                 bool use_querywithparser)
 {
-    using value_t =  std::conditional_t<std::is_integral_v<T>, long long, double>;
+    using value_t =  std::conditional_t<std::is_integral_v<T> && !std::is_same_v<bool,T>,
+                                        long long, double>;

     std::vector<std::string> prefixes;
     prefixes.reserve(3);
@@ -2048,6 +2051,12 @@ bool squeryarrWithParser (const ParmParse::Table& table,
 }
 }

+int
+ParmParse::queryWithParser (const char* name, bool& ref) const
+{
+    return squeryWithParser(*m_table,m_parser_prefix,prefixedName(name),ref);
+}
+
 int
 ParmParse::queryWithParser (const char* name, int& ref) const
 {
@@ -2078,6 +2087,12 @@ ParmParse::queryWithParser (const char* name, double& ref) const
     return squeryWithParser(*m_table,m_parser_prefix,prefixedName(name),ref);
 }

+int
+ParmParse::queryarrWithParser (const char* name, int nvals, bool* ref) const
+{
+    return squeryarrWithParser(*m_table,m_parser_prefix,prefixedName(name),nvals,ref);
+}
+
 int
 ParmParse::queryarrWithParser (const char* name, int nvals, int* ref) const
 {

diff --git a/Tests/ParmParse/inputs b/Tests/ParmParse/inputs
index c7b5af019b7..d109fa5579a 100644
--- a/Tests/ParmParse/inputs
+++ b/Tests/ParmParse/inputs
@@ -63,6 +63,11 @@ ny = nx

 do_this = 1

+# query bool with parser
+my_value = 10
+my_threshold = 3
+my_bool_flag = "(my_threshold < my_value) and (my_value < 100)"
+
 # queryline and getline
 my_string_line = a  b c

diff --git a/Tests/ParmParse/main.cpp b/Tests/ParmParse/main.cpp
index a367b1acaaa..1752d7f4833 100644
--- a/Tests/ParmParse/main.cpp
+++ b/Tests/ParmParse/main.cpp
@@ -143,6 +143,14 @@ int main(int argc, char* argv[])
         pp.queryAsDouble("do_that", o_do_that);
         AMREX_ALWAYS_ASSERT(!o_do_that.has_value());
     }
+    {
+        ParmParse pp;
+        bool my_bool_flag_1 = false;
+        bool my_bool_flag_2 = false;
+        pp.queryAddWithParser("my_bool_flag", my_bool_flag_1);
+        pp.query("my_bool_flag", my_bool_flag_2);
+        AMREX_ALWAYS_ASSERT(my_bool_flag_1 && my_bool_flag_2);
+    }
     {
         ParmParse pp;
         std::string line;


Adding support to memory collection of Particles and MultiFabs within a grid (AMReX-Codes/amrex#4629)

## Summary
This PR adds functions to return memory usage from `Particles` and `FabArray`/`MultiFabs` within a grid/box. The main goal is to use this information for load balancing. We aim to consider the memory allocated to a worker as a factor in the load-balancing algorithms for AMReX, so there is a need to capture specific grid/box memory information.

## Additional background

## Checklist

The proposed changes:
- [ ] fix a bug or incorrect behavior in AMReX
- [x] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX users
- [ ] include documentation in the code and/or rst files, if appropriate

diff --git a/Src/Base/AMReX_FabArray.H b/Src/Base/AMReX_FabArray.H
index 3588a8f0714..a47a9278f34 100644
--- a/Src/Base/AMReX_FabArray.H
+++ b/Src/Base/AMReX_FabArray.H
@@ -817,6 +817,10 @@ public:
     template <class F=FAB, std::enable_if_t<IsBaseFab<F>::value,int> = 0>
     void setDomainBndry (value_type val, int strt_comp, int ncomp, const Geometry& geom);

+    //! Get capacity of the FabArray
+    template <typename I, class F=FAB, std::enable_if_t<IsBaseFab<F>::value && std::is_integral_v<I> && (sizeof(I) >= sizeof(Long)), int> = 0>
+    void capacityOfFabs (LayoutData<I>& mem) const;
+
     /**
     * \brief Returns the sum of component "comp"
     *
@@ -2506,6 +2510,17 @@ FabArray<FAB>::setDomainBndry (value_type val,
     }
 }

+
+template <class FAB>
+template <typename I, class F, std::enable_if_t<IsBaseFab<F>::value && std::is_integral_v<I> && (sizeof(I) >= sizeof(Long)), int> FOO>
+void
+FabArray<FAB>::capacityOfFabs (LayoutData<I>& mem) const {
+    AMREX_ASSERT(amrex::isMFIterSafe(*this, mem));
+    for (MFIter mfi(*this, MFItInfo{}.DisableDeviceSync()); mfi.isValid(); ++mfi) {
+        mem[mfi] += static_cast<I>((*this)[mfi].nBytesOwned());
+    }
+}
+
 template <class FAB>
 template <class F, std::enable_if_t<IsBaseFab<F>::value,int> FOO>
 typename F::value_type
diff --git a/Src/Particle/AMReX_ParIter.H b/Src/Particle/AMReX_ParIter.H
index 39e5d1a9eda..459ff31757a 100644
--- a/Src/Particle/AMReX_ParIter.H
+++ b/Src/Particle/AMReX_ParIter.H
@@ -93,6 +93,8 @@ public:

     [[nodiscard]] int numNeighborParticles () const { return GetParticleTile().numNeighborParticles(); }

+    [[nodiscard]] Long capacity () const { return GetParticleTile().capacity(); }
+
     [[nodiscard]] int GetLevel () const { return m_level; }

     [[nodiscard]] std::pair<int, int> GetPairIndex () const { return std::make_pair(this->index(), this->LocalTileIndex()); }
diff --git a/Src/Particle/AMReX_ParticleContainer.H b/Src/Particle/AMReX_ParticleContainer.H
index a6a05b28702..2397a7d4823 100644
--- a/Src/Particle/AMReX_ParticleContainer.H
+++ b/Src/Particle/AMReX_ParticleContainer.H
@@ -578,6 +578,12 @@ public:

     Vector<Long> NumberOfParticlesInGrid  (int level, bool only_valid = true, bool only_local = false) const;

+    /**
+     * \brief Return capacity of memory for particles at specific grid
+     */
+    template <typename I, std::enable_if_t<std::is_integral_v<I> && (sizeof(I) >= sizeof(Long)), int> = 0>
+    void CapacityOfParticlesInGrid (LayoutData<I>& mem, int lev) const;
+
     /**
     * \brief Returns # of particles at all levels
     *
diff --git a/Src/Particle/AMReX_ParticleContainerI.H b/Src/Particle/AMReX_ParticleContainerI.H
index 5927b0be7d0..aac96dad877 100644
--- a/Src/Particle/AMReX_ParticleContainerI.H
+++ b/Src/Particle/AMReX_ParticleContainerI.H
@@ -586,6 +586,25 @@ Long ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, Cell
     return nparticles;
 }

+template <typename ParticleType, int NArrayReal, int NArrayInt,
+          template<class> class Allocator, class CellAssignor>
+template <typename I, std::enable_if_t<std::is_integral_v<I> && (sizeof(I) >= sizeof(Long)), int> FOO>
+void
+ParticleContainer_impl<ParticleType, NArrayReal, NArrayInt, Allocator, CellAssignor>::CapacityOfParticlesInGrid (LayoutData<I>& mem, int lev) const
+{
+    AMREX_ASSERT(lev >= 0 && lev < int(m_particles.size()));
+
+    AMREX_ASSERT(BoxArray::SameRefs(mem.boxArray(), ParticleBoxArray(lev)) &&
+                 mem.DistributionMap() == ParticleDistributionMap(lev));
+
+    [[maybe_unused]] Gpu::NoSyncRegion no_sync{};
+    for (ParConstIterType pti(*this, lev); pti.isValid(); ++pti)
+    {
+        int gid = pti.index();
+        mem[gid] += static_cast<I>(pti.capacity());
+    }
+}
+
 //
 // This includes both valid and invalid particles since invalid particles still take up space.
 //
diff --git a/build_docs.sh b/build_docs.sh
index 774238037f7..33e3df7ae64 100755
--- a/build_docs.sh
+++ b/build_docs.sh
@@ -7,6 +7,7 @@ cd Docs/Doxygen
 doxygen doxygen.conf &> doxygen.out
 if grep -q "warning:" doxygen.out; then
     echo "Doxygen warnings detected! Failing..."
+    cat doxygen.out
     exit 1
 fi
 cd ../..

diff --git a/Tests/CMakeLists.txt b/Tests/CMakeLists.txt
index 5e0877e6c24..3db6cccbf8d 100644
--- a/Tests/CMakeLists.txt
+++ b/Tests/CMakeLists.txt
@@ -125,9 +125,9 @@ else()
    #
    # List of subdirectories to search for CMakeLists.
    #
-   set( AMREX_TESTS_SUBDIRS Amr AsyncOut CallNoinline CLZ CommType CTOParFor DeviceGlobal Enum
-                            MultiBlock MultiPeriod ParmParse Parser Parser2 ParserUserFn Reinit
-                            RoundoffDomain SmallMatrix)
+   set( AMREX_TESTS_SUBDIRS Amr AsyncOut CallNoinline CLZ CommType CTOParFor DeviceGlobal
+                            Enum HeatEquation MultiBlock MultiPeriod ParmParse Parser
+                            Parser2 ParserUserFn Reinit RoundoffDomain SmallMatrix)

    if (AMReX_PARTICLES)
      list(APPEND AMREX_TESTS_SUBDIRS Particles)
diff --git a/Tests/HeatEquation/GNUmakefile b/Tests/HeatEquation/GNUmakefile
new file mode 100644
index 00000000000..3edbf32b70b
--- /dev/null
+++ b/Tests/HeatEquation/GNUmakefile
@@ -0,0 +1,21 @@
+# AMREX_HOME defines the directory in which we will find all the AMReX code.
+AMREX_HOME ?= ../../../../amrex
+
+USE_ASSERTION = TRUE
+
+DEBUG        = FALSE
+USE_MPI      = FALSE
+USE_OMP      = FALSE
+PRECISION	 = FLOAT
+COMP         = gnu
+DIM          = 3
+
+include $(AMREX_HOME)/Tools/GNUMake/Make.defs
+
+include Make.package
+# VPATH_LOCATIONS  += ../Source
+# INCLUDE_LOCATIONS += ../Source
+
+include $(AMREX_HOME)/Src/Base/Make.package
+
+include $(AMREX_HOME)/Tools/GNUMake/Make.rules
diff --git a/Tests/HeatEquation/Make.package b/Tests/HeatEquation/Make.package
new file mode 100644
index 00000000000..fa194a35db8
--- /dev/null
+++ b/Tests/HeatEquation/Make.package
@@ -0,0 +1,2 @@
+CEXE_sources += main.cpp
+CEXE_headers += myfunc.H
diff --git a/Tests/HeatEquation/inputs b/Tests/HeatEquation/inputs
new file mode 100644
index 00000000000..af7ab2d4a3b
--- /dev/null
+++ b/Tests/HeatEquation/inputs
@@ -0,0 +1,7 @@
+n_cell = 32
+max_grid_size = 16
+
+nsteps = 1000
+plot_int = 0
+
+dt = 1.e-5
diff --git a/Tests/HeatEquation/main.cpp b/Tests/HeatEquation/main.cpp
new file mode 100644
index 00000000000..eb7850889f5
--- /dev/null
+++ b/Tests/HeatEquation/main.cpp
@@ -0,0 +1,208 @@
+#include <AMReX_PlotFileUtil.H>
+#include <AMReX_ParmParse.H>
+#include <AMReX_MultiFabUtil.H>
+// #include <AMReX.H>
+// #include <AMReX_FabArray.H>
+
+#include "myfunc.H"
+
+using namespace amrex;
+
+int main (int argc, char* argv[])
+{
+    amrex::Initialize(argc,argv);
+
+    main_main();
+
+    amrex::Finalize();
+    return 0;
+}
+
+void main_main ()
+{
+
+    // **********************************
+    // SIMULATION PARAMETERS
+
+    // number of cells on each side of the domain
+    int n_cell;
+
+    // size of each box (or grid)
+    int max_grid_size;
+
+    // total steps in simulation
+    int nsteps;
+
+    // how often to write a plotfile
+    int plot_int;
+
+    // time step
+    Real dt;
+
+    // inputs parameters
+    {
+        // ParmParse is way of reading inputs from the inputs file
+        // pp.get means we require the inputs file to have it
+        // pp.query means we optionally need the inputs file to have it - but we must supply a default here
+        ParmParse pp;
+
+        // We need to get n_cell from the inputs file - this is the number of cells on each side of
+        //   a square (or cubic) domain.
+        pp.get("n_cell",n_cell);
+
+        // The domain is broken into boxes of size max_grid_size
+        pp.get("max_grid_size",max_grid_size);
+
+        // Default nsteps to 10, allow us to set it to something else in the inputs file
+        nsteps = 10;
+        pp.query("nsteps",nsteps);
+
+        // Default plot_int to -1, allow us to set it to something else in the inputs file
+        //  If plot_int < 0 then no plot files will be written
+        plot_int = -1;
+        pp.query("plot_int",plot_int);
+
+        // time step
+        pp.get("dt",dt);
+    }
+
+    // **********************************
+    // SIMULATION SETUP
+
+    // make BoxArray and Geometry
+    // ba will contain a list of boxes that cover the domain
+    // geom contains information such as the physical domain size,
+    //               number of points in the domain, and periodicity
+    BoxArray ba;
+    Geometry geom;
+
+    // AMREX_D_DECL means "do the first X of these, where X is the dimensionality of the simulation"
+    IntVect dom_lo(AMREX_D_DECL(       0,        0,        0));
+    IntVect dom_hi(AMREX_D_DECL(n_cell-1, n_cell-1, n_cell-1));
+
+    // Make a single box that is the entire domain
+    Box domain(dom_lo, dom_hi);
+
+    // Initialize the boxarray "ba" from the single box "domain"
+    ba.define(domain);
+
+    // Break up boxarray "ba" into chunks no larger than "max_grid_size" along a direction
+    ba.maxSize(max_grid_size);
+
+    // This defines the physical box, [0,1] in each direction.
+    RealBox real_box({AMREX_D_DECL( 0., 0., 0.)},
+                     {AMREX_D_DECL( 1., 1., 1.)});
+
+    // periodic in all direction
+    Array<int,AMREX_SPACEDIM> is_periodic{AMREX_D_DECL(1,1,1)};
+
+    // This defines a Geometry object
+    geom.define(domain, real_box, CoordSys::cartesian, is_periodic);
+
+    // extract dx from the geometry object
+    GpuArray<Real,AMREX_SPACEDIM> dx = geom.CellSizeArray();
+
+    // Nghost = number of ghost cells for each array
+    int Nghost = 1;
+
+    // Ncomp = number of components for each array
+    int Ncomp = 1;
+
+    // How Boxes are distributed among MPI processes
+    DistributionMapping dm(ba);
+
+    // we allocate two phi multifabs; one will store the old state, the other the new.
+    MultiFab phi_old(ba, dm, Ncomp, Nghost);
+    MultiFab phi_new(ba, dm, Ncomp, Nghost);
+
+    // time = starting time in the simulation
+    Real time = 0.0;
+
+    // **********************************
+    // INITIALIZE DATA
+
+    // loop over boxes
+    for (MFIter mfi(phi_old); mfi.isValid(); ++mfi)
+    {
+        const Box& bx = mfi.validbox();
+
+        const Array4<Real>& phiOld = phi_old.array(mfi);
+
+        // set phi = 1 + e^(-(r-0.5)^2)
+        amrex::ParallelFor(bx, [=] AMREX_GPU_DEVICE(int i, int j, int k)
+        {
+            Real x = (i+0.5) * dx[0];
+            Real y = (j+0.5) * dx[1];
+#if (AMREX_SPACEDIM == 2)
+            Real rsquared = ((x-0.5)*(x-0.5)+(y-0.5)*(y-0.5))/0.01;
+#elif (AMREX_SPACEDIM == 3)
+            Real z= (k+0.5) * dx[2];
+            Real rsquared = ((x-0.5)*(x-0.5)+(y-0.5)*(y-0.5)+(z-0.5)*(z-0.5))/0.01;
+#endif
+            phiOld(i,j,k) = 1. + std::exp(-rsquared);
+        });
+    }
+
+    // Write a plotfile of the initial data if plot_int > 0
+    if (plot_int > 0)
+    {
+        int step = 0;
+        const std::string& pltfile = amrex::Concatenate("plt",step,5);
+        WriteSingleLevelPlotfile(pltfile, phi_old, {"phi"}, geom, time, 0);
+    }
+
+    for (int step = 1; step <= nsteps; ++step)
+    {
+        // fill periodic ghost cells
+        phi_old.FillBoundary(geom.periodicity());
+
+        // new_phi = old_phi + dt * Laplacian(old_phi)
+        // loop over boxes
+        for ( MFIter mfi(phi_old); mfi.isValid(); ++mfi )
+        {
+            const Box& bx = mfi.validbox();
+
+            const Array4<Real>& phiOld = phi_old.array(mfi);
+            const Array4<Real>& phiNew = phi_new.array(mfi);
+
+            // advance the data by dt
+            amrex::ParallelFor(bx, [=] AMREX_GPU_DEVICE (int i, int j, int k)
+            {
+                phiNew(i,j,k) = phiOld(i,j,k) + dt *
+                    ( (phiOld(i+1,j,k) - 2.*phiOld(i,j,k) + phiOld(i-1,j,k)) / (dx[0]*dx[0])
+                     +(phiOld(i,j+1,k) - 2.*phiOld(i,j,k) + phiOld(i,j-1,k)) / (dx[1]*dx[1])
+#if (AMREX_SPACEDIM == 3)
+                     +(phiOld(i,j,k+1) - 2.*phiOld(i,j,k) + phiOld(i,j,k-1)) / (dx[2]*dx[2])
+#endif
+                        );
+            });
+        }
+
+        // update time
+        time = time + dt;
+
+        // copy new solution into old solution
+        MultiFab::Copy(phi_old, phi_new, 0, 0, 1, 0);
+
+        // Tell the I/O Processor to write out which step we're doing
+        amrex::Print() << "Advanced step " << step << "\n";
+
+        // Write a plotfile of the current data (plot_int was defined in the inputs file)
+        if (plot_int > 0 && step%plot_int == 0)
+        {
+            const std::string& pltfile = amrex::Concatenate("plt",step,5);
+            WriteSingleLevelPlotfile(pltfile, phi_new, {"phi"}, geom, time, step);
+        }
+    }
+
+    // Testing capacity function
+    amrex::LayoutData<std::size_t> mem(ba, dm);
+    phi_old.capacityOfFabs(mem);
+    phi_new.capacityOfFabs(mem);
+
+    long total_bytes = 0;
+    for (amrex::MFIter mfi(mem); mfi.isValid(); ++mfi) {
+        amrex::Print() << "Box " << mfi.index() << " uses "
+                     << mem[mfi] << " bytes\n";
+    }
+}
diff --git a/Tests/HeatEquation/myfunc.H b/Tests/HeatEquation/myfunc.H
new file mode 100644
index 00000000000..0f828c4e180
--- /dev/null
+++ b/Tests/HeatEquation/myfunc.H
@@ -0,0 +1,6 @@
+#ifndef MYFUNC_H_
+#define MYFUNC_H_
+
+void main_main ();
+
+#endif
diff --git a/Tests/Particles/NeighborList/main.cpp b/Tests/Particles/NeighborList/main.cpp
index 359f784d538..ec3f680e997 100644
--- a/Tests/Particles/NeighborList/main.cpp
+++ b/Tests/Particles/NeighborList/main.cpp
@@ -138,4 +138,13 @@ void testNeighborList ()
         auto& nlist2 = neighbors_type_2[0][std::make_pair(gid, tid)];
         nlist2.print();
     }
+
+    LayoutData<std::size_t> mem(ba, dm);
+    pc1.CapacityOfParticlesInGrid(mem, 0);
+
+    Vector<Long> particles_in_grid = pc1.NumberOfParticlesInGrid(0, false,true);
+    for (amrex::MFIter mfi(mem); mfi.isValid(); ++mfi) {
+        amrex::Print() << "Box " << mfi.index() << " uses "
+                     << mem[mfi] << " bytes and have " << particles_in_grid[mfi.LocalIndex()] << " particles\n";
+    }
 }