On this page
Tuesday 1st Cleaning up August notes Investigating "non blocking" queries in Myriad.ECS, to allow for more chaining of multithreaded work before waiting on itAdding non-blocking to some query types (blocking by default) Updating Myriad/Unity integration package Wednesday 2nd Implementing non-blocking queries in Myriad/Unity integration package Using in main projectFixing breakage from moved namespaces Vastly reduced sensor system time! (Roughly 1/5th the time) Rechecking RADAR system maths Upgrading project to Unity 6.5 Thursday 3rd Continuing upgrade processJob safety issueCarefully reviewed, can't spot any issues Works in Unity 6.4, broken in Unity 6.5 and 6.6 Creating minimal repro in 6.6 It's a Unity regression, building test project to report bug Fixing IMPACT scatter graph VFX Reimported uiextensions package, it was used extensively in IMPACT Friday 4th Removing soe usages of legacy input system (due to be deprecated soon) Upgrading packagesLots of warnings from Sonity Refactoring IMPACTRemoving use of VFXType attribute for graphics buffer. This has broken twice on engine upgrades, removing it makes the next update easier. Expanding scale on RADAR test scene to astronomical rangesJob safety error with > 1024 trackablesAlready handled this (using Interlocked) but the safety system needs convincing Monday 7th Testing sensor system with huge number of objects (20,000)It gets slow! Slow systems:CopyScenePositionToUnityTransformCan't be moved to job (interacts with scene Transform) SensorTrack.UpdateWorldPositionCacheCan't be moved to job (fetches data from other entities in ECS) Cross reference systems PhasedArrayRadarTrackSystem3 Jobbifying cross reference jobs CalculateRangeFromTrackingToTrackedChunk based: ~1100us @ 24,000 entities Jobbified: ~130us @ 24,000 entities Tuesday 8th Adding complete sensor system test coverage Wednesday 9th Finishing off sensor testing Exposing scheduled job count in query job handleFixed a major bug with job queries not always respecting the QueryDescription filter! Thursday 10th Cleaning up testsSimplifying setup of some of the more complex ones Creating a "phased chunk query" which skips some chunks in a queryCan be used for slower processing of large number of items (e.g. tracks) This can be turned into a more general chunk filtering system Adding chunk filtering to Myriad.ECS Friday 11th Creating adaptive filtering system which automatically reduces work as entity count increases Using a static (non-adaptive) filter in scanning Using a static filter in trackingCan't do filtering for most of the work, so it barely helps here Removing scan system interlocked counting (probably slow)Replacing with a per-sensor-per-thread counter, with an accumulation afterwards. This avoids interlocked entirely (less false sharing) Building a system to assign persistent IDs to sensorsThis work can be shared between all sensor types It can also be moved ahead of the parallel work, reducing the risk of stalling the job pipeline Fixed tests to use new IDs Reverted to Interlocked increment, it doesn't seem to be any slower and is significantly simpler. Monday 14th Designing inter-platform comms (sharing tracks)There's a major issue with sharing: if we cross reference tracks (assuming they're independent) then even an inert platform with a good track (because it was shared to the platform) can improve track quality! Might have to move to proper state tracking?Kalman state update Orbital propagation for tracksThis is very very costly if we do n-body! use Kepler instead?If so, track can store (pos, vel, time) and we can propagate it forward to any future time instantly Building a Kalman test project to learn how they work Tuesday 15th Finishing off support infrastructure (various size 6 matrices) Building a test scene with a single moving object and some sensorsMoving object Platform SensorsOptical (Direction and weak range) Doppler (Radial velocity) Investigating commsCannot just merge Kalman states - would cause double counting of information.Covariance Intersection for combining when correlation is unknownSplit Covariance Intersection, enhancement of plain CI. keeps track of what part of covariance is definitely unique (from local sensors) Wednesday 16th Adding some sensor visualisations Implementing comms linksBasic covariance intersection Thursday 17th Implementing Split CovarianceMaintains a separate independent and dependent covariance matrix pair. Storing covariance from local sensor updates and remote data with unknown provenance. This turns out to still be a pain to use - need to know the provenance of data to know what is potentially correlated or not. Doesn't really help in a disorganised/gossip network. Reverting back to simple covariance intersection Built a simple KalmanState6D which neatly wraps all operations Friday 18th Cleaning up covariance intersection codeImproving omega (merge weight) selection General clean upPushing for 100% test coverage Remove unused Kalman helpers Moving prototype code into a package Saturday 19th Creating Unity package, porting some of the mathematics code Sunday 20th Completed porting into package Converting package to double precision Optimising some matrix ops Monday 21st Optimising some more matrix ops Researching UKF (unscented Kalman filter), possibly handles orbital dynamics better Implementing cholesky (needed for UKF) Implementing UKF core functionsmake_sigma_pointsGetWeightedMeanGetWeightedCovarianceGenerating tests against reference python impl Thinking about tracksDon't want to integrate all tracks every tick Tracks can be converted to kepler when "cold" Cold (kepler) to hot (Verlet) conversion is tricky Possible 3 stage model:Cold: No updates, just store last known state Loose: Updated and integrated every frame Attached: Updated every frame, stored as an offset from true positionConvert from attached to loose as soon as an engine burn happens Tuesday 22nd More track design: [Sensor Tracks v3](ImplementationDetails/Sensors/Sensor Tracks v3) Designing "oracle" update for Kalman state - reading true pos/vel deltas and updating state without leaking info Optimised all matmul operations to use explicit SIMD mad instead of multiply and addition chains Deep dive on Burst compilation of maths code Wednesday 23rd Thursday 24th Investigating potential bias in measurements Adding better modelling of acceleration (optional, we can just use linear velocity model if necessary)Feeding it in as a control input Testing Euclidean distance metric in oracle state controller Experimenting with player-facing lock state Friday 25th Replacing schur_inverse with cholesky_inverse, faster and more stableInconsistent with baseline full inverse Fixed a bug in normal inverse Adding some safety checks properly handling a failure to invertShould never happen unless a sensor is buggy Switching KalmanState6D to lazily invert matrix when needed instead of eagerly on update Optimising storage by only storing half of the covariance matrix (it's symmetric)Lots of new maths ops (mostly matmul of various shapes) Fixing endless call sites Applying the same optimisation to information matrix Monday 28th Adding more special matrix types, to elide work in matmul where it's not requireddoubleSymmetric6x6doubleSymmetric3x3SpecialMatrixFConstantVelocitySpecialMatrixQConstantVelocitySpecialMatrixHPosition Pushed up to 100% test coverage Imported Kalman package into main projectCreating Kalman2 test scene Systems todo: Tuesday 29th Wednesday 30th Investigating frame-rate independent sensor updatesJust dividing observation matrix (R) by delta time does a good enough job Implementing it in Kalman6D library Using it in main project Port AESA Radar from old sensor system Adding a flag to track lifecycle creator, indicating if any flag were created. Using this to skip all track init systems. Extracting helper functions for building R matrix (common pattern from optical and RADAR code)