Digital Twin¶

Traditional electrical grid management treats data as a static snapshot a point-in-time record that quickly becomes obsolete. When you have a large CIM model representing thousands of grid assets, conventional databases force impossible trade-offs:

• Relational databases choke on graph-shaped topology queries like Connectivity Check on 10M nodes? Impossible.
• Triple stores excel at relationships but fail at real-time state management
• Time-series databases handle telemetry but lose structural context for N-1 analysis

The grid isn't a record. It's a living system constantly changing through switching operations, equipment replacements, feeder reconfigurations, and parallel planning projects. Your query surface reflects this complexity: from millisecond ID lookups to 5-second island detection across 10 million nodes.

Architectural Vision:¶

We must recognize a fundamental truth: no single database technology can optimally serve all query categories. Instead, the digital twin must present different query surfaces optimized for different access patterns—all synchronized through the event stream.

Users should never be aware of the underlying storage complexity. To the domain expert or external system, the digital twin presents a single, coherent query interface that abstracts away the multi-model architecture.

Consider a fusion query: "Find all open breakers within 500 meters of coordinate X that were added after 2024." Under the hood, the system decomposes this request across purpose-built stores:

• Elasticsearch handles the geospatial + temporal filter for speed
• Triple store validates connectivity impact when needed
• Orleans confirms the real-time, authoritative state of each breaker

This orchestration delivers a unified result without the user ever seeing "a database." Instead, they interact with a grid they can:

• Query historically (time travel to any point)
• Modify speculatively (what-if scenarios without impact)
• Monitor in real-time (current operational state)

This is the vision: a multi-model, event-sourced, query-centric digital twin where the user sees one coherent grid.

System Overview¶

The system follows a Reactive Ingestion Pipe pattern. It transforms static CIM XML/RDF or JSON/LD BLOBs into a living, distributed graph of Aggregate Roots (Orleans Grains) while maintaining multiple optimized query surfaces (like Elasticsearch for inventory/geo, Triple Store for topology/semantics, Event Store for temporal queries).

This is a sophisticated and highly effective approach for managing Digital Twins. Using an Event Store (EventDB) for change management while maintaining the "Live Network Status" in Microsoft Orleans aligns perfectly with the Event Sourcing and Snapshotting patterns.

Technology Mapping¶

High-Level Architecture¶

Core Design Principles¶

The system is built on five fundamental principles that guide every architectural decision:

• Eventual Consistency— Query stores lag milliseconds behind authoritative grains. This allows high-throughput writes to Orleans Grains without blocking reads, while still delivering coherent results within acceptable time windows.
• Versioned Baseline— The Large RDF model serves as the starting point, not an immutable artifact. Accepted deltas (project deployments, approved changes) merge into the baseline over time. Previous versions remain accessible for time travel and audit. The baseline evolves; nothing is lost.
• Delta-Driven State— Every change is captured as an event. No state is ever overwritten. This enables time travel, audit trails, and what-if scenarios without duplicating the complete model.
• Purpose-Built Query Surfaces— Each store optimizes for specific query categories: Elasticsearch for inventory and geospatial, Triple Store for topology and semantics, Event Store for temporal queries. No single database can optimally serve all access patterns.
• Aggregate Root-Based Authority— Each aggregate root (Transformer, Breaker, Substation, etc.) is owned by a single Orleans Grain. The grain maintains strong consistency within its aggregate boundary, including owned child entities. Cross-aggregate relationships use shared identifiers (mRID), not direct references. This eliminates distributed locking, provides natural sharding, and ensures strong consistency where it matters most.

The Data Ingestion Lifecycle¶

The system processes data through a multi-stage pipeline, ensuring that massive baseline files do not become bottlenecks.

1. Ingestion & Decomposition¶

Instead of storing large files as monolithic BLOBs, we decompose them into granular events.

• Streaming Ingestion: Using dotnetRDF and XmlReader, the system parses large files without loading them into memory.
• Event-Sourcing: Every CimDefinition from CIM Full Model is treated as a Genesis Event for a specific mRID. These are stored in an Event Store.
• Delta-Driven State: Future changes CimDescriptions from CIM Difference Model are appended to the same mRID stream, allowing for perfect auditability and time travel.

2. The Reactive Pipeline¶

Once data is committed to the Event Store, Projectors propagate this data to specialized "Read Models".

• Elasticsearch Projector: The event is flattened into a JSON document. Attributes like name or ratedVoltage are indexed, and complex technical parameters are stored as nested objects for high-speed searching.
• Triple Store Projector: The projector extracts relationships (e.g., ConnectivityNode to Terminal) and updates the RDF graph to support complex SPARQL connectivity queries.
• Orleans Projector: Activates specific Grain (Aggregate Root) associated with the mRID. If the grain is inactive, it "wakes up," reads its stream, and updates its in-memory state (backed by Redis) to reflect the authoritative operational status.

Architectural Strengths of the Reactive Pipeline¶

The event-driven nature of this architecture provides critical resilience and flexibility for managing a large-scale Digital Twin. By utilizing the Event Store as the Source of Truth, the system achieves high Fault Tolerance; if a target system such as Elasticsearch or the Triple Store crashes, it can resume operations without data loss by simply re-reading events from its last known Checkpoint.

This design also ensures total System Decoupling, as the processing speed of the Triple Store does not impact the performance of Orleans or Elasticsearch; each projector is an independent subscriber that consumes the event stream at its own optimal pace. Furthermore, the architecture supports seamless Replay & Rebuild capabilities. Since every change is recorded as an immutable event, a new search engine or analytical tool can be introduced at any time and populated from scratch by replaying the entire grid history from the Event Store.

Baseline Optimization: Snapshotting and Re-basing¶

In an Event Sourcing architecture, Snapshotting and Baseline Re-basing are critical for maintaining system performance over time. As thousands of project deltas are applied over the initial huge baseline, reconstructing the state from the very beginning becomes computationally expensive and inefficient. To manage this, the system employs a multi-tiered re-basing strategy:

1. Orleans Snapshot Mechanism (The Live Baseline)¶

Orleans Grains (Aggregate Roots) automatically apply every delta to their internal state.

- **The Process:** When `WriteStateAsync()` is triggered in Orleans, the current _most up-to-date and merged_ model is written to a Snapshot Store, such as `Redis`.
- **The Result:** When a Grain reactivates, it no longer needs to scan the original huge file or historical deltas. Instead, it loads its last saved _current state (Snapshot)_ in a single operation, serving as your live operational baseline.

2. "As-Built" Physical Re-basing¶

Periodically—such as monthly or after a specific number of major project deployments—the Official Baseline is updated to prevent event stream bloat.

- **Merging:** A background worker reads all cumulative events (the original baseline plus all subsequent deltas) from the Event Store and merges them into a single, unified RDF Full Model.
- **Version Control:** This new consolidated file (e.g., now 3.2 GB) is uploaded to BLOB storage as new Version.
- **Cut-off Point:** A _Marker_ is placed in the Event Store, indicating that all events prior to this point are now encapsulated within the new physical baseline file.

3. Event Store Snapshots¶

For granular stream management, a SnapshotCreated event can be emitted for each specific mRID stream within the Event Store.

- This event contains the entire history of that specific entity (baseline + deltas) in a single packet.
- Once this event is recorded, all preceding triple and delta events for that `mRID` can be archived, significantly reducing the primary database load and improving retrieval speeds.

4. Query Store Compaction (Elasticsearch & Triple Store)¶

Unlike Orleans, the Triple Store and Elasticsearch are inherently designed to hold the latest state.

- When a `CimDifferenceModel` is processed, the Triple Store performs `DELETE` and `INSERT` operations, meaning the graph naturally evolves into the _new baseline_ state.
- For these systems, _re-basing_ primarily involves background compaction and index optimization to clean up old data fragments and maintain peak query performance.

Handling Real-Time Telemetry (Hot vs. Cold Paths)¶

The architecture distinguishes between Engineering State (structural changes) and Operational State (real-time signals).

- **Cold Path (Engineering):** Baseline and Delta models flow through the Event Store. This defines the _bones_ of the network.
- **Hot Path (Telemetry):** High-frequency SCADA signals (e.g., Breaker status, Voltage) bypass the Event Store and flow directly into Orleans Grains via a Live Bus (MQTT).
- **Convergence:** The Orleans Grain acts as the _Single Source of Truth,_ merging the structural engineering data with the real-time operational status in RAM.