feat: Introduce Hybrid Logical Clocks in protobuf, update architecture documentation, and add HLC test cases.

.gitignore

@@ -0,0 +1,17 @@
+# Rust / Cargo
+/target/
+Cargo.lock
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Environment
+.env
+.env.local

Cargo.toml

@@ -0,0 +1,48 @@
+[workspace]
+resolver = "2"
+members = [
+    "lattice-core",
+    "lattice-net",
+    "lattice-store",
+]
+
+[workspace.package]
+version = "0.1.0"
+edition = "2021"
+license = "MIT"
+
+[workspace.dependencies]
+# Workspace crates
+lattice-core = { path = "lattice-core" }
+lattice-net = { path = "lattice-net" }
+lattice-store = { path = "lattice-store" }
+
+# Networking (Iroh)
+iroh = "0.95"
+iroh-gossip = "0.95"
+
+# Cryptography
+ed25519-dalek = { version = "2", features = ["rand_core"] }
+rand = "0.8"
+
+# Serialization
+prost = "0.13"
+prost-types = "0.13"
+prost-build = "0.13"
+
+# Async runtime
+tokio = { version = "1", features = ["full"] }
+
+# Utilities
+thiserror = "2"
+tracing = "0.1"
+bytes = "1"
+
+# Testing
+tokio-test = "0.4"
+
+[workspace.lints.rust]
+unsafe_code = "forbid"
+
+[workspace.lints.clippy]
+all = "warn"

docs/architecture.md

@@ -0,0 +1,112 @@
+# Architecture
+
+## Ideas
+
+- SigChains Ed25519-signed, hash-chained append-only logs per node. Trust via local signature verification.
+- Log-Based State: KV store derived by replaying entries. Watermarks enable safe log pruning + snapshots.
+- Offline-First: Iroh for networking. Vector clocks identify missing entries on reconnect—converges mathematically.
+- Full Replication: All nodes keep all logs until watermark consensus, then prune and snapshot.
+
+## Concepts
+
+- Transitive Pairing. Nodes can introduce new nodes to the mesh.
+
+## Stack
+
+- rust
+- iroh
+- prost protocol buffers
+
+### Bootstrap
+
+- New peers request a full state snapshot from their first connection.
+- The snapshot allows them to skip replaying the entire log history.
+- After bootstrap, the node receives incremental updates via gossip.
+
+### Networking
+
+- Designed for mobile clients that may only sync a few times per day.
+- When peers connect, they exchange vector clocks to identify missing entries.
+- Missing entries are fetched via unicast.
+- MAX_DRIFT should be generous (e.g., hours) to accommodate sleeping devices.
+
+Networking modes:
+- Active (servers/laptops on power): Frequent gossip broadcasts, proactive sync.
+- Low-power (mobile/battery): Pull-based sync on wake. Query peers instead of relying on push gossip.
+
+## Parts
+
+### Nodes
+
+- Identified by their Ed25519 public key.
+- Private key stored locally in `identity.key` (not replicated).
+- Node data stored in KV:
+  - `/nodes/{pubkey}/info` = static metadata (name, added_by, added_at)
+  - `/nodes/{pubkey}/status` = `active` | `dormant` | `disabled`
+  - `/nodes/{pubkey}/role` = `server` | `device` (optional, hints sync priority)
+- Inviting a node = writing entries to `/nodes/{pubkey}/...`.
+- Liveness: Each node tracks `last_seen` locally (from watermark gossip). UI alerts if a peer hasn't been seen for threshold (e.g., 30 days). User decides to mark dormant/disabled.
+- Status effects:
+  - `active`: Normal sync participant, blocks watermark until acknowledged.
+  - `dormant`: Excluded from watermark consensus, can be reactivated.
+  - `disabled`: Permanently removed from mesh.
+- Sync priority: Low-power clients prefer peers marked as `server` or recently active.
+
+### Data Model
+
+- Keys are flat strings using path conventions (e.g., `/nodes/{pubkey}`, `/config/sync/interval`).
+- Prefix queries via string matching (sorted map enables efficient range scans).
+- State is computed by replaying `Put`/`Delete` operations from all authors.
+- Entry ordering: by HLC timestamp, then by author ID as tiebreaker.
+- Conflicts resolved by last-write-wins (using the ordering above).
+
+### Timestamps (Hybrid Logical Clocks)
+
+Timestamps use HLC `<wall_time, counter>` with Causal Clamping:
+
+- Each entry includes an HLC and a reference to its parent (prev_hash).
+- Standard HLC: `new_hlc = max(local_wall_clock, max_seen_hlc + 1)`.
+- On receive: if `entry.hlc > local_wall_clock + MAX_DRIFT`, clamp to `parent.hlc + 1`.
+- All nodes compute the same clamped time from the parent (deterministic).
+- Genesis entries (no parent) with future timestamps are dropped.
+
+Pre-flight check (before signing):
+- Compare local_clock to max_peer_hlc (from recent gossip/entries).
+- If `local_clock > max_peer_hlc + MAX_DRIFT`, use `max_peer_hlc + 1` instead.
+- This catches future-clock nodes before they poison the log.
+
+Authors apply their own entries through the standard receive path to ensure consistent clamping.
+
+### Storage
+
+Each node stores logs as one file per author:
+
+```
+data/
+├── identity.key               # Local node's Ed25519 private key
+├── logs/
+│   └── {author_id_hex}.log    # Append-only SignedEntry stream per author
+└── state.db                   # redb: KV snapshot + vector clocks + indexes
+```
+
+- Logs: Append-only binary files per author, containing serialized `SignedEntry` messages.
+- State DB (redb): Combined KV state, vector clocks, and indexes. Updated as entries are applied.
+
+#### state.db Tables (redb)
+
+```
+Table           Key                     Value           Purpose
+─────────────────────────────────────────────────────────────────────────────
+kv              String (path)           Vec<u8>         Replicated key-value data
+vector_clocks   [u8; 32] (author_id)    u64 (max seq)   Track sync state per author
+entry_index     (author_id, seq)        u64 (offset)    Fast entry lookup by position
+meta            String                  Vec<u8>         System metadata (own_seq, watermark, etc.)
+```
+
+### Watermarks
+
+- Nodes gossip their watermarks periodically (throttled).
+- A watermark is a vector clock: how much of each author's log the node has seen.
+- All nodes keep all logs (own + others) for redundancy until watermark consensus.
+- Once all peers have acknowledged entries, they can be pruned and replaced by the snapshot.
+- If a node is offline too long, it re-bootstraps with a fresh snapshot when it reconnects.

docs/testcases.md

@@ -0,0 +1,37 @@
+# Test Cases
+
+## Timestamp / HLC
+
+### Time-traveling node applies own entry
+- Node X has clock at year 2050
+- Node X creates entry, signs, broadcasts
+- All nodes (including X) should clamp to parent.hlc + 1
+- Verify: X's state.db matches other nodes' state.db
+- Failure mode: X applies using 2050, others use 101 → divergence
+
+### Clamping with no parent (genesis entry)
+- Node X creates first-ever entry with future timestamp
+- All nodes should DROP the entry (no parent to anchor to)
+- Verify: entry is not applied anywhere
+
+### Out-of-order entry arrival
+- Entry B (hlc=91) arrives after Entry A (hlc=100)
+- Both write to same key
+- Verify: A's value wins (LWW with timestamp tracking)
+- Verify: no rollback needed, just comparison on apply
+
+### Clock drift detection
+- Node consistently sees its entries clamped
+- Verify: UI alerts user about clock being ahead
+
+### Clock in past (Pi without RTC, boots at 1970)
+- Node X has clock at 1970
+- Node X receives entries from peers with HLC around 2024
+- Standard HLC: X uses max(1970, peer_hlc + 1) = peer_hlc + 1
+- Verify: X's entries slot in correctly (no special handling needed)
+
+### Pre-flight peer sanity check (future clock)
+- Node X has clock at 2050
+- Before creating entry, X compares local_clock to max_peer_hlc
+- If local_clock > max_peer_hlc + MAX_DRIFT, use max_peer_hlc + 1
+- Verify: X's entry uses sane timestamp, all nodes agree

lattice-core/Cargo.toml

@@ -0,0 +1,16 @@
+[package]
+name = "lattice-core"
+description = "Core types for Lattice: nodes, sigchains, entries, and vector clocks"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[dependencies]
+ed25519-dalek = { workspace = true }
+rand = { workspace = true }
+prost = { workspace = true }
+thiserror = { workspace = true }
+bytes = { workspace = true }
+
+[dev-dependencies]
+tokio-test = { workspace = true }

lattice-core/src/entry.rs

@@ -0,0 +1,9 @@
+//! Log entries (atomic operations)
+
+/// An atomic, batched operation in the sigchain.
+///
+/// Entries are the fundamental unit of change in Lattice.
+/// The KV store is a "view" generated by replaying these entries.
+pub struct Entry {
+    // TODO: operation data, signature, prev_hash
+}

lattice-core/src/lib.rs

@@ -0,0 +1,17 @@
+//! Lattice Core
+//!
+//! Core types for the Lattice distributed mesh:
+//! - **Node**: Identity with Ed25519 keypair
+//! - **SigChain**: Append-only cryptographically signed log
+//! - **Entry**: Atomic operations in the log
+//! - **VectorClock**: Causality tracking for reconciliation
+
+pub mod node;
+pub mod sigchain;
+pub mod entry;
+pub mod vector_clock;
+
+pub use node::Node;
+pub use sigchain::SigChain;
+pub use entry::Entry;
+pub use vector_clock::VectorClock;

lattice-core/src/node.rs

@@ -0,0 +1,30 @@
+//! Node identity and cryptographic keys
+
+use ed25519_dalek::{SigningKey, VerifyingKey};
+use rand::rngs::OsRng;
+
+/// A node in the Lattice mesh.
+///
+/// Each node has an Ed25519 keypair used for signing sigchain entries
+/// and establishing trust within the network.
+pub struct Node {
+    signing_key: SigningKey,
+}
+
+impl Node {
+    /// Generate a new node with a random keypair.
+    pub fn generate() -> Self {
+        let signing_key = SigningKey::generate(&mut OsRng);
+        Self { signing_key }
+    }
+
+    /// Get the node's public key (identity).
+    pub fn public_key(&self) -> VerifyingKey {
+        self.signing_key.verifying_key()
+    }
+
+    /// Get the signing key for creating signatures.
+    pub fn signing_key(&self) -> &SigningKey {
+        &self.signing_key
+    }
+}

lattice-core/src/sigchain.rs

@@ -0,0 +1,20 @@
+//! Cryptographic SigChain (append-only signed log)
+
+/// An append-only log where each entry is cryptographically signed
+/// and hash-linked to the previous entry.
+pub struct SigChain {
+    // TODO: entries, hash chain
+}
+
+impl SigChain {
+    /// Create a new empty sigchain.
+    pub fn new() -> Self {
+        Self {}
+    }
+}
+
+impl Default for SigChain {
+    fn default() -> Self {
+        Self::new()
+    }
+}

lattice-core/src/vector_clock.rs

@@ -0,0 +1,41 @@
+//! Vector clocks for causality tracking
+
+use std::collections::HashMap;
+
+/// A vector clock for tracking "how much" of each node's log has been seen.
+///
+/// Used during reconciliation to identify missing entries between peers.
+pub struct VectorClock {
+    clocks: HashMap<[u8; 32], u64>,
+}
+
+impl VectorClock {
+    /// Create a new empty vector clock.
+    pub fn new() -> Self {
+        Self {
+            clocks: HashMap::new(),
+        }
+    }
+
+    /// Get the clock value for a node (returns 0 if not present).
+    pub fn get(&self, node_id: &[u8; 32]) -> u64 {
+        self.clocks.get(node_id).copied().unwrap_or(0)
+    }
+
+    /// Set the clock value for a node.
+    pub fn set(&mut self, node_id: [u8; 32], value: u64) {
+        self.clocks.insert(node_id, value);
+    }
+
+    /// Increment the clock for a node.
+    pub fn increment(&mut self, node_id: [u8; 32]) {
+        let current = self.get(&node_id);
+        self.set(node_id, current + 1);
+    }
+}
+
+impl Default for VectorClock {
+    fn default() -> Self {
+        Self::new()
+    }
+}

lattice-net/Cargo.toml

@@ -0,0 +1,18 @@
+[package]
+name = "lattice-net"
+description = "Networking layer for Lattice using Iroh"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[dependencies]
+lattice-core = { workspace = true }
+iroh = { workspace = true }
+iroh-gossip = { workspace = true }
+tokio = { workspace = true }
+thiserror = { workspace = true }
+tracing = { workspace = true }
+bytes = { workspace = true }
+
+[dev-dependencies]
+tokio-test = { workspace = true }

lattice-net/src/gossip.rs

@@ -0,0 +1,3 @@
+//! Gossip protocol for broadcasting changes
+
+// TODO: Implement gossip using iroh-gossip

lattice-net/src/lib.rs

@@ -0,0 +1,8 @@
+//! Lattice Networking
+//!
+//! Networking layer using Iroh:
+//! - **Gossip**: Broadcasting changes across the mesh
+//! - **Unicast**: Point-to-point communication for reconciliation
+
+pub mod gossip;
+pub mod unicast;

lattice-net/src/unicast.rs

@@ -0,0 +1,3 @@
+//! Unicast communication for direct peer-to-peer messaging
+
+// TODO: Implement unicast using iroh

lattice-store/Cargo.toml

@@ -0,0 +1,15 @@
+[package]
+name = "lattice-store"
+description = "Log-based KV store with snapshots and watermarks"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[dependencies]
+lattice-core = { workspace = true }
+tokio = { workspace = true }
+thiserror = { workspace = true }
+tracing = { workspace = true }
+
+[dev-dependencies]
+tokio-test = { workspace = true }

lattice-store/src/lib.rs

@@ -0,0 +1,12 @@
+//! Lattice Store
+//!
+//! Log-based Key-Value store:
+//! - State is a "view" generated by replaying entries
+//! - Watermarks for agreeing on safe compaction points
+//! - Snapshots for replacing old logs
+
+pub mod store;
+pub mod snapshot;
+pub mod watermark;
+
+pub use store::Store;

lattice-store/src/snapshot.rs

@@ -0,0 +1,6 @@
+//! Snapshots for log compaction
+
+/// A verified snapshot that can replace old log entries.
+pub struct Snapshot {
+    // TODO: snapshot data, watermark, verification
+}

lattice-store/src/store.rs

@@ -0,0 +1,35 @@
+//! The main KV store
+
+use std::collections::HashMap;
+
+/// A log-based Key-Value store.
+///
+/// The store is a dynamic "view" generated by replaying entries.
+pub struct Store {
+    data: HashMap<Vec<u8>, Vec<u8>>,
+}
+
+impl Store {
+    /// Create a new empty store.
+    pub fn new() -> Self {
+        Self {
+            data: HashMap::new(),
+        }
+    }
+
+    /// Get a value by key.
+    pub fn get(&self, key: &[u8]) -> Option<&[u8]> {
+        self.data.get(key).map(|v| v.as_slice())
+    }
+
+    /// Set a value (this would normally go through the log).
+    pub fn set(&mut self, key: Vec<u8>, value: Vec<u8>) {
+        self.data.insert(key, value);
+    }
+}
+
+impl Default for Store {
+    fn default() -> Self {
+        Self::new()
+    }
+}

lattice-store/src/watermark.rs

@@ -0,0 +1,9 @@
+//! Watermarks for agreeing on safe compaction points
+
+/// A watermark representing a safe cut-off point for log compaction.
+///
+/// Nodes use watermarks to agree on which entries can be safely
+/// deleted and replaced with snapshots.
+pub struct Watermark {
+    // TODO: watermark data
+}

proto/lattice.proto

@@ -0,0 +1,66 @@
+syntax = "proto3";
+
+package lattice;
+
+// 1. The Wrapper (What flies over the wire)
+message SignedEntry {
+  // The serialized bytes of the 'Entry' message.
+  // We keep this as raw bytes so the signature verification is stable.
+  bytes entry_bytes = 1;
+  
+  // Ed25519 Signature of 'entry_bytes'
+  bytes signature = 2;
+  
+  // Public Key of the author (32 bytes)
+  bytes author_id = 3;
+}
+
+// 2. The Log Entry (The Atomic Unit)
+message Entry {
+  // Versioning allows us to change the format radically later if needed
+  uint32 version = 1;
+
+  // Ordering Metadata
+  bytes prev_hash = 2;   // Link to previous entry (32 bytes)
+  uint64 seq = 3;        // Monotonic sequence number
+  HLC timestamp = 4;     // Hybrid Logical Clock
+
+  // The Batch of Operations
+  repeated Operation ops = 5;
+}
+
+// Hybrid Logical Clock
+message HLC {
+  uint64 wall_time = 1;  // Unix timestamp (ms)
+  uint32 counter = 2;    // Logical counter for same-ms ordering
+}
+
+// 3. The Operation (The Change)
+message Operation {
+  // "oneof" is how Protobuf handles Rust Enums
+  oneof op_type {
+    PutOp put = 1;
+    DeleteOp delete = 2;
+    // Future: MergeOp merge = 3;
+  }
+}
+
+message PutOp {
+  string key = 1;
+  bytes value = 2; // Raw bytes allows storing images, JSON, binary, etc.
+}
+
+message DeleteOp {
+  string key = 1;
+}
+
+// 4. The Sync Handshake (Vector Clocks)
+message SyncState {
+  repeated Frontier frontiers = 1;
+  HLC sender_hlc = 2;  // Sender's current clock (for peer time awareness)
+}
+
+message Frontier {
+  bytes author_id = 1;  // Ed25519 public key (32 bytes)
+  uint64 max_seq = 2;   // Highest sequence number seen from this author
+}