Cluster Analysis: main_run_n100k (n=100,000)¶
Analyzes a real run captured with the cluster event capture feature (see
but_cluster_event_capture_feature in project memory) - unlike
analyze_stability_clusters.ipynb's target run, this one has EXACT cluster
tracking and EXACT particle IDs recorded live by the engine itself
(cluster_events/cluster_events_index.csv), not a post-hoc reconstruction
from centroid proximity and nearest-baked-frame guessing. Every number and
every particle ID below is exact, not approximate.
import numpy as np
import pandas as pd
from pathlib import Path
import sys
sys.path.insert(0, str(Path.cwd()))
import but_binary_io as bio
RUN_DIR = Path("/Users/ian/Documents/sim/cluster_event_capture_n100k/main_run_n100k")
header = bio.read_header(RUN_DIR / "simulation_data.bin")
print(header)
clusters_df = pd.read_csv(RUN_DIR / "stability_clusters.csv")
print(f"{len(clusters_df):,} cluster-observation rows, steps {clusters_df['step'].min()}-{clusters_df['step'].max()}")
SimHeader(num_posons=50000, num_negons=50000, initial_width=126.0, total_frames=51) 584,254 cluster-observation rows, steps 152-4999
1. The headline number¶
biggest = clusters_df.loc[clusters_df["size"].idxmax()]
pct = 100 * biggest["size"] / header.total_particles
print(f"Largest cluster observed: {int(biggest['size']):,} particles at step {int(biggest['step'])}")
print(f"That's {pct:.2f}% of all {header.total_particles:,} particles in one bonded structure -"
f" proportionally almost 3x the {5.0:.1f}% found in the earlier n=250,000 run.")
Largest cluster observed: 14,385 particles at step 4730 That's 14.38% of all 100,000 particles in one bonded structure - proportionally almost 3x the 5.0% found in the earlier n=250,000 run.
2. Exact cluster tracks from the live engine¶
cluster_events_index.csv is written directly by SimulationController's
live cross-step tracking (Octree.ClusterTracking) - real, exact, matching
this project's own already-validated real-hardware testing, not a notebook
reconstruction.
events = bio.load_cluster_events_index(RUN_DIR)
events_df = pd.DataFrame(events)
print(f"{len(events_df):,} events captured")
print(events_df[["stepSpan", "peakSize"]].describe())
print("\n=== Top 5 by peak size ===")
print(events_df.sort_values("peakSize", ascending=False).head(5)[["eventIndex", "trackId", "category", "paddedStartStep", "lastLiveStep", "stepSpan", "peakSize"]].to_string(index=False))
print("\n=== Top 5 by step span (longest lasting) ===")
print(events_df.sort_values("stepSpan", ascending=False).head(5)[["eventIndex", "trackId", "category", "paddedStartStep", "lastLiveStep", "stepSpan", "peakSize"]].to_string(index=False))
2,000 events captured
stepSpan peakSize
count 2000.000000 2000.000000
mean 2372.313500 147.046000
std 794.619624 443.094839
min 1246.000000 4.000000
25% 1752.750000 34.000000
50% 2211.500000 67.000000
75% 2839.250000 123.250000
max 4853.000000 14385.000000
=== Top 5 by peak size ===
eventIndex trackId category paddedStartStep lastLiveStep stepSpan peakSize
1727 39134 large 3492 4999 1508 14385
606 11896 large 2334 4999 2666 6044
1719 38686 large 3479 4999 1521 3853
1717 39232 large 3495 4999 1505 3751
1695 38574 large 3475 4999 1525 3625
=== Top 5 by step span (longest lasting) ===
eventIndex trackId category paddedStartStep lastLiveStep stepSpan peakSize
0 0 long_duration 147 4999 4853 5
1 11 long_duration 233 4999 4767 6
2 18 long_duration 245 4999 4755 15
3 20 long_duration 250 4999 4750 22
4 29 long_duration 272 4999 4728 4
3. Resolving the headline cluster to exact particle IDs¶
No reconstruction needed - stability_clusters.csv's memberIds column at
the exact peak step gives the real, complete membership directly.
member_ids = [int(x) for x in str(biggest["memberIds"]).split("|")]
print(f"Exact membership at step {int(biggest['step'])}: {len(member_ids):,} particles")
print(f"Representative particle ID (first in list, for the viewer's focus-on-particle picker): {member_ids[0]}")
print(f"Full list saved to cluster_members_export.csv for reference.")
pd.DataFrame({"particle_id": member_ids}).to_csv(RUN_DIR / "biggest_cluster_members_export.csv", index=False)
Exact membership at step 4730: 14,385 particles Representative particle ID (first in list, for the viewer's focus-on-particle picker): 5 Full list saved to cluster_members_export.csv for reference.
4. What this cluster actually looks like¶
Rendering the exact captured frames (from cluster_events/cluster_frames.bin
- the deduplicated shared archive, see project memory) for the event containing this cluster shows something qualitatively different from the n=250,000 run's finding: not a small, tight, dense core, but a large, diffuse structure spanning nearly the swarm's full visible extent - consistent with it holding almost 3x the population share. See the GIF on the experiment page for the full picture; a single frame is enough to see the qualitative difference here.
target_event = events_df.loc[events_df["peakSize"].idxmax()]
print(target_event.to_dict())
steps, positions, charges = bio.load_cluster_event_frames(RUN_DIR, target_event.to_dict())
print(f"Loaded {len(steps)} exact frames spanning steps {steps.min()}-{steps.max()}")
import matplotlib.pyplot as plt
last_pos = positions[-1]
member_set = set(member_ids)
member_mask = np.array([i in member_set for i in range(header.total_particles)])
fig = plt.figure(figsize=(6, 6), facecolor="#050508")
ax = fig.add_subplot(111, projection="3d", facecolor="#050508")
bg_idx = np.random.default_rng(0).choice(np.where(~member_mask)[0], size=5000, replace=False)
hi_idx = np.random.default_rng(0).choice(np.where(member_mask)[0], size=1200, replace=False)
ax.scatter(*last_pos[bg_idx].T, c="#888", s=2, alpha=0.25, linewidths=0)
ax.scatter(*last_pos[hi_idx].T, c="#00ffcc", s=4, alpha=0.9, linewidths=0)
ax.set_axis_off()
ax.view_init(elev=20, azim=45)
plt.tight_layout()
plt.savefig("n100k_cluster_still.png", dpi=120, facecolor="#050508")
plt.show()
{'eventIndex': 1727, 'trackId': 39134, 'category': 'large', 'paddedStartStep': 3492, 'lastLiveStep': 4999, 'stepSpan': 1508, 'peakSize': 14385}
Loaded 1508 exact frames spanning steps 3492-4999
5. Size and age distribution¶
Same visual summary as analyze_stability_clusters.ipynb - duration vs.
peak size across every tracked cluster, plus cluster count over the run.
Here it's built from the exact cluster_events_index.csv data rather than
reconstructed tracks, since this run has it.
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
axes[0].scatter(events_df["stepSpan"], events_df["peakSize"], s=10, alpha=0.4)
axes[0].set_xlabel("Event duration (steps)")
axes[0].set_ylabel("Peak cluster size (particles)")
axes[0].set_yscale("log")
axes[0].set_title("Duration vs. peak size, every captured event")
axes[0].grid(True, alpha=0.3)
step_counts = clusters_df.groupby("step").size()
axes[1].plot(step_counts.index, step_counts.values, linewidth=0.8)
axes[1].set_xlabel("Step")
axes[1].set_ylabel("Number of clusters that step")
axes[1].set_title("Cluster count over the run")
axes[1].grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("n100k_cluster_distribution.png", dpi=150)
plt.show()
Summary¶
| n=250,000 run | n=100,000 run (this notebook) | |
|---|---|---|
| Largest cluster | 12,472 particles | 14,385 particles |
| Share of swarm | ~5.0% | ~14.4% |
| Particle IDs | Approximate (nearest baked frame, predates memberIds fix) | Exact (live engine tracking) |
| Visual character | Tight, dense, small core | Large, diffuse, spans much of the swarm |
Both runs used loose default cluster-event thresholds (size>50 & duration>3,
or duration>50 & size>3), so neither number should be read as a ceiling -
just what these two particular runs happened to produce. The qualitative
difference in visual character (tight core vs. diffuse spread) is a real,
concrete observation worth investigating further before drawing any
conclusion about why - it could be run-to-run variation, a genuine n-scaling
effect, or something else; this notebook doesn't settle that.