Cluster Analysis: main_run_n100k_rapidity (n=100,000, hyperbolic rapidity integration)¶
Analyzes a real run captured with the cluster event capture feature (see
but_cluster_event_capture_feature in project memory), same as
analyze_n100k_clusters.ipynb - exact live cluster tracking, exact particle
IDs, no reconstruction. The difference this time: integrationMode=1
(hyperbolic rapidity) instead of Direct Normalization, and 56,000 frames
instead of 5,000 (~11x longer), to see whether the integration mode itself
changes what kind of structure emerges.
In [1]:
import numpy as np
import pandas as pd
import csv
from collections import Counter
from pathlib import Path
import sys
sys.path.insert(0, str(Path.cwd()))
import but_binary_io as bio
csv.field_size_limit(2**31 - 1)
RUN_DIR = Path("/Users/ian/Documents/sim/cluster_event_capture_n100k_rapidity/main_run_n100k_rapidity")
header = bio.read_header(RUN_DIR / "simulation_data.bin")
print(header)
# stability_clusters.csv is 6.7GB (6.9M rows, memberIds column) - too large to load
# into a DataFrame on this machine. Stream it instead: track the single biggest-size
# row (keeping only its memberIds string, not everyone else's), row/step counts, and a
# per-step cluster-count tally (a plain int-keyed Counter, <=56,000 keys - negligible).
row_count = 0
min_step = None
max_step = None
biggest = None
step_counts = Counter()
with open(RUN_DIR / "stability_clusters.csv", newline="") as f:
reader = csv.DictReader(f)
for row in reader:
row_count += 1
step = int(row["step"])
step_counts[step] += 1
if min_step is None or step < min_step:
min_step = step
if max_step is None or step > max_step:
max_step = step
size = int(row["size"])
if biggest is None or size > biggest["size"]:
biggest = {"step": step, "clusterId": row["clusterId"], "size": size, "memberIds": row["memberIds"]}
print(f"{row_count:,} cluster-observation rows, steps {min_step}-{max_step}")
SimHeader(num_posons=50000, num_negons=50000, initial_width=126.0, total_frames=57) 6,945,299 cluster-observation rows, steps 180-55999
1. The headline number¶
In [2]:
pct = 100 * biggest["size"] / header.total_particles
print(f"Largest cluster observed: {biggest['size']:,} particles at step {biggest['step']}")
print(f"That's {pct:.2f}% of all {header.total_particles:,} particles in one bonded structure.")
print(f"For comparison: 5.0% at n=250,000 (Direct Normalization), 14.4% at n=100,000 (Direct Normalization).")
Largest cluster observed: 24,521 particles at step 55918 That's 24.52% of all 100,000 particles in one bonded structure. For comparison: 5.0% at n=250,000 (Direct Normalization), 14.4% at n=100,000 (Direct Normalization).
2. Exact cluster tracks from the live engine¶
In [3]:
events = bio.load_cluster_events_index(RUN_DIR)
events_df = pd.DataFrame(events)
print(f"{len(events_df):,} events captured")
print(events_df[["stepSpan", "peakSize"]].describe())
print("\n=== Top 5 by peak size ===")
print(events_df.sort_values("peakSize", ascending=False).head(5)[["eventIndex", "trackId", "category", "paddedStartStep", "lastLiveStep", "stepSpan", "peakSize"]].to_string(index=False))
print("\n=== Top 5 by step span (longest lasting) ===")
print(events_df.sort_values("stepSpan", ascending=False).head(5)[["eventIndex", "trackId", "category", "paddedStartStep", "lastLiveStep", "stepSpan", "peakSize"]].to_string(index=False))
2,000 events captured
stepSpan peakSize
count 2000.000000 2000.000000
mean 54466.304500 120.237500
std 379.446999 564.143412
min 53950.000000 4.000000
25% 54152.000000 51.000000
50% 54382.000000 71.000000
75% 54729.250000 117.250000
max 55644.000000 24521.000000
=== Top 5 by peak size ===
eventIndex trackId category paddedStartStep lastLiveStep stepSpan peakSize
344 10506 long_duration 1125 55999 54875 24521
445 14083 large 1254 55999 54746 2763
1776 38646 large 1945 55999 54055 1742
830 22970 large 1524 55999 54476 1351
1693 38155 large 1933 55999 54067 1214
=== Top 5 by step span (longest lasting) ===
eventIndex trackId category paddedStartStep lastLiveStep stepSpan peakSize
0 161 long_duration 356 55999 55644 11
1 199 long_duration 372 55999 55628 22
2 237 long_duration 384 55999 55616 15
3 369 long_duration 416 55999 55584 11
4 396 long_duration 421 55999 55579 18
3. Resolving the headline cluster to exact particle IDs¶
In [4]:
member_ids = [int(x) for x in str(biggest["memberIds"]).split("|")]
print(f"Exact membership at step {biggest['step']}: {len(member_ids):,} particles")
print(f"Representative particle ID (first in list, for the viewer's focus-on-particle picker): {member_ids[0]}")
print(f"Full list saved to biggest_cluster_members_export.csv for reference.")
pd.DataFrame({"particle_id": member_ids}).to_csv(RUN_DIR / "biggest_cluster_members_export.csv", index=False)
Exact membership at step 55918: 24,521 particles Representative particle ID (first in list, for the viewer's focus-on-particle picker): 3 Full list saved to biggest_cluster_members_export.csv for reference.
4. Did the cluster settle, or was it still growing when the run ended?¶
This run went 11x longer than the earlier n=100,000 run specifically to see whether a cluster like this eventually stabilizes at a size, rather than just catching it mid-growth.
In [5]:
target_event = events_df.loc[events_df["peakSize"].idxmax()]
print(target_event.to_dict())
# NOTE: this event's span (paddedStartStep=1125 to lastLiveStep=55999) covers almost the
# entire run, so load_cluster_event_frames (which loads every matching frame) would try to
# pull ~51,000 full 100k-particle snapshots into memory at once - it OOM-killed this machine
# (16GB RAM). Loading just the single frame closest to the event's last live step is enough
# to see its final state.
with open(RUN_DIR / "cluster_events" / "cluster_frames_index.csv", newline="") as f:
step_to_frame_index = {int(row["step"]): int(row["frameIndex"]) for row in csv.DictReader(f)}
print(f"Archive holds {len(step_to_frame_index):,} unique captured steps")
last_live_step = int(target_event["lastLiveStep"])
closest_step = max(s for s in step_to_frame_index if s <= last_live_step)
print(f"Closest captured step to lastLiveStep={last_live_step}: {closest_step}")
bin_path = RUN_DIR / "cluster_events" / "cluster_frames.bin"
archive_header = bio.read_header(bin_path)
final_pos, _ = bio.load_frame(bin_path, step_to_frame_index[closest_step], archive_header)
print(f"Event's peak ({int(target_event['peakSize']):,}) occurred at step {biggest['step']},"
f" only {max_step - biggest['step']} steps before the run ended at step {max_step}.")
still_growing = (max_step - biggest["step"]) < 200
print(f"Reads as still actively growing when the run ended: {still_growing}")
{'eventIndex': 344, 'trackId': 10506, 'category': 'long_duration', 'paddedStartStep': 1125, 'lastLiveStep': 55999, 'stepSpan': 54875, 'peakSize': 24521}
Archive holds 55,644 unique captured steps
Closest captured step to lastLiveStep=55999: 55999
Event's peak (24,521) occurred at step 55918, only 81 steps before the run ended at step 55999.
Reads as still actively growing when the run ended: True
In [6]:
import matplotlib.pyplot as plt
member_set = set(member_ids)
member_mask = np.array([i in member_set for i in range(header.total_particles)])
fig = plt.figure(figsize=(6, 6), facecolor="#050508")
ax = fig.add_subplot(111, projection="3d", facecolor="#050508")
bg_idx = np.random.default_rng(0).choice(np.where(~member_mask)[0], size=5000, replace=False)
hi_size = min(1200, int(member_mask.sum()))
hi_idx = np.random.default_rng(0).choice(np.where(member_mask)[0], size=hi_size, replace=False)
ax.scatter(*final_pos[bg_idx].T, c="#888", s=2, alpha=0.25, linewidths=0)
ax.scatter(*final_pos[hi_idx].T, c="#00ffcc", s=4, alpha=0.9, linewidths=0)
ax.set_axis_off()
ax.view_init(elev=20, azim=45)
plt.tight_layout()
plt.savefig("n100k_rapidity_cluster_still.png", dpi=120, facecolor="#050508")
plt.show()
5. Size and age distribution¶
Same visual summary as analyze_stability_clusters.ipynb - duration vs.
peak size across every captured event, plus cluster count over the run
(from the per-step tally kept during the streaming scan in cell 1, not a
full reload of the 6.7GB CSV).
In [7]:
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
axes[0].scatter(events_df["stepSpan"], events_df["peakSize"], s=10, alpha=0.4)
axes[0].set_xlabel("Event duration (steps)")
axes[0].set_ylabel("Peak cluster size (particles)")
axes[0].set_yscale("log")
axes[0].set_title("Duration vs. peak size, every captured event")
axes[0].grid(True, alpha=0.3)
steps_sorted = sorted(step_counts)
axes[1].plot(steps_sorted, [step_counts[s] for s in steps_sorted], linewidth=0.8)
axes[1].set_xlabel("Step")
axes[1].set_ylabel("Number of clusters that step")
axes[1].set_title("Cluster count over the run")
axes[1].grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("n100k_rapidity_cluster_distribution.png", dpi=150)
plt.show()
Summary¶
| n=250,000 (Direct Norm.) | n=100,000 (Direct Norm.) | n=100,000 (Rapidity, this notebook) | |
|---|---|---|---|
| Steps run | 3,211 | 5,000 | 56,000 |
| Largest cluster | 12,472 particles | 14,385 particles | 24,521 particles |
| Share of swarm | ~5.0% | ~14.4% | ~24.5% |
| Particle IDs | Approximate | Exact (live engine) | Exact (live engine) |
| Still growing at run end? | Unknown (interrupted) | Yes (still active) | Yes - peaked only 81 steps before the run's last step |
The rapidity-mode run produced the largest bound structure seen in this project so far by a wide margin - almost a quarter of the entire swarm in one connected cluster - and critically, the tracking data shows its size was still climbing right up to the final captured step (peak at step 55,918 of 56,000). That means this number is a lower bound on what this configuration produces, not a settled result: a longer run would likely show an even bigger structure, or reveal where it actually plateaus.
As with the earlier runs, this used loose default cluster-event thresholds
(size>50 & duration>3, or duration>50 & size>3), so none of these
numbers should be read as a ceiling - just what this particular run
happened to produce before it was stopped.