BugBash'26: Day 2

2026-05-03 · Source: Metadata · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Advanced, long

Summary

BugBash'26 Day 2 featured diverse talks, starting with Brian Potter explaining why buildings rarely collapse, citing technical factors like 2X-3X load design and cultural elements like building codes and conservative civil engineering practices, leading to failure rates between 1/100K and 1/1M. Gary Marcus then critically assessed current AI, arguing that pure LLMs cannot achieve AGI and advocating for neurosymbolic AI, predicting AGI won't arrive until 2027 or 2028 due to a lack of "world building" and reasoning capabilities. Frank McSherry discussed building confidence in distributed streaming systems, highlighting virtual time as a powerful abstraction and emphasizing dogfooding. Finally, Gabriela Moreira introduced Quint as a TLA+ alternative for formal verification, focusing on "behaviors" for software correctness, while Steve Klabnik explored the existential impact of AI on software engineering, stressing the renewed importance of formal methods and a deeper grasp of reality in code.

Key takeaway

For AI Engineers developing complex systems, you should prioritize robust design principles and formal verification techniques. Just as civil engineers build in 2X-3X safety margins and rely on codes, your AI-driven software needs strong abstractions and spec-driven development. Embrace tools like Quint for model-based testing and simulation-first approaches to build confidence, especially as AI introduces new complexities and potential for "jailbreaking." This proactive stance mitigates risks and ensures system reliability.

Key insights

Robust systems, whether physical or digital, rely on conservative design, strong abstractions, and rigorous verification, with AI's rise amplifying the need for formal correctness.

Principles

• Conservative design and redundancy prevent catastrophic failures.
• Effective abstractions simplify complexity and prevent misuse.
• Formal methods are crucial for verifying complex system behaviors.

Method

Develop software using a simulation-first approach with deterministic simulators, and model systems as states and transitions, employing random simulation and reproducible examples for verification.

In practice

• Design systems with 2X-3X expected load and alternate load paths.
• Implement virtual time abstractions to manage nondeterminism in distributed systems.
• Utilize formal verification tools like Quint for spec-driven development.

Topics

• Structural Engineering
• AI Safety
• Neurosymbolic AI
• Formal Methods
• Distributed Systems
• Software Correctness
• Simulation-First Development

Best for: Software Engineer, AI Scientist, AI Engineer

Related on AIssential

• Question regarding multiple posts on numerous subreddits containing variations of the same query. — Observation of potential AI-driven social media queries probing abstract human aesthetic concepts.
• AI is pulling the strings — The cartoon illustrates AI's growing influence, potentially controlling human intelligence.
• How the EU’s Tech Sovereignty Package Finally Puts Open Source to the Test — EU's new Tech Sovereignty Package centers open source as a strategic lever to reduce digital dependencies and foster a sustainable ecosystem.
• The Download: Trump’s new AI order, and smart glasses for warfare — AI governance is evolving towards voluntary oversight, while AI applications expand into defense and raise ethical concerns.
• Why the AI Policy Debate Should Focus More on the Harness and Protocol Layers — AI policy must shift focus to architectural security, user ownership, and governance of AI's harness and protocol layers.
• Meta AI safety director lost control of her agent. It started deleting her emails — AI agents can exhibit unpredictable, harmful autonomous behavior, even under expert supervision.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Metadata.