A comprehensive tool for analyzing sprint planning effectiveness by comparing Monte Carlo simulations, Planning Poker estimates, and story point correlations with actual delivery data. Based on Daniel Vacanti's principles from "When will it be done?"
- 📊 Monte Carlo vs Planning Poker Analysis: Compare prediction accuracy retrospectively
- 📈 Story Points Correlation Analysis: Validate if story points predict cycle time
- 📉 Multi-team Comparison: Analyze patterns across multiple teams
- 🎨 Rich Visualizations: Generate charts for compelling evidence
- 🔍 Statistical Insights: Confidence intervals, bias analysis, and correlation metrics
# Clone the repository
git clone <repository-url>
cd retroactive-estimation-check
# Install with uv
uv sync
# Or install as a package
uv pip install -e .# Show available commands and options
estimation-check --help
# Show detailed information about Vacanti's approach
estimation-check info
# Analyze a single team
estimation-check analyze "Team Alpha" --data pbis.json --confidence 0.85 --charts
# Analyze all teams
estimation-check analyze-all --data pbis.json --charts --output-dir results/
# Analyze story points correlation
estimation-check correlation --data pbis.json --chartsAnalyze estimation effectiveness for a specific team using Monte Carlo vs Planning Poker.
estimation-check analyze TEAM_NAME [OPTIONS]Arguments:
TEAM_NAME(required): Name of the team to analyze
Options:
--data PATH: JSON file with PBI data (recommended format)--sprints PATH: CSV file with sprint data (legacy format)--work-items PATH: CSV file with work item data (legacy format)--confidence FLOAT: Confidence level for Monte Carlo simulation (default: 0.85)--iterations INT: Number of Monte Carlo iterations (default: 10000)--history INT: Number of historical sprints to use (deprecated)--charts: Generate visualization charts--output-dir PATH: Directory to save charts (defaults to current directory)
Analyze estimation effectiveness for all teams in the dataset.
estimation-check analyze-all [OPTIONS]Options:
--data PATH: JSON file with PBI data (recommended format)--sprints PATH: CSV file with sprint data (legacy format)--work-items PATH: CSV file with work item data (legacy format)--confidence FLOAT: Confidence level for Monte Carlo simulation (default: 0.85)--iterations INT: Number of Monte Carlo iterations (default: 10000)--history INT: Number of historical sprints to use (deprecated)--charts: Generate visualization charts--output-dir PATH: Directory to save charts (defaults to current directory)
Analyze correlation between story points and cycle time to validate estimation practices.
estimation-check correlation [TEAM_NAME] [OPTIONS]Arguments:
TEAM_NAME(optional): Specific team to analyze (analyzes all teams if omitted)
Options:
--data PATH: JSON file with PBI data (required)--charts: Generate correlation visualization charts--output-dir PATH: Directory to save charts and reports
Display information about Vacanti's Monte Carlo approach and tool usage.
estimation-check infoUses PBICleanItem schema from workitem-cli pbi-sprints --clean:
[
{
"sprint_number": 1,
"team_name": "Team Alpha",
"work_item_id": "PROJ-123",
"work_item_description": "Implement user authentication",
"work_item_estimation": 5.0,
"date_started": "2024-01-02T09:00:00",
"date_finished": "2024-01-05T17:00:00"
}
]Sprints CSV:
sprint_id,sprint_name,start_date,end_date,planning_poker_estimate
SPRINT-1,Team Alpha - Sprint 1,2024-01-01,2024-01-14,8
SPRINT-2,Team Alpha - Sprint 2,2024-01-15,2024-01-28,10
Work Items CSV:
key,summary,created,started,completed
PROJ-1,User login,2023-12-15T09:00:00,2024-01-02T10:00:00,2024-01-08T16:30:00
PROJ-2,Database setup,2023-12-16T10:00:00,2024-01-03T09:00:00,2024-01-12T14:00:00
- Accuracy Over Time: Line chart showing prediction accuracy trends
- Prediction vs Actual: Scatter plots comparing predictions to actual delivery
- Error Distribution: Box plots showing prediction error patterns
- Sprint-by-Sprint Comparison: Bar charts with actual vs predicted values
- Multi-Team Comparison: Overall accuracy comparison across teams
- Correlation Scatter Plot: Story points vs cycle time with trend line
- Cycle Time Distribution: Histogram with percentile markers (50th, 85th, 95th)
- Box Plots by Story Points: Cycle time distribution for each story point value
- Uses historical throughput (items completed per sprint)
- Uses all available historical data up to 12 sprints (follows Vacanti's recommendation)
- Requires minimum 3 sprints of history for statistical validity
- Provides confidence-based forecasts (e.g., 85% confident in completing 5+ items)
- No story points - focuses on item count per Vacanti
- Pearson correlation: Measures linear relationships
- Spearman correlation: Captures any monotonic relationship
- R-squared: Shows proportion of variance explained
- Interprets results in context of Vacanti's critique
- Accuracy: 1 - relative error (100% = perfect prediction)
- Bias: Mean prediction error (positive = over-estimation)
- Hit Rate: Percentage of predictions within 80% accuracy
# Generate evidence for retrospective discussion
estimation-check analyze "Team Alpha" \
--data sprint_data.json \
--confidence 0.85 \
--charts \
--output-dir retro_charts/# Analyze all teams with visualizations
estimation-check analyze-all \
--data all_teams.json \
--charts \
--output-dir management_review/
# Add correlation analysis
estimation-check correlation \
--data all_teams.json \
--charts \
--output-dir management_review/correlations/# Check if story points actually predict delivery time
estimation-check correlation "Team Beta" \
--data team_beta_pbis.json \
--charts- MC Accuracy > PP Accuracy + 5%: Monte Carlo shows superior accuracy
- PP Accuracy > MC Accuracy + 5%: Planning Poker remains more effective
- Similar accuracy: Consider hybrid approach
- r < 0.3: Negligible correlation - story points don't predict cycle time
- 0.3 ≤ r < 0.5: Weak correlation - limited predictive value
- 0.5 ≤ r < 0.7: Moderate correlation - some predictive value
- r ≥ 0.7: Strong correlation - unusual, suggests consistent practices
This project follows Test-Driven Development (TDD) principles:
# Run tests
uv run pytest
# Run with coverage
uv run pytest --cov=src/retroactive_estimation_check
# Type checking
uv run mypy src/
# Code formatting
uv run ruff check src/
uv run ruff format src/Sample data files are included for testing:
# Test multi-team analysis
uv run estimation-check analyze-all \
--data sample_multi_team.json \
--charts
# Test correlation analysis
uv run estimation-check correlation \
--data sample_multi_team.json \
--chartsBased on Daniel Vacanti's "When will it be done?", this tool implements:
- Historical Throughput Analysis: Extract actual item completion rates from past sprints
- Monte Carlo Simulation: Sample from historical distributions to forecast capacity
- Correlation Analysis: Validate if story points predict cycle time
- Confidence Intervals: Provide probabilistic predictions (e.g., 85% confidence)
- Bias Analysis: Compare systematic over/under-prediction patterns
The goal is to help teams make data-driven decisions about their estimation and planning practices.