Data analytics & DWH integration¶

Codefang produces richly structured JSON output designed for loading into columnar data warehouses (ClickHouse, Greenplum, BigQuery, Snowflake) and building BI dashboards. This guide covers the optimal pipeline from repository analysis to production dashboards.

Quick start¶

Analyze a repository and produce DWH-ready output:

# JSON for small-to-medium repos (< 5K files, < 10K commits)
codefang run --format json --per-file --memory-budget 4GB /path/to/repo > report.json

# NDJSON for large repos (streaming, one line per analyzer)
codefang run --format ndjson --per-file --memory-budget 8GB /path/to/repo > report.ndjson

# Limit history depth for faster iteration
codefang run --format json --per-file --limit 5000 /path/to/repo > report.json

Output format selection¶

JSON format¶

codefang run --format json --per-file /repo > report.json

Produces a single JSON object with versioned envelope:

{
  "version": "codefang.run.v1",
  "metadata": {
    "repo_name": "myproject",
    "analyzed_at": "2026-04-08T10:00:00Z",
    "codefang_version": "0.1.0"
  },
  "analyzers": [
    {
      "id": "static/complexity",
      "mode": "static",
      "schema": { ... },
      "report": { ... }
    }
  ]
}

NDJSON format¶

codefang run --format ndjson --per-file /repo > report.ndjson

One JSON line per analyzer. First line is metadata:

{"version":"codefang.run.v1","metadata":{"repo_name":"myproject",...}}
{"id":"static/complexity","mode":"static","report":{...}}
{"id":"history/sentiment","mode":"history","report":{...}}

Process with standard tools:

# Extract one analyzer
grep '"static/complexity"' report.ndjson | jq .report.aggregate

# Count analyzers
wc -l report.ndjson

# Stream into ClickHouse
cat report.ndjson | clickhouse-client --query "INSERT INTO codefang_raw FORMAT JSONEachRow"

Memory budget¶

Always set --memory-budget for repos with history analysis. Without it, the streaming pipeline uses a conservative 2GB default that may OOM on large repos.

The budget controls the streaming chunk planner — larger budgets mean fewer, bigger chunks and faster processing. The actual RSS will be ~2x the budget due to Go runtime overhead and native memory.

# 64GB machine, kubernetes-sized repo (~56K commits)
codefang run --format ndjson --per-file --memory-budget 8GB ~/sources/kubernetes

Commit limiting¶

Use --limit N to analyze only the most recent N commits. This is useful for:

• Fast iteration: Test your ETL pipeline on a subset before running full history
• Incremental analysis: Analyze only recent changes for daily dashboards
• Memory control: Fewer commits = less memory, faster processing

# Last 1000 commits (fast, ~2 min)
codefang run --format json --per-file --limit 1000 /repo > recent.json

# Last 10000 commits (moderate, ~15 min)
codefang run --format json --per-file --limit 10000 --memory-budget 4GB /repo > report.json

# Full history (slow, may take hours for large repos)
codefang run --format json --per-file --memory-budget 8GB /repo > full.json

Key fields for analytics¶

Every function-level record includes fields designed for DWH joins and aggregation:

Schema manifest¶

Every analyzer section includes a schema field describing its output:

{
  "schema": {
    "function_complexity": {
      "type": "list",
      "grain": "function",
      "description": "Per-function cyclomatic and cognitive complexity"
    },
    "aggregate": {
      "type": "aggregate",
      "description": "Summary statistics"
    }
  }
}

Field types: list, aggregate, time_series, risk, scalar

Grain values: function, file, tick, pair, developer, node, comment, import

Use the schema to auto-generate ETL mappings:

# Python: extract schema for table generation
import json
with open('report.json') as f:
    data = json.load(f)
for analyzer in data['analyzers']:
    schema = analyzer.get('schema', {})
    for field, meta in schema.items():
        if meta['type'] == 'list':
            print(f"CREATE TABLE {analyzer['id'].replace('/', '_')}_{field} ...")

Star schema design¶

Dimensions¶

-- dim_repository
CREATE TABLE dim_repository (
    repo_id     UInt64,
    repo_name   String,
    repo_path   String,
    analyzed_at DateTime,
    version     String
) ENGINE = MergeTree() ORDER BY repo_id;

-- dim_file (extract from source_file + directory + language)
CREATE TABLE dim_file (
    file_id     UInt64,
    repo_id     UInt64,
    source_file String,
    directory   String,
    language    LowCardinality(String)
) ENGINE = MergeTree() ORDER BY (repo_id, source_file);

-- dim_developer
CREATE TABLE dim_developer (
    dev_id  UInt32,
    repo_id UInt64,
    name    String,
    email   String
) ENGINE = MergeTree() ORDER BY (repo_id, dev_id);

-- dim_tick
CREATE TABLE dim_tick (
    tick_id    UInt32,
    repo_id    UInt64,
    tick       UInt32,
    start_time DateTime,
    end_time   DateTime
) ENGINE = MergeTree() ORDER BY (repo_id, tick);

Fact tables¶

-- Static analysis facts (per-function grain)
CREATE TABLE fact_function_complexity (
    repo_id              UInt64,
    source_file          String,
    directory            LowCardinality(String),
    language             LowCardinality(String),
    name                 String,
    cyclomatic_complexity UInt32,
    cognitive_complexity  UInt32,
    nesting_depth        UInt8,
    lines_of_code        UInt32,
    complexity_density   Float64,
    risk_level           LowCardinality(String)
) ENGINE = MergeTree()
ORDER BY (repo_id, directory, source_file, name);

-- Time-series facts (per-tick grain)
CREATE TABLE fact_tick_sentiment (
    repo_id        UInt64,
    tick           UInt32,
    start_time     DateTime,
    end_time       DateTime,
    sentiment      Float32,
    classification LowCardinality(String),
    comment_count  UInt32,
    commit_count   UInt32
) ENGINE = MergeTree()
ORDER BY (repo_id, tick);

-- Developer activity (per-tick-per-developer grain)
CREATE TABLE fact_developer_activity (
    repo_id  UInt64,
    tick     UInt32,
    dev_id   UInt32,
    commits  UInt32
) ENGINE = MergeTree()
ORDER BY (repo_id, tick, dev_id);

-- File coupling (per-pair grain)
CREATE TABLE fact_file_coupling (
    repo_id           UInt64,
    file1             String,
    file2             String,
    co_changes        UInt32,
    coupling_strength Float64
) ENGINE = MergeTree()
ORDER BY (repo_id, file1, file2);

ETL pipeline¶

Python (with dbt or standalone)¶

import json

with open('report.json') as f:
    data = json.load(f)

# Extract metadata
meta = data['metadata']
repo_id = hash(meta['repo_path'])  # or use a sequence

# Extract analyzers by ID
analyzers = {a['id']: a['report'] for a in data['analyzers']}

# Load function complexity
functions = analyzers['static/complexity']['function_complexity']
# Each record already has: name, source_file, language, directory,
# cyclomatic_complexity, cognitive_complexity, etc.

# Load time-series with timestamps
sentiment_ts = analyzers['history/sentiment']['time_series']
# Each tick has: tick, start_time, end_time, sentiment, classification, ...

# Load developers
developers = analyzers['history/devs']['developers']
# Each has: id, name, email, commits, lines_added, languages (array), ...

# Load file coupling (can be millions of rows)
coupling = analyzers['history/couples']['file_coupling']
# Each has: file1, file2, co_changes, coupling_strength

ClickHouse direct load¶

# Extract function complexity from NDJSON
grep '"static/complexity"' report.ndjson \
  | jq -c '.report.function_complexity[]' \
  | clickhouse-client --query "INSERT INTO fact_function_complexity FORMAT JSONEachRow"

# Extract sentiment time-series
grep '"history/sentiment"' report.ndjson \
  | jq -c '.report.time_series[]' \
  | clickhouse-client --query "INSERT INTO fact_tick_sentiment FORMAT JSONEachRow"

Recommended analyzer selection¶

Not all 17 analyzers are needed for every use case. Select based on your dashboard needs:

Code quality dashboard¶

codefang run \
  -a static/complexity,static/halstead,static/cohesion,static/comments \
  -a history/quality \
  --format json --per-file /repo

Produces: Function-level metrics, quality trend over time. Row count: ~200K functions + ~4K tick entries for a medium repo.

Developer analytics dashboard¶

codefang run \
  -a history/devs,history/couples,history/sentiment \
  --format json /repo

Produces: Developer profiles, coupling networks, sentiment trends. Row count: ~500 developers + ~5K coupling pairs + ~4K ticks.

File health dashboard¶

codefang run \
  -a static/complexity,static/clones \
  -a history/file-history,history/couples \
  --format json --per-file /repo

Produces: Per-file complexity, churn hotspots, coupling networks. Row count: ~30K files + ~100K coupling pairs.

Full analysis (everything)¶

codefang run --format ndjson --per-file --memory-budget 8GB /repo

Produces: All 17 analyzers. Use NDJSON for large repos.

Performance tuning¶

Static analysis workers¶

Control parallelism for the UAST parsing phase:

# Use all CPUs (default: min(NumCPU, 8))
codefang run --static-workers 16 --format json /repo

More workers = faster static phase but higher peak memory.

History analysis¶

The streaming pipeline auto-tunes chunk sizes based on --memory-budget. No manual tuning needed. Key parameters:

# Analyze only last 6 months, first-parent only
codefang run --since 6m --first-parent --format json /repo

Incremental analysis & checkpointing¶

Codefang supports two persistence mechanisms for long-running analysis: incremental caching (skip already-processed commits) and checkpointing (crash recovery).

Incremental cache¶

The incremental cache stores analysis results keyed by repository root SHA and branch. On subsequent runs, only new commits since the last cached position are processed.

# History-only run with cache (incremental)
codefang run -a 'history/*' --format json --memory-budget 8GB \
  --cache-dir ~/.codefang/cache /repo > history.json

# Static run (always full, no caching needed — fast)
codefang run -a 'static/*' --format json --per-file /repo > static.json

# Force full re-analysis (ignore cache)
codefang run -a 'history/*' --format json --memory-budget 8GB \
  --cache-dir ~/.codefang/cache --no-cache /repo > history-full.json

The cache stores a metadata file (cache.json) with head SHA, branch, commit count, and analyzer IDs. If the root SHA changes (force-push or history rewrite), the cache is automatically invalidated.

Checkpointing (crash recovery)¶

For very long runs (e.g., full kubernetes at ~3 hours), checkpointing saves progress periodically so a crash doesn't lose all work.

# Enable checkpointing (on by default)
codefang run --format json --memory-budget 8GB \
  --checkpoint --checkpoint-dir ~/.codefang/checkpoints /repo

# Resume from checkpoint after crash
codefang run --format json --memory-budget 8GB \
  --resume --checkpoint-dir ~/.codefang/checkpoints /repo

# Clear old checkpoint and start fresh
codefang run --format json --memory-budget 8GB \
  --clear-checkpoint /repo

The checkpoint stores:

• Current chunk position (which commits have been processed)
• Aggregator spill state (intermediate results on disk)
• Repository hash (for validation on resume)

Production pipeline example¶

A daily cron job that incrementally analyzes a repository:

#!/bin/bash
REPO=/opt/repos/kubernetes
CACHE_DIR=/var/lib/codefang/cache
CHECKPOINT_DIR=/var/lib/codefang/checkpoints
OUTPUT_DIR=/var/lib/codefang/output

# Pull latest
cd "$REPO" && git pull --ff-only

# Static analysis (always full, fast)
codefang run \
  -a 'static/*' \
  --format ndjson \
  --per-file \
  "$REPO" > "$OUTPUT_DIR/static-$(date +%Y%m%d).ndjson"

# History analysis (incremental via cache)
codefang run \
  -a 'history/*' \
  --format ndjson \
  --memory-budget 8GB \
  --cache-dir "$CACHE_DIR" \
  --checkpoint-dir "$CHECKPOINT_DIR" \
  "$REPO" > "$OUTPUT_DIR/history-$(date +%Y%m%d).ndjson"

# Load into ClickHouse
cat "$OUTPUT_DIR/report-$(date +%Y%m%d).ndjson" \
  | clickhouse-client --query "INSERT INTO codefang_raw FORMAT JSONEachRow"

Advanced tuning for history pipeline¶

Fine-tune the history streaming pipeline for specific hardware:

codefang run \
  --memory-budget 8GB \
  --commit-batch-size 200 \
  --blob-cache-size 2GB \
  --diff-cache-size 20000 \
  --blob-arena-size 8MB \
  --tmp-dir /fast-ssd/tmp \
  --format ndjson /repo

Row count estimates¶

Use these to plan DWH capacity:

Storage: ~2GB JSON for 50K files + 56K commits (kubernetes scale). Compressed in ClickHouse: ~200MB.

Materialized views¶

Pre-aggregate for common dashboard queries:

-- Complexity by directory (for treemap)
CREATE MATERIALIZED VIEW mv_complexity_by_directory
ENGINE = AggregatingMergeTree() ORDER BY (repo_id, directory)
AS SELECT
    repo_id,
    directory,
    avg(cyclomatic_complexity) AS avg_complexity,
    max(cyclomatic_complexity) AS max_complexity,
    count() AS function_count,
    countIf(risk_level = 'CRITICAL') AS critical_count
FROM fact_function_complexity
GROUP BY repo_id, directory;

-- Sentiment trend (for time-series chart)
CREATE MATERIALIZED VIEW mv_sentiment_weekly
ENGINE = AggregatingMergeTree() ORDER BY (repo_id, week)
AS SELECT
    repo_id,
    toMonday(start_time) AS week,
    avg(sentiment) AS avg_sentiment,
    sum(comment_count) AS total_comments
FROM fact_tick_sentiment
GROUP BY repo_id, week;

Troubleshooting¶

OOM kills¶

Symptom: Process killed during history analysis. Fix: Set --memory-budget explicitly.

# Check available RAM
free -h

# Set budget to ~25% of available RAM
codefang run --memory-budget 4GB --format ndjson /repo

Empty history analyzers¶

Some analyzers require specific conditions:

Large file coupling tables¶

file_coupling can produce millions of rows for large repos. Filter in your ETL:

# Only keep strong couplings
strong = [p for p in coupling if p['coupling_strength'] > 0.3]

Or limit at query time:

SELECT * FROM fact_file_coupling
WHERE coupling_strength > 0.3
ORDER BY coupling_strength DESC
LIMIT 1000;

Missing language/directory on some records¶

The language and directory fields are populated by the UAST parser. If a file's language is not supported by the parser, these fields will be empty. Supported languages include Go, Python, Java, JavaScript, TypeScript, C, C++, Ruby, Rust, and 40+ others.