funded-report-assistant
Tracker lint, source-referenced progress reports and review packs for a funded R&D project. By Pavan Yadav Annappa. Code on GitHub
All project data on these pages is fictional. "eDrive-X" and Partners A-D are generated by a seeded script; no real company, person or project data is used.
Evaluation
Generated by
scripts/make_results.pyfromresults/*.json. Do not edit by hand. All data is FICTIONAL and generated byfra.generator.
Summary
Tracker lint on 20 generated trackers (seeds 1000-1019, 13 defect kinds planted per tracker where placeable, formatting noise on) with 257 planted inconsistencies, plus the same 20 trackers without defects:
| Method | Planted defects found | False alarms (planted trackers) | Findings on clean trackers |
|---|---|---|---|
fra lint (13 rules) |
257/257 (100.0 %) | 0 | 0 |
| Baseline: date filter only | 20/257 (7.8 %) | 56 | 45 |
| Baseline: spreadsheet checks (date filter, blanks, duplicates) | 80/257 (31.1 %) | 56 | 45 |
Mean runtime per tracker on this host: lint 0.3 ms, deterministic report 0.7 ms (after loading the .xlsx).
Closed world: the generator and the checker were written together by the same author, so 100 % recall here shows that each rule fires on the defect it was written for and stays quiet on clean data. It does not show how the rules do on a real project tracker. The baseline false alarms are cancelled actions with a past due date, which a plain "not closed" filter reports as overdue.
LLM narrative + guard: llama3.1:8b via Ollama 0.33.3, temperature 0, seed 42, device: CPU only (num_gpu=0, 32 vCPU). 21 projects (the sample + 20 generated), period 2026-H1, 126 work-package summaries.
| Metric | Value |
|---|---|
| Sentences where every number, date and ID is in the source facts | 380/393 (96.7 %) |
| Summaries with all sentences grounded | 113/126 |
| Ungrounded tokens caught by the guard | 15 (num: 15) |
| Summaries with a milestone status contradiction | 0/126 |
| Summaries that dropped at least one fact (ID, date or number) | 109/126 (515 of 1748 fact tokens dropped) |
| LLM summaries kept (pass all three checks) | 17/126 (13.5 %) |
| Kept summaries with a meaning error found on manual reading (one reader, not blind, every kept summary compared with its facts) | 6/17 |
| Guard on the deterministic text itself | 592/592 sentences grounded |
| Planted one-digit changes in grounded LLM sentences, new value not in facts | caught 364/364 |
| Planted one-digit changes that land on another value present in the facts | caught 0/14 (not detectable by a token check) |
| Mean time per WP summary / per report (6 WPs) | 10.29 s / 61.74 s |
Rejected summaries fall back to the deterministic text, so the report never contains an ungrounded number. The guard checks tokens, not meaning: a sentence can use only true numbers and still bind them to the wrong claim. In this run the status check found no contradiction, yet manual reading of the kept summaries found 'ahead of plan' for late milestones and 'exceeds the plan by 153%' for 153 % of plan. The other way round, several rejected sentences hold a correct derived number (for example 'exceeding the plan by 8%' for 32.5 vs 30.0 PM): the guard rejects any number that is not literally in the facts. The LLM ran on CPU because the host GPU (NVIDIA L4) was in use by another job; runtimes are CPU runtimes. No time saving is claimed; only runtime was measured.
Lint detection per rule
| Rule | Check | Planted | fra lint | Date filter | Spreadsheet checks |
|---|---|---|---|---|---|
| FRA001 | Duplicate id | 20 | 20 | 0 | 20 |
| FRA002 | Status outside allowed set | 20 | 20 | 0 | 0 |
| FRA003 | Action without owner | 20 | 20 | 0 | 20 |
| FRA004 | Open action without due date | 20 | 20 | 0 | 20 |
| FRA005 | Open action overdue | 20 | 20 | 20 | 20 |
| FRA006 | Milestone done, linked action open | 20 | 20 | 0 | 0 |
| FRA007 | Milestone done without evidence | 19 | 19 | 0 | 0 |
| FRA008 | Past due date still planned | 18 | 18 | 0 | 0 |
| FRA009 | Person-month burn above plan | 20 | 20 | 0 | 0 |
| FRA010 | Action in minutes, missing in tracker | 20 | 20 | 0 | 0 |
| FRA011 | Tracker action not in cited minutes | 20 | 20 | 0 | 0 |
| FRA012 | Unknown work package | 20 | 20 | 0 | 0 |
| FRA013 | Open risk without mitigation | 20 | 20 | 0 | 0 |
Examples of sentences the guard rejected
- seed2000 WP2: "The team used 32.5 person-months of effort, exceeding the planned 30.0 person-months by 8%, and also met the planned total effort of 30.0 person-months." (not in facts: num 8)
- seed2000 WP5: "The project has utilized 17.9 person-months, exceeding the planned 17.3 person-months by 3% to date, with a total planned expenditure of 58.0 person-months." (not in facts: num 3)
- seed2007 WP3: "In the first half of 2026, Milestone 5 was delayed and Milestone 7 was completed on 2026-06-29, although it was initially planned for 2026-06-30." (not in facts: num 5, num 7)
- seed2007 WP4: "The project has used 24.6 person-months, exceeding the planned 23.0 person-months by 7%, and a total of 52.0 person-months is still planned." (not in facts: num 7)
- seed2009 WP4: "To date, 25.3 person-months have been used, exceeding the planned 23.0 person-months by 10%, with a total of 52.0 person-months planned for the project." (not in facts: num 10)
- seed2010 WP6: "To date, 7.5 person-months have been used, exceeding the planned 6.9 person-months by 9%, with a total of 44.0 person-months planned for the project." (not in facts: num 9)
- seed2011 WP3: "In the first half of 2026, two actions were completed, but Milestone 6 was delayed and Milestone 7 was completed ahead of schedule on 2026-07-09." (not in facts: num 6, num 7)
- seed2012 WP5: "In the first half of 2026, 2 actions were completed out of 2 due, and 18.1 person-months have been used, exceeding the planned 17.3 person-months by 5%." (not in facts: num 5)
Manual reading of the kept LLM summaries
- sample WP2: says M4 was done 'ahead of' plan (it was 8 days late) and that 45.9 PM exceed the plan 'by 153%' (it is 153 % of plan)
- seed2002 WP2: 'M5 not being completed until after its planned date' suggests M5 was completed late; it is still delayed and open
- seed2004 WP2: says M3 was done 'ahead of' plan (1 day late) and that 31 PM exceed the plan 'by 103%' (it is 103 % of plan)
- seed2005 WP4: says M2 was done 'ahead of' plan (it was 19 days late)
- seed2006 WP3: says 37.2 PM exceed the plan 'by 107%' (it is 107 % of plan)
- seed2018 WP1: uses the decision date 2026-06-28 as the status date of the action count
Environment
LLM run: Linux-6.1.158-178.288.amzn2023.x86_64-x86_64-with-glibc2.34, Python 3.11.16, host GPU NVIDIA L4, LLM device CPU only (num_gpu=0, 32 vCPU). Wall time 1297.3 s.
Lint run: Python 3.11.16, Linux-6.1.158-178.288.amzn2023.x86_64-x86_64-with-glibc2.34, wall time 1.1 s.
Pre-rendered by scripts/build_site.py from the files in the repository.