B.S. Data Analytics Capstone · Applied AI · NLP · Python
This was my B.S. Data Analytics capstone at Western Governors University. I built a pipeline to examine whether public Reddit discussions about WGU courses could be organized into structured, traceable reports of recurring student concerns.
The project combines deterministic data preparation, language-model classification, semantic clustering, model evaluation, and static reporting. Its purpose was to surface patterns for human review, not to measure overall student satisfaction or establish that a reported course problem was real.
Problem
Reddit discussions contain potentially useful course feedback, but the posts are scattered, informal, and difficult to compare. Negative sentiment alone cannot distinguish a fixable course problem from ordinary frustration.
Solution
I combined deterministic filtering with language-model classification and semantic grouping. A labeled benchmark guided model selection and prompt refinement before processing the frozen corpus.
Result
The pipeline produced traceable course-level and cross-course issue data, published through a static viewer. The results identify patterns for human review rather than verified course defects.
From Reddit Posts to Structured Issues
- Frozen analytical corpus
- 1,103 posts
- Courses in frozen corpus
- 242
- Stage 1 positive classifications
- 396
- Eligible records entering clustering
- 390
- Courses represented in clustering
- 170
- Course-level clusters
- 327
- Global issue instances
- 63
The counts describe different stages of the pipeline. They are not estimates of how common these problems are among WGU students.
A Competitive Local Model
One of the most interesting results came from running llama3 locally through Ollama.
On the refined development benchmark, it scored 0.625 F1 and ranked third among four evaluated model configurations. It outperformed gpt-5-nano at 0.500; gpt-5 scored 0.651 and the selected gpt-5-mini configuration led at 0.687.
The local model had no recorded per-call API charge. It still required computing resources and execution time.
This result showed why I wanted to compare models using measured performance and cost rather than assume that a paid hosted model would be the strongest option.
Improving the Classifier Through Evaluation
Prompt refinement reduced false positives from 53 to 18 and increased development F1 from 0.485 to 0.687 for the selected model configuration.
I used labeled development data to examine classification errors, refine the instructions, and compare predictions.
The improvement involved a tradeoff. Recall decreased from 1.000 to 0.885 as the classifier became more selective about what qualified as an actionable course problem.
| Metric | Few-shot | Refined |
|---|---|---|
| Precision | 0.321 | 0.561 |
| Recall | 1.000 | 0.885 |
| F1 | 0.485 | 0.687 |
| False positives | 53 | 18 |
How the Pipeline Works
The pipeline separates tasks that can be handled with explicit rules from tasks requiring interpretation of informal language.
Deterministic code handles corpus selection, course-code validation, structured output checks, and reporting. Language models handle classification, within-course clustering, and cross-course issue normalization.
Human-labeled benchmark data informs the selected Stage 1 configuration. It is not an extra processing step applied to every post.
What the Analysis Surfaced
The final output organized reported concerns by course and recurring issue category.
For D427, the saved analysis includes a cluster of student reports concerning ZyBooks autograder, test-harness, or environment behavior.
For C207, it includes reports concerning rubric expectations and evaluator feedback.
These are patterns in public student discussions. The project did not independently verify that the reported course or platform problems occurred.
The published viewer lets readers inspect the structured analysis rather than relying only on these examples.
Interpreting the Results
The most difficult classification decision was whether a negative post described a specific, potentially fixable course-side issue.
Many posts fall near that boundary. A complaint about an assignment, for example, might reflect unclear instructions, an ordinary learning difficulty, or both. The benchmark included ambiguous cases rather than only obvious examples.
The selected classifier was evaluated on a separate held-out test set of 60 posts.
| Metric | Held-out TEST |
|---|---|
| Precision | 0.444 |
| Recall | 0.800 |
| F1 | 0.571 |
| Accuracy | 0.700 |
The classifier identified 12 of the 15 labeled positive cases, but also classified 15 negative cases as positive.
These results measure Stage 1 classification against the project’s labeling policy. They do not establish the accuracy of later clustering or normalization.
The frozen corpus was also intentionally selective. It contains negative-sentiment Reddit posts, not a representative sample of WGU students. The results should be treated as starting points for human review.
Technical Notes
Dataset Construction
The capstone collection and discovery scope covered 51 communities. The frozen analytical corpus used for this project contained 1,103 posts covering 242 courses and 24 represented subreddits. The latter is a processed corpus subset, not the project-wide community count.
Deterministic filtering required a valid single-course reference and a VADER compound sentiment score below -0.2. Course-code matching accounted for variations such as D334, D-334, and D 334, then checked identified codes against a pinned course catalog.
The selection rules made course-level analysis possible, but excluded comments, multi-course posts, and neutral or positive posts.
Classification Policy and Structured Output
Stage 1 classified posts as y, n, or u, representing a detected actionable pain point, no qualifying pain point, or an uncertain case.
The classification policy distinguished potentially fixable course-side problems from general frustration, personal circumstances, and ordinary learning difficulty.
For qualifying posts, the model produced structured issue records with a root-cause summary and supporting source excerpt.
The implementation tracked parse errors, schema errors, and fallback behavior rather than assuming every model response would satisfy the output contract.
Human Labeling and Benchmark Design
I used a documented labeling policy to create development and held-out test examples.
The development set supported model comparisons, prompt refinement, and error analysis. The test set was reserved for evaluation after selecting the configuration.
The benchmark used precision, recall, F1, accuracy, and confusion matrices. F1 was the primary comparison metric.
The labels followed one documented policy. The project did not establish independent agreement among multiple human raters.
Model Experiments and Statistical Testing
The benchmark compared four recorded model configurations across three prompt strategies: zero-shot, few-shot, and refined prompting. Development examples supported model comparison, prompt refinement, and error analysis. A held-out TEST set was reserved for evaluation after selection.
The hosted configurations were recorded as gpt-5-nano, gpt-5-mini, and gpt-5. The local configuration was recorded as llama3 through Ollama.
The benchmark compared model and prompt configurations using labeled examples, classification metrics, and error analysis. The capstone report documents the more detailed statistical comparison and evaluation methodology.
Clustering and Cross-Course Normalization
Stage 2 grouped related extracted issues within each course. Its saved input contained 390 eligible records across 170 courses, producing 327 course-level clusters.
Stage 3 aligned those course-level clusters with shared issue categories. Its saved output contained 63 global issue instances and 421 post-to-issue memberships.
The membership count is not a count of unique posts. A post may be associated with more than one issue.
Structural checks verified record relationships and coverage. The project did not establish an equivalent supervised accuracy score for the semantic quality of these groupings.
Cost, Latency, and Model Selection
Benchmark records preserved model configurations, token usage, elapsed time, and historical API-cost estimates.
These measurements made it possible to compare classification quality with inference expense.
The locally run Llama configuration had no recorded API charge, but still required local compute and execution time. The recorded prices describe the historical experiments, not current provider pricing.
Pinned Runs and Reproducibility
The project preserved a frozen input corpus, saved prompt snapshots, execution manifests, intermediate artifacts, and reporting tables.
Once the selected inference outputs were saved, deterministic reporting could rebuild the published presentation without new model calls.
This is reproducibility from pinned artifacts. Rerunning a language model does not guarantee identical responses.
Capstone Report
My B.S. Data Analytics capstone report documents the research question, dataset, model benchmarking, pipeline implementation, evaluation, analytical results, and limitations.
The capstone paper is retained for owner review. It is not included in this review build or publication output until its factual and privacy checklist is approved.
Roadmap
- Document the 51-community collection scope and 24-subreddit frozen-corpus subset with a source-level reconciliation.
- Preserve the held-out Stage 1 evaluation boundary while making the distinct development denominators easier to inspect.
- Improve reproducibility documentation for pinned artifacts without reopening the historical analysis as a live service.
Changelog
- 2026-10-15: Project Notes refreshed from the frozen artifact and saved evaluation evidence; the capstone-paper publication checklist remained open.
- Undated frozen milestone: Saved corpus, prompt snapshots, intermediate records and reporting tables were retained as the historical analysis artifact.
North Star
The long-term research direction is a responsible, reviewable workflow for organizing reported course concerns. Any future LLM-assisted workflow would require fresh methodology, privacy review and separate validation; it is not an active product feature here.