Using Fabric to summarise a paper
After my post about Fabric AI I had a bunch of questions asking how to use it in practice. The bad news is if you aren’t comfortable with a terminal or command line then let’s chat offline! For the rest of us, here’s a very simple demo of Fabric.
As I said, I don’t have the time to read all the interesting articles I come across everyday. I wanted a way of quickly summarising and article to see if I wanted learn more.
As an example, I found a recent research paper called “Unlocking the Future of Code Assessment with Generative Artificial Intelligence And Large Language Models”. The paper looked interesting but did I have time to read 22 pages? I could just read the abstract, but that’s too easy. I copied the text from the paper (here), opened up a terminal and typed pbpaste | fabric -sp summarize, which gave me:
**ONE SENTENCE SUMMARY:**
Large Language Models (LLMs) offer potential in automated coding assessments, providing valuable insights and reducing workload for educators, but still require human oversight.
**MAIN POINTS:**
1. Current LLMs can't fully replace human coding assessments.
2. An automated tool was developed for grading coding assignments.
3. The tool provides quantitative scores and qualitative feedback.
4. Evaluated with engineering students over 12 months.
5. Web-based platform ensures academic integrity.
6. Comparisons made between various LLMs and human graders.
7. LLMs excel in understanding and analyzing text and code.
8. Few-shot learning techniques enhance LLM adaptability.
9. LLMs can supplement human graders but can't fully replace them.
10. Structured evaluation rubrics improve LLM assessment accuracy.
**TAKEAWAYS:**
1. LLMs can significantly aid in grading coding assignments.
2. Human oversight remains crucial for accurate assessments.
3. Few-shot learning enhances LLM performance in generating rubrics.
4. Web-based tools can enforce academic integrity.
5. LLMs offer scalable and cost-effective solutions for automated grading.
Ok, that looked interesting, let’s dig deeper with pbpaste | fabric -sp extract_article_wisdom:
# SUMMARY
Large Language Models (LLMs) aid in automated coding assessments but require human oversight for accuracy.
# IDEAS:
- LLMs can't fully replace human assessments in coding.
- An automated grading tool was developed for coding assignments.
- The tool provides both scores and feedback.
- It was evaluated over 12 months with engineering students.
- A web-based platform ensures academic integrity.
- LLMs were compared with human graders.
- LLMs excel in text and code analysis.
- Few-shot learning enhances LLM adaptability.
- LLMs can supplement but not replace human graders.
- Structured rubrics improve LLM assessment accuracy.
- LLMs significantly aid in grading coding assignments.
- Human oversight is crucial for accurate assessments.
- Few-shot learning improves LLM rubric generation.
- Web-based tools can enforce academic integrity.
# QUOTES:
- "Current LLMs can't fully replace human coding assessments."
- "An automated tool was developed for grading coding assignments."
- "The tool provides quantitative scores and qualitative feedback."
- "Evaluated with engineering students over 12 months."
- "Web-based platform ensures academic integrity."
- "Comparisons made between various LLMs and human graders."
- "LLMs excel in understanding and analyzing text and code."
- "Few-shot learning techniques enhance LLM adaptability."
- "LLMs can supplement human graders but can't fully replace them."
- "Structured evaluation rubrics improve LLM assessment accuracy."
# FACTS:
- LLMs offer potential in automated coding assessments.
- Human oversight remains crucial in using LLMs for grading.
- Few-shot learning techniques enhance the adaptability of LLMs.
- Web-based platforms can ensure academic integrity.
- Structured rubrics can improve the accuracy of LLM assessments.
# REFERENCES:
- Automated grading tool for coding assignments
- Web-based platform for academic integrity
- Few-shot learning techniques
- Structured evaluation rubrics
# RECOMMENDATIONS:
- Use LLMs to aid in grading coding assignments.
- Ensure human oversight in LLM-based assessments.
- Implement few-shot learning to enhance LLM performance.
- Utilize web-based tools to maintain academic integrity.
- Develop structured rubrics to improve LLM assessment accuracy.
The output gave me enough information and very quickly.
The entire process of getting the initial high-level summary and the more in-depth summary took less than 20 seconds to complete and that’s the beauty of Fabric. It does the same thing many other tools do, but way more efficiently. I’m getting through 5x the number of articles whilst extracting far more information from every article. Fabric is designed to solve real world problems.
# Notes
In my demo, Fabric was using OpenAI’s GPT-4o model but it’ll work with all the popular LLMs from Anthropic, Azure, Gemini, Groq, Ollam and many more.