Experiment with LLM for Personal Knowledge Management
/ 3 min read
Recently, I’ve been experimenting with using Large Language Models (LLMs) for personal knowledge management. However, I’ve found that even when I feed information into an LLM, the output often falls within my own limited range of imagination. On the other hand, LLMs excel at summarizing and organizing diverse information compared to humans. My goal is to leverage these strengths to enhance my intellectual productivity.
Approach Overview
1. Human-Driven Information Dump
First, I write down all the miscellaneous information I acquire into notes:
- Record thoughts and ideas in a Daily Journal.
- Save interesting web articles and resources.
2. LLM-Driven Information Organization
Next, I use an LLM to organize and summarize this information:
- Generate summarized files.
- Implement LLM-based tagging.
- Enclose important keywords with links.
I believe that the interplay between these two steps can lead to more efficient intellectual activity.
Tried Methods
Previously, I experimented with tagging. The knowledge management technique of linking ideas is called Linking Your Thinking (LYT). This time, I decided to try a prompt that more organically summarizes weekly notes and organizes next actions. I’m using OpenAI’s o1-preview model, which has more advanced reasoning capabilities.
Prompt Used
Here is the prompt I actually used:
You are a research assistant for personal knowledge management. You will search a directory containing notes. Summarize the results and consider and suggest the next actions to develop the thinking process.
Consider original and novel research topics. Limit the topics to the field of computer science. For each topic, develop a detailed research plan, including hypotheses, verification methods, expected conclusions, and how it differs from existing research.
In addition, by looking back at the content, please tell me the knowledge gaps that I am currently lacking.
Search Command and Results
I used the following command to search for notes:
find . -type f -name "*.md" -mtime -7 -exec sh -c 'for file; do echo "<File path=\"$file\">\n"; cat "$file"; echo "\n</File>"; done' _ {} +
Results and Challenges
When I actually ran the prompt, I was not satisfied with the output, and it has not yet reached the point of automation. However, by asking various additional questions, I am using it to help with weekly reviews and introspection.
Challenges
- Output Quality: The results obtained from the LLM have not met my expectations.
- Automation Difficulty: I have not been able to establish an automated process because I cannot get satisfactory results.
Conclusion
LLM-powered knowledge management shows promise, but there are still many challenges at this stage. In particular, further consideration needs to be given to prompt design to improve the quality of output and how to maximize the capabilities of LLMs. I will continue to experiment and try to establish more effective knowledge management methods.
References
This paper investigates whether LLMs can generate novel research ideas by comparing research ideas generated by LLMs and human NLP researchers. The study found that LLM-generated ideas were judged to be more novel than those of human experts, while being judged as slightly less feasible. To generate ideas at scale, the paper employed an approach where the LLM generated 4000 ideas for a single research topic. This “large-scale generation” aimed to produce a larger pool of candidate ideas from which high-quality ideas could be selected.
However, it was found that only about 5% (200) of the 4000 ideas were truly unique and non-duplicate. As the generation process continued, the number of duplicate ideas increased, and the number of new ideas decreased. This demonstrates that large-scale generation has limitations as the diversity of ideas gradually diminishes.
IMO: It’s interesting that 200 unique ideas were generated.