A few years ago I wrote this post encouraging new developers to learn jq, awk and sed.
I still think it is worthwhile to do so, because these tools are everywhere and make processing text and structured text easy.
However, I think there’s a new text manipulation tool in town that you should experiment with too. That’s an LLM.
Today, I wanted to take the latest posts from the hacker news ‘whoishiring’ user posts and get a ratio. I wanted to find the ratio of ‘who is hiring’ comments to the number of ‘who wants to be hired’. I thought this would be a proxy for the health of the job market, at least the job market of companies who post on HN.
I could have solved this with text manipulation tools, maybe some google sheets manipulation, but instead I pasted the results into ChatGPT and asked it to extract out the comments and show me the ratio.
It took about 30 seconds. Amazing.
I’ve also used it for fiddly jq commands. Instead of peering through the jq manual and trying to figure out how to capture the first value of an array and to later extract a certain key, I just describe what I want and the LLM gives it to me.
Tips for doing text manipulation:
- use a new window every time
- iterate on your prompt
- give examples of current and future state
Why do I think folks should still learn those other tools and not rely just on LLMs? Because the base tools are far more reliable, cheaper, and scriptable. For me, LLMs shine in text manipulation when the problem is small and adhoc, especially when the text is not well structured or completely unstructured. But if I’m writing a script to extract data from log files repeatedly, I think using the LLM to help write the script is a bigger win and better in the long term than to use the LLM to actually do the extraction.
Either way it is a powerful tool, so next time you want to do some text manipulation, try your LLM.
Follow me on Twitter
Follow me on Blue Sky