Basically I’ve being trying to sort a list of food by date. I took a rough note of each food and date and it gave it to ChatGPT and asked it to format and order it. It formatted it just fine but it couldn’t order it by date. Most of them were in the right place but there were a few out of place. For example at one point it gave me:
– 1st February 2024 – Cookies
– 1st March 2024 – Biscuits
– 1st June 2024 – Soup
– 3rd June 2024 – Chocolate
– 9th May 2024 – Chocolate
– 1st August 2024 – Eggs
– 1st August 2024 – Chicken
– 15th September 2024 – Yogurt
– 25th November 2024 – Sauce
– 16th November 2024 – Soup
– 19th November 2024 – Apple Juice
– 1st November 2024 – Potatoes
– 1st November 2024 – Soup
– 1st May 2024 – Carrots
– 1st January 2025 – Shortbread
– 1st January 2025 – Pasta
– 11th January 2025 – Noodles
– 1st January 2025 – Carrots
– 2nd February 2025 – Cereal
– 7th April 2025 – Green Beans
– 26th March 2025 – Rice
– 28th April 2025 – Pasta
– 1st May 2025 – Stock Cubes
I tried both written and numerical date formats. I also tried asking it to format it and then order it in separate queries so it was only doing one thing at once. I’ve tried a few separate lists and it happened with each. I also got the same results with copilot. When I pointed out the mistake it would say something like “sorry, here’s the correct list” and output the exact same thing. I then remembered something similar happened about a year ago when I asked it to list the Agatha Christie books in publication order and tell me which ones were in thr public domain. It listed them all but there were mistakes in the order. It would then tell me that only books published after (for example) 1926 or later are in the public domain, and then tell me that a book published in 1925 was.
So why can’t it do this? It seems like a very basic task, one that much less sophisticated programs could do. It has so much information, surely some of that information includes which order the months come in and that 25 comes after 16. I’ve had it do relatively complicated calculations based on a rough written description, so ordering a few dates thst are all formatted the same should be a walk in the park, right?
In: Technology
The other explanations are correct and probably easier to understand, but there is another very specific reason that LLMs aren’t good at this, at least not within a single prompt iteration:
Algorithmic complexity, specifically time complexity.
Sorting algorithms inherently take multiple steps to complete. There are many different sorting algorithms, but all of them (with some exceptions that aren’t worth getting into) require you to do some comparison and then swap elements around, many times in some order. Even if you try to parallelize this you still have to do it in multiple steps. You would probably not be surprised to learn that as the size of your list grows, the number of steps does, too.
LLMs, however, have a fixed “depth” to them. That is, they always process information in the name number of steps. Just to throw some jargon out there, they are “transformers” which have something called “self attention” layers which essentially allow the LLM to connect information in one part of the text to other parts of the text. It repeats this process in multiple steps, but crucially it’s the same number of steps every time, regardless of what your prompt is.
It’s quite possible that, despite just being text prediction machines, that LLMs actually have learned the concept of sorting lists, but this limited number of computation steps when predicting what to say next means that they will never be able to solve complex computations in a single prompt cycle.
Given this insight, one experiment you can try to get around this is by responding with “That list isn’t actually sorted. Fix it.” Repeat several times until it gets it right or continues to fail. I haven’t actually tried that with date sorting so I’m not sure if this actually works, however this technique can apply to many situations not just sorting.
Latest Answers