• gay_king_prince_charles [she/her, he/him]@hexbear.net
    link
    fedilink
    English
    arrow-up
    27
    arrow-down
    1
    ·
    2 days ago

    A more technical explanation for this is that LLMs split words into tokens (whole words or parts of words) and then use the “distance” between tokens (how frequently they follow each other) to generate the next most likely token. This results in them not knowing what is actually in a token, just what it’s related to. There is a version of ChatGPT called o1 which should be able to solve this by putting its output back into itself and some more parsing that mitigates this problem, but it costs like $30/mo.

      • I didn’t see that. That’s weird. Oddly enough, 4o-mini outperformed o1 on this front.

        prompt: c o u n t How many states have the letter 'p' in their names

        Let’s list all the U.S. states and check how many contain the letter “p” in their names:

        1. Alabama
        2. Alaska
        3. Arizona
        4. Arkansas
        5. California
        6. Colorado
        7. Connecticut
        8. Delaware
        9. Florida
        10. Georgia
        11. Hawaii
        12. Idaho
        13. Illinois
        14. Indiana
        15. Iowa
        16. Kansas
        17. Kentucky
        18. Louisiana
        19. Maine
        20. Maryland
        21. Massachusetts
        22. Michigan
        23. Minnesota
        24. Mississippi
        25. Missouri
        26. Montana
        27. Nebraska
        28. Nevada
        29. New Hampshire ✔️
        30. New Jersey
        31. New Mexico
        32. New York
        33. North Carolina
        34. North Dakota
        35. Ohio
        36. Oklahoma
        37. Oregon
        38. Pennsylvania ✔️
        39. Rhode Island
        40. South Carolina
        41. South Dakota
        42. Tennessee
        43. Texas
        44. Utah
        45. Vermont
        46. Virginia
        47. Washington
        48. West Virginia
        49. Wisconsin
        50. Wyoming

        The U.S. states that contain the letter “p” in their names are:

        • Pennsylvania
        • Mississippi
        • New Hampshire

        So, there are 3 states with the letter “p” in their names.

          • llama3.1:7b returns:

            Only one state, Pennsylvania (#38), contains the letter “p”

            mistral:7b returns:

            Counting these states, we have 15 states with the letter “p” in their names. This list excludes territories and excludes Alaska because it does not use “The” at the beginning of its official name. [after counting 33 states]

            Gemma2:2b returns: Here are the states that contain the letter “P” in their names:

            Pennsylvania 
            Rhode Island  (this is a tricky one, as it's often forgotten!)
            

            This is generally assessing a fish’s ability to climb a tree, though. The point of LLMs isn’t character recognition, so it doesn’t make much sense to assess LLMs on their ability to do this.

            • plinky [he/him]@hexbear.netOP
              link
              fedilink
              English
              arrow-up
              5
              ·
              20 hours ago

              It’s mainly trolling to show they don’t have cognition or anything resembling reasoning or understanding really. it’s some people (and mine) personal windmills, their ability is bullshit fast enough to mimic a person, not “intelligence”