• BlueKey@fedia.ioOP
    link
    fedilink
    arrow-up
    60
    ·
    4 months ago

    I’m new to Go and wanted to copy some text-data from a stream into the outputstream of the HTTP response. I was copying the data to and from a []byte with a single Read() and Write() call and expexted everything to be copied as the buffer is always the size of the while data. Turns out Read() sometimes fills the whole buffer and sometimes don’t.
    Now I’m using io.Copy().

    • dfyx@lemmy.helios42.de
      link
      fedilink
      arrow-up
      51
      ·
      4 months ago

      Note that this isn’t specific to Go. Reading from stream-like data, be it TCP connections, files or whatever always comes with the risk that not all data is present in the local buffer yet. The vast majority of read operations returns the number of bytes that could be read and you should call them in a loop. Same of write operations actually, if you’re writing to a stream-like object as the write buffers may be smaller than what you’re trying to write.

      • bl_r@lemmy.dbzer0.com
        link
        fedilink
        arrow-up
        13
        ·
        4 months ago

        I’ve run into the same problem with an API server I wrote in rust. I noticed this bug 5 minutes before a demo and panicked, but fixed it with a 1 second sleep. Eventually, I implemented a more permanent fix by changing the simplistic io calls to ones better designed for streams

      • xthexder@l.sw0.com
        link
        fedilink
        arrow-up
        3
        ·
        4 months ago

        Ah yes… several years ago now I was working on a tool called Toxiproxy that (among other things) could slice up the stream chunks into many random small pieces before forwarding them along. It turned out to be very useful for testing applications for this kind of bug.