• BlueKey@fedia.ioOP
    link
    fedilink
    arrow-up
    59
    ·
    1 month ago

    I’m new to Go and wanted to copy some text-data from a stream into the outputstream of the HTTP response. I was copying the data to and from a []byte with a single Read() and Write() call and expexted everything to be copied as the buffer is always the size of the while data. Turns out Read() sometimes fills the whole buffer and sometimes don’t.
    Now I’m using io.Copy().

    • dfyxA
      link
      fedilink
      arrow-up
      51
      ·
      1 month ago

      Note that this isn’t specific to Go. Reading from stream-like data, be it TCP connections, files or whatever always comes with the risk that not all data is present in the local buffer yet. The vast majority of read operations returns the number of bytes that could be read and you should call them in a loop. Same of write operations actually, if you’re writing to a stream-like object as the write buffers may be smaller than what you’re trying to write.

      • bl_r@lemmy.dbzer0.com
        link
        fedilink
        arrow-up
        12
        ·
        1 month ago

        I’ve run into the same problem with an API server I wrote in rust. I noticed this bug 5 minutes before a demo and panicked, but fixed it with a 1 second sleep. Eventually, I implemented a more permanent fix by changing the simplistic io calls to ones better designed for streams

        • dfyxA
          link
          fedilink
          arrow-up
          7
          ·
          1 month ago

          The actual recommended solution is to just read in a loop until you have everything.

      • xthexder@l.sw0.com
        link
        fedilink
        arrow-up
        3
        ·
        1 month ago

        Ah yes… several years ago now I was working on a tool called Toxiproxy that (among other things) could slice up the stream chunks into many random small pieces before forwarding them along. It turned out to be very useful for testing applications for this kind of bug.