Just started self hosting this instance. Nothing on the docs mentioned anything about storage considerations.

  • Ruud@lemmy.worldM
    link
    fedilink
    English
    arrow-up
    274
    arrow-down
    3
    ·
    1 year ago

    This is lemmy.world after 4 weeks:

    58G	pictrs
    34G	postgres
    
      • manitcor@lemmy.intai.tech
        link
        fedilink
        English
        arrow-up
        57
        ·
        edit-2
        1 year ago

        im running 50 users right now, subbed to A LOT of communities, seeing db growth of about 100mb per day.

          • terribleplan@lemmy.nrd.li
            link
            fedilink
            English
            arrow-up
            69
            ·
            edit-2
            1 year ago

            It’s all about how many communities your user(s) subscribe to since your instance basically acts as a mirror for those.

            My instance has been running for 23 days, and I am pretty much the only active local user:

            7.3G    pictrs
            5.3G    postgres
            

            edit: I may have a slight Reddit Lemmy problem

            • Pleonasm@programming.dev
              link
              fedilink
              English
              arrow-up
              9
              ·
              1 year ago

              So if you’re the only user (let’s assume for ease) then, that represents all the updates (posts, comments, votes) from each community that you are subscribed to?

              • terribleplan@lemmy.nrd.li
                link
                fedilink
                English
                arrow-up
                16
                ·
                1 year ago

                Yeah, and I purposely subscribe to (or sometimes have a dedicated “federation helper bot” account I run subscribe to) most of the most popular communities on the most popular instances so I can get a decent sampling of what’s going on in the fediverse on the “All” feed. So I assume my storage usage is maybe a bit higher than what an “average” single-user instance may be…

                • bdonvrA
                  link
                  fedilink
                  English
                  arrow-up
                  4
                  ·
                  1 year ago

                  lmao same here. I have a spare account that I use to sub to everything worth subbing to. I haven’t automated it yet though.

                • ipkpjersi@lemmy.one
                  link
                  fedilink
                  English
                  arrow-up
                  3
                  ·
                  1 year ago

                  Ooh, that’s a really good idea, I need a federation helper bot/account when I start self-hosting a Lemmy instance!

                  • terribleplan@lemmy.nrd.li
                    link
                    fedilink
                    English
                    arrow-up
                    3
                    ·
                    1 year ago

                    Yeah it’s not automated or anything, I just pop an incognito window and use it when there is a communitI think is worth seeing sometimes in “All” (or just for archiving purposes) but don’t want to clutter “Subscribed”. I may make something to auto-subscribe to communities meeting some criteria or something at some point in the future…

            • Alexander Kehr@lemmy.alexware.systems
              link
              fedilink
              English
              arrow-up
              3
              ·
              1 year ago

              Do you also post stuff? I mean my instance is only about an hour old, but I’ve subscribed to some communities, yet I don’t see the picture service consuming the S3 storage I’ve configured

              • terribleplan@lemmy.nrd.li
                link
                fedilink
                English
                arrow-up
                14
                arrow-down
                1
                ·
                1 year ago

                Lemmy caches every thumbnail of every post for like a month or something using Pictrs, so that storage will eventually hit a sort of equilibrium and start growing much more slowly (only reflecting post/thumbnail volume during the cache time).

                Between profile images, community banners/icons, post images etc. there are probably a few dozen images that will be sticking around for the long haul at the moment.

              • bdonvrA
                link
                fedilink
                English
                arrow-up
                5
                ·
                1 year ago

                Your instance only caches thumbnails, so it won’t take much space. The full images are served from the remote instance. So you basically only store whatever your users upload.

          • ChickenBoo@lemmy.jnks.xyz
            link
            fedilink
            English
            arrow-up
            23
            arrow-down
            1
            ·
            1 year ago

            It won’t scale linearly. A lot of those users will be subscribed to subs the instance is already replicating. It would only be new subs that would add to the growth.

            • bdonvrA
              link
              fedilink
              English
              arrow-up
              4
              ·
              1 year ago

              And only active subs. And even then, it’s just text and tiny thumbnails.

      • Dran@lemmy.world
        link
        fedilink
        English
        arrow-up
        16
        ·
        1 year ago

        Question if you know: does a lemmy instance have to be publically accessable to work? Like, if I make an instance on my homelab can the instance “fetch” content and serve it faster locally? Could I reply to a post and have others see it? Etc

    • HappyHam@lemm.ee
      link
      fedilink
      English
      arrow-up
      18
      ·
      1 year ago

      Now I wonder how viable it would be to support video hosting. The answer is almost certainly “God no!”

      • BigWigglyStyle@lemmynsfw.com
        link
        fedilink
        English
        arrow-up
        17
        ·
        1 year ago

        At the end of the day the vast majority of what needs to be saved is text. If media content is embedded, the the server just has to save the path to the file not the file itself.

    • bdonvrA
      link
      fedilink
      English
      arrow-up
      11
      ·
      edit-2
      1 year ago

      Yeah lemmy seems to use just about nothing for data storage.

    • lightrush@lemmy.ca
      link
      fedilink
      English
      arrow-up
      6
      arrow-down
      1
      ·
      edit-2
      1 year ago

      Feels like this will benefit from some sort of fuzzy deduplication in the pictrs storage. I bet there are a lot of similar pics in there. E.g. if one pic or a gif is very similar to another, say just different quality or size, or compression, it should keep only one copy. It might already do this for the same files uploaded by different people as those can be compared trivially via hashing, but I doubt it does similarity based deduplication.