Your answer could potentially help me un-fuck my website!
Edit: The total capacity is currently 160 GB, so no amount of pruning will free up 1 TB. I need to mount an external share or something.
Edit2: I’m trying this but the command isn’t working despite zero exit status.
Do you have your own physical server or are renting one?
Renting a VPS in the Netherlands.
You may be able to just buy more capacity from the vendor by writing to their support. It’s all containers
Storage isn’t one of the things the sell on their business page but yeah, maybe their team can rig up something for a fee. It’s worth a shot.
Some of these companies can be really good, others can be like “no”. InmotionHosting support were the primary reason I used them despite being slightly more expensive because their support would literally help with any issue even ones no other company would, and they would do stuff like give free storage for a problem like this. At one point I was using like 21TB of space on their service and really testing the limitations of “unlimited storage” they offered on their plan and they were totally cool with it even though I was clearly taking the piss a little. The only thing they held me to was that all things being stored on it had to be frontend accessible.
frontend accessible
What does that mean, not encrypted at rest?
No like physically used as part of the website. Not just sitting on the server that the website was on without actually being in use on the website itself.
At the time what we were doing was archiving all pictures and video from a very very large and active discord. These were then searchable, you could see who posted what pictures and video, in what order historically, even sort by total number of reacts.
I don’t like the fact that discords buries all media for guilds and communities behind a god awful chat. So many screenshots and videos and all kinds of media are just buried in discord, extracting JUST the media and making it viewable and searchable by user etc preserves genuinely emotionally sentimental content. Things like that time a group of friends defeated their first dungeon together and stuff like that deserves to not get forgotten about never to be found ever again somewhere in the depths of a discord server.
Also I am a hoarder.
Idk if you pay for bandwidth or anything but if you do that might be cheaper than exchanging 1 TB of traffic or more over their network
Bandwidth in and out of my VPS is free. Object storage ingress and egress is what will cost me.
IDK I’m drunk right now but maybe they meant “rent a 2TB VM in the same datacenter for a few days”
Unfortunately the storage on their largest VM is only 320 GB.
I have been in similar situations before and there is no easy answer
I have no idea how Peertube works but if you’re on Linux you might be able to turn on filesystem compression… but if the data you’re storing is already compressed (like video) that probably won’t help very much
You could try
sshfslike you mentioned in your post but it might be unreliable :( , maybe you could try NFS or even mount remote storage over Plan 9 protocolTo mount an
sshfsshare I think I need to shell into my PC from the webserver (not the other way around) and I can’t do that.Unlimited destruction upon NAT
Do both hosts have internet connection via IPv6? You could try this over IPv6, usually no NAT, but your ISP might do some kind of firewalling
You could also probably put up a Wireguard (VPN) connection between them since the webserver is not behind NAT
Or also “forward a port” if you have access to the NATing router between your PC and webserver
I might be able to do IPv6 but I would need to drop my VPN (IPv4 only).
I know very little about Wireguard.
Sorry, had to log off for a bit
It seems you need some kind of NAT traversal mechanism. Wireguard is pretty easy to set up, it’s built-into the Linux kernel these days. I’ve heard good things about Tailscale as well.
Sorry I wish I could help more :(
I just setup tailscale on my home Ubuntu jellyfin server the other day for the first time and it was a totally painless experience. Been working great.🤞
I could totally see tailscale working in this scenario lol
regarding your cgnat idea you can try to work with an ssh tunnel between your PC and the VPS to bridge them and do what you were planning to by mounting
You probably have some archival shit under
/var/logthat you can prune. I will usually go level-by-level runningdu -sh {PARENT_PATH}/*to narrow things down one level at a time until I find the offender. Check under /opt and user home dirs, too. Otherwise, you’re probably stuck adding another disk or pushing data out to an S3 bucket.Can I make a BackBlaze bucket and mount that on the filesystem?
Depends on if you can free up enough space on
/to install any packages you may need for the filesystem drivers. I’ve never had to do it either way; if you have a local rig that you can test with, that might be a start.
I don’t know what peertube can do but can you move stuff to /dev/null as a storage device?
There’s a cloud option if you don’t want to self host: https://devnull-as-a-service.com/
lol reminds me of the odd/even API.
npm and its consequences etc etc
That’s a neat idea, but when peertube goes to put the files back into remote storage it would fail to find them locally.
Alternate solution: use atop sorting by fs access and kill the process doing it.
Stop the service that’s doing the thing causing you a problem.
Fix the thing you screwed up
Restart the fixed service that was causing the problem
What could possibly go wrong?
In this case, and I’m making wild assumptions because I don’t have a peertube instance to play with, stop peertube, connect your local system up to it with a vpn to traverse your nat, mount some volume in the local system as the remote systems target for the file system move, then start peertube back up and see if it starts filling your local drive with some data.
When I made the wrong command, the application registered hundreds of move jobs in its database. The developer said there is no way to cancel the jobs. If I restart
peertube.servicethen it will notice the incomplete jobs and immediately resume them. I think I’m going to use btrfs next time so I can take a system snapshot before executing risky server commands.connect your local system up to it with a vpn to traverse your nat
I need a little more detail on this. Here’s my takeaway:
- Host an (e.g.) OpenVPN server on the webserver, after creating the proper certificate infrastructure.
- Connect as client from home PC.
- VPN Success. Now I can access each other on the local subnet that starts with a
10.or something. - Mount home PC’s NFS share on the webserver.
Yeah that’s what I was thinking. Standing up openvpn especially is pretty well documented.
So if I’m reading this right, the issue is going to be that when you revive peertube, it’s going to resume trying to move stuff? Is it running in a container or directly on that machine?
If I’m understanding this correctly, and keep in mind I know nothing of peertube, I’m just an old coder tearing through their source, you could break your config. Configure it not to use object storage and restart. If can’t find the storage it’s trying to move stuff from, it can’t move that stuff.
OBJECT_STORAGE: { ENABLED: config.get<boolean>('object_storage.enabled'),Or make the target folder/file read only
