I want to have a mirror of my local music collection on my server, and a script that periodically updates the server to, well, mirror my local collection.

But crucially, I want to convert all lossless files to lossy, preferably before uploading them.

That’s the one reason why I can’t just use git - or so I believe.

I also want locally deleted files to be deleted on the server.

Sometimes I even move files around (I believe in directory structure) and again, git deals with this perfectly. If it weren’t for the lossless-to-lossy caveat.

It would be perfect if my script could recognize that just like git does, instead of deleting and reuploading the same file to a different location.

My head is spinning round and round and before I continue messing around with find and scp it’s time to ask the community.

I am writing in bash but if some python module could help with it I’m sure I could find my way around it.

TIA

additional info:

  • Not all files in the local collection are lossless. A variety of formats.
  • The purpose of the remote is for listening/streaming with various applications
  • The lossy version is for both reducing upload and download (streaming) bandwidth. On mobile broadband FLAC tends to buffer a lot.
  • The home of the collection (and its origin) is my local machine.
  • The local machine cannot act as a server
  • bastion@feddit.nl
    4·
    9 days ago

    Make a script. I’d use xonsh or python with sh.py.

    • create a dict for remote to local filename map
    • walk your local collection
      • for each file, determine what the correct remote name (including a valid extension) would be, and add the pair to the dict, with remote filenames as keys, local filenames as values
    • make a set like local_munged_names from that dict’s keys
    • walk your remote tree, and store the filenames in a set like remote_names
    • names_to_upload = local_munged_names - remote_names
    • for each name in names to upload, look up the local filename from the remote to local filename map. Then, encode it if it needs encoding, and upload.
    • vandsjov@feddit.dk
      2·
      6 days ago

      I like this. I would probably over complicate things with a index CSV file (or SQL) that stores checksum values of files to identify renamed or moved files.

    • A_norny_mousse@feddit.orgOP
      2·
      9 days ago

      xonsh or python with sh.py.

      Very interesting!

      And thanks for the coding tips. It seems git is not the best option here because it keeps a full history of all files in their fullness - a gigantic waste of space in the case of a media collection.

      I am now thinking more rsync minus lossless formats, then deal with lossless formats separately.

      • bastion@feddit.nl
        1·
        2 days ago

        you know, you could also either include a lossy copy next to the lossless ones, then rsync only lossy extensions, or, if that pollutes your collection, have a separate but identically-structured directory tree, where all your lossless files have lossy copies. Then, you can rsync both folders (send-only) to your single remote (lossy extensions only).

        but, yeah, Git really isn’t the tool for this, agreed.