Loading and saving files in a compressed format

Files can get big, very big. and they compress quite well. a 4gig file goes down to 200meg when zipped. Of course saving and loading those files is slower when working on an SSD, but might be faster when working with a spinning disk or over the network. and save lots of space when storing files for longer periods of time

  • Remco Hosman - Alflex Products
  • Jul 11 2018
  • Hardkrash commented
    July 11, 2018 21:06

    I did a quick benchmark using tar using xz compression at level 1 compression a 1.99GB logic data file down to to 75MB in 40 seconds on my 2.6 GHz Intel Core i7, 16GB MacBook Pro.

  • Hardkrash commented
    July 11, 2018 21:06

    Typically loading and saving compressed files is faster, as io bandwidth is a limiting factor. Even with fast SSD drives minimal compression should help. Network drives and other interfaces would benefit as well.
    See:
    - zlib https://en.wikipedia.org/wiki/Zlib
    - xz https://en.wikipedia.org/wiki/Xz

  • woferry commented
    July 11, 2018 21:06

    It would be great if .logicdata files were simply compressed always (or some new extension that becomes the new default), just like how .xlsx files are zip'ed XML files. As noted they do tend to compress quite well so they would take much less space and be easier/faster to transfer.