Loading and saving files in a compressed format

Files can get big, very big. and they compress quite well. a 4gig file goes down to 200meg when zipped. Of course saving and loading those files is slower when working on an SSD, but might be faster when working with a spinning disk or over the network. and save lots of space when storing files for longer periods of time

  • Remco Hosman - Alflex Products
  • Jul 11 2018
  • Planned
  • Hardkrash commented
    July 11, 2018 21:06

    I did a quick benchmark using tar using xz compression at level 1 compression a 1.99GB logic data file down to to 75MB in 40 seconds on my 2.6 GHz Intel Core i7, 16GB MacBook Pro.

  • Hardkrash commented
    July 11, 2018 21:06

    Typically loading and saving compressed files is faster, as io bandwidth is a limiting factor. Even with fast SSD drives minimal compression should help. Network drives and other interfaces would benefit as well.
    See:
    - zlib https://en.wikipedia.org/wiki/Zlib
    - xz https://en.wikipedia.org/wiki/Xz

  • woferry commented
    July 11, 2018 21:06

    It would be great if .logicdata files were simply compressed always (or some new extension that becomes the new default), just like how .xlsx files are zip'ed XML files. As noted they do tend to compress quite well so they would take much less space and be easier/faster to transfer.

  • Tom Hughes commented
    01 Apr 22:39

    +1

  • Guest commented
    29 Apr 15:45

    storing 50GB is long in all cases (took me like 5 minutes), but when I compress the resulting file using fast mode with 7-zip it takes like 40 seconds to end with a 1.6BG file!! So it would make sense that doing both operations as a single step (file "compressed-logicdata") from RAM (or pagefile..) to disk, would be both more efficient and convenient.

  • Admin
    Joe Garrison commented
    01 May 14:57

    Def. will do this!  Will optimize for speed and should make a large difference.  Look for this in the Alpha https://discuss.saleae.com/ by ~ July.

    Later on the plan is to offer optional analog downsampling and/or lossy compression.

  • Admin
    Joe Garrison commented
    01 May 14:58

    We'll also try and make it so that saving to disk is a background task that still lets you use the app.