comparison with exdupe

[Dialga]

How does dwarfs compare with exdupe? Perhaps a section should be added for comparison?

[mhx]

Hi and welcome!

Didn't know about exdupe and gave it a quick try (-x3 chooses "best compression" per the docs):

$ time ./exdupe -x3 ~/perl-install perl-install-official-binary.exdupe
Compressed 51,161,953,159 B in 1,927,501 files into 2.89 GB                                                                                                

user	30:55.01
system	1:50.93
total	9:03.74

Interestingly, building a binary myself from the v3.0.0 tag in the repo, I seem to get a slightly faster binary:

$ time ./exdupe -x3 ~/perl-install perl-install.exdupe
Compressed 51,161,953,159 B in 1,927,501 files into 2.89 GB                                                                                                

user	29:36.16
system	1:55.65
total	8:47.00

Using mkdwarfs (v0.12.4) on the same data:

$ time mkdwarfs -i ~/perl-install -o perl-install.dwarfs -l9
I 15:22:24.034597 saved 28.03 GiB / 47.49 GiB in 1,780,386/1,927,501 duplicate files
I 15:24:12.463522 compressed 47.49 GiB to 310 MiB (ratio=0.00637466)
I 15:24:12.594030 compression CPU time: 24.83m
I 15:24:12.596842 filesystem created without errors [2.334m]

user	28:30.73
system	1:42.71
total	2:20.12

It's the same input data, it's just that exdupe uses GB and mkdwarfs uses GiB/MiB.

In summary, mkdwarfs runs about 4x faster, uses slightly less CPU, and builds an image that's roughly an order of magnitude smaller.

.rw-r--r--   325,026,313 mhx users  2 Sep 15:24 perl-install.dwarfs
.rw-r--r-- 3,108,800,243 mhx users  2 Sep 15:16 perl-install-official-binary.exdupe
.rw-r--r-- 3,104,227,380 mhx users  2 Sep 15:07 perl-install.exdupe

I'm a bit surprised by the fact that the images created by the two different builds of exdupe are different in size. I'd question if it's able to produce images that are bit-identical (mkdwarfs produces bit-identical images if you configure it to not store create/history timestamps in the metadata).

As far as I can tell (please correct me if I'm wrong), exdupe images also cannot be mounted and need to be extracted in order to get access to the data.

Please let me know if I've overlooked something in my quick test, otherwise I can add a section to the README.

[Dialga]

Hi, thanks for the response, it's good to hear first hand from the developers to see how similar software, which helps end users decide on what is the best solution to adopt for backup.