Página 1 de 7 123 ... ÚltimoÚltimo
Resultados 1 a 10 de 70
  1. #1
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239

    Online backups for the truly paranoid

    Estou procurando uma solução para backups utilizando "untrusted remote storage", praticamente o caso de qualquer servidor dedicado ou VPS

    Certamente existem serviços cloudy que oferecem armazenamento criptografado com (multiplas) chaves em seu poder mas o meu objetivo é utilizar dedicados.

    Temporariamente (temporário = permanente mal feito) venho utilizando tar + gpg, o que tem implicado em grande utilização de CPU e espaço em disco, o tipo de impacto que o Mindnet postou recentemente com o novo sistema de backup do cpanel.

    Na busca pelo software adequado, não faltam soluções nem divergências Esta discussão 3 anos atrás sobre o bup que descambou para o Attic (**) é interessante. Também existem alhures indicações positivas sobre o Camlistore -- cuja home page está alinhada com o que procuro (*) exceto por "Many bits and pieces are actively being developed, so be prepared for bugs and unfinished features". Não estou preparado para bugs em sistemas de backup

    Temporariamente vou experimentar o Attic.


    (*)
    Things Camlistore believes:

    • Your data is entirely under your control
    • Paranoid about privacy, everything private by default
    • No SPOF: don't rely on any single party (including yourself)
    • Your data should be alive in 80 years, especially if you are

    (**)
    Attic is one of the new-generation hash-backup tools (like obnam, zbackup, Vembu Hive etc). It provides encrypted incremental-forever (unlike duplicity, duplicati, rsnapshot, rdiff-backup, Ahsay etc) with no server-side processing and a convenient CLI interface, and it does let you prune old backups.

    All other common tools seem to fail on one of the following points

    - Incremental forever (bandwidth is expensive in a lot of countries)

    - Untrusted remote storage (so i can hook it up to a dodgy lowendbox VPS)

    - Optional: No server-side processing needed (so i can hook it up to S3 or Dropbox)

    If your backup model is based on the old' original + diff(original, v1) + diff(v1, v2).. then you're going to have a slow time restoring. rdiff-backup gets this right by reversing the incremental chain. However, as soon as you need to consolidate incremental images, you lose the possibility of encrypting the data (since encrypt(diff()) is useless from a diff perspective).

    But with a hash-based backup system? All restore points take constant time to restore.

    Duplicity, Duplicati 1.x, and Ahsay 5 don't support incremental-forever. Ahsay 6 supports incremental-forever at the expense of requiring trust in the server (server-side decrypt to consolidate images). Duplicati 2 attempted to move to a hash-based system but they chose to use fixed block offsets rather than checksum-based offsets, so the incremental detection is inefficient after an insert point.
    Última edição por 5ms; 11-06-2017 às 12:26.

  2. #2
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239

    Angry

    ThomasWaldmann commented on Apr 9

    attic is not compatible with openssl 1.1.0

    use borgbackup.


    BorgBackup

    Easy installation on multiple platforms

    We offer single-file binaries that do not require installing anything - you can just run them on these platforms:

    Linux
    Mac OS X
    FreeBSD
    OpenBSD and NetBSD (no xattrs/ACLs support or binaries yet)
    Cygwin (not supported, no binaries yet)
    Linux Subsystem of Windows 10 (not supported)

    https://borgbackup.readthedocs.io/en/stable/
    Mentir até ser verdade ...
    Última edição por 5ms; 11-06-2017 às 13:05.

  3. #3
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239
    Prossegui no teste do Attic utilizando 2 servidores rodando Debian 8 e instalação via apt -- que alterou arquivos de configuração do Lighttpd (não instalado via apt) causando a saida do ar do Lighttpd e necessidade de reparos.

    Aqui um comparativo entre a solução temporaria tar+gpg vs attic:


    tar+gpg

    gpg --output arquivo.tar.gpg --cipher-algo AES256 \
    --compress-algo ZLIB --verbose --symmetric arquivo.tar

    59324446720 arquivo.tar (load average: 4.00 E3-1220)
    33083011437 arquivo.tar.gpg (load average: 1.00 E3-1220)

    ~31GB local para transferir



    attic (rodando nos 2 servidores)


    Servidor origem

    # find diretorio -type f | wc -l
    137680

    # du -s -B1 diretorio
    59382378496



    attic init --encryption=passphrase ssh://usuario@dominio: porta/path

    #attic create ssh://usuario@dominio: porta/path::diretorio.attic diretorio (load average: 1.00 E3-1220)

    # attic info sh://usuario@dominio: porta/path::diretorio.attic

    Código:
    Original size      Compressed size    Deduplicated size
      59.15 GB             33.43 GB             26.46 GB

    Servidor destino:

    # find diretorio -type f | wc -l
    5019

    # du -s -h
    25G



    O próximo passo é conhecer o custo do "backup incremental" com e sem modificações no "diretorio"
    Última edição por 5ms; 11-06-2017 às 15:43.

  4. #4
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239

    Borg

    Ocorreu o seguinte impasse:

    attic pode ser instalado via apt no Debian 8 mas não está disponivel para Debian 9 e não compila no 9 devido a mudanças no openssl.

    borgbackup pode ser instalado via apt no Debian 9 (versão 1.09) mas não está disponivel para Debian 8. Entretanto existe a possibilidade de download do binario (versão 1.10) mas mostrou incompatibilidade com a 1.09.

    borg é um fork do attic mas não é compativel. Aparentemente o desenvolvimento está focado no borg.

    O binário 1.10 roda no Debian 8, 9 e outras distribuições Linux.

    Copiei o borg 1.10 no VPS Storage Debian 9 da HttpZoom e no dedicado Debian 8 da Dacentec. Estão "se falando".

    Para testar o borg utilizei um diretório backup do Google Apps.

    Cliente: Servidor Dacentec

    # du -s -B1 gapps
    58245558272 gapps

    # du -s -h gapps
    55G

    # find gapps -type f | wc -l
    40649

    tar+gpg

    30586597483 gapps.tar.gpg



    Dacentec -> HttpZoom usando borg com armazenamento criptogrado


    borg create ssh://..

    real 43m57.148s
    user 30m28.504s
    sys 2m31.916s

    borg info

    Código:
                           Original size      Compressed size    Deduplicated size
    All archives:               58.15 GB             43.12 GB             38.34 GB
    
                           Unique chunks         Total chunks
    Chunk index:                   57140                60854

    Server: VPS httpZoom

    # du -s -B1 gapps
    38347456512

    # du -s -h gapps
    36G

    # find gapps -type f | wc -l
    86



    Compression

    All data can be compressed by lz4 (super fast, low compression), zlib (medium speed and compression) or lzma (low speed, high compression).

    Why is my backup bigger than with attic? Why doesn’t Borg do compression by default?

    Attic was rather unflexible when it comes to compression, it always compressed using zlib level 6 (no way to switch compression off or adjust the level or algorithm).

    Borg offers a lot of different compression algorithms and levels. Which of them is the best for you pretty much depends on your use case, your data, your hardware – so you need to do an informed decision about whether you want to use compression, which algorithm and which level you want to use. This is why compression defaults to none.



    CPU client:

    borg create: does chunking, hashing, compression, crypto (high CPU usage) chunks cache sync: quite heavy on CPU, doing lots of hashtable operations. borg extract: crypto, decompression (medium to high CPU usage) borg check: similar to extract, but depends on options given. borg prune / borg delete archive: low to medium CPU usage borg delete repo: done on the server It won’t go beyond 100% of 1 core as the code is currently single-threaded. Especially higher zlib and lzma compression levels use significant amounts of CPU cycles. Crypto might be cheap on the CPU (if hardware accelerated) or expensive (if not).

    CPU server:

    It usually doesn’t need much CPU, it just deals with the key/value store (repository) and uses the repository index for that.

    borg check: the repository check computes the checksums of all chunks (medium CPU usage) borg delete repo: low CPU usage





    Compatibility notes

    EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER CHANGES (like when going from 0.x.y to 1.0.0 or from 1.x.y to 2.0.0).

    NOT RELEASED DEVELOPMENT VERSIONS HAVE UNKNOWN COMPATIBILITY PROPERTIES.

    THIS IS SOFTWARE IN DEVELOPMENT, DECIDE YOURSELF WHETHER IT FITS YOUR NEEDS.

  5. #5
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239

    attic

    backup incremental (arquivos novos ou alterados)
    Código:
    Duration: 32.27 seconds
    Number of files: 137675
    
                           Original size      Compressed size    Deduplicated size
    This archive:               59.15 GB             33.43 GB              1.63 MB
    All archives:              118.31 GB             66.86 GB             26.46 GB

    backup incremental (arquivos apagados)
    Código:
    Duration: 29.68 seconds
    Number of files: 137618
    
                           Original size      Compressed size    Deduplicated size
    This archive:               59.15 GB             33.43 GB            581.32 kB
    All archives:              177.46 GB            100.30 GB             26.47 GB

    remoção do primeiro backup
    Código:
                           Original size      Compressed size    Deduplicated size
    Deleted data:              -59.15 GB            -33.43 GB             -1.64 MB
    All archives:              118.31 GB             66.86 GB             26.46 GB

    remoção de backup incremental
    Código:
                           Original size      Compressed size    Deduplicated size
    Deleted data:              -59.15 GB            -33.43 GB           -871.96 kB
    All archives:               59.15 GB             33.43 GB             26.46 GB

    backup incremental (sem ter ocorrido modificação de arquivos)
    Código:
    Duration: 31.25 seconds
    Number of files: 137618
    
                           Original size      Compressed size    Deduplicated size
    This archive:               59.15 GB             33.43 GB             35.88 kB
    All archives:              118.31 GB             66.86 GB             26.46 GB
    Última edição por 5ms; 11-06-2017 às 21:25.

  6. #6
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239

    borg create --compression zlib,9

    Citação Postado originalmente por 5ms Ver Post
    Para testar o borg utilizei um diretório backup do Google Apps.

    Cliente: Servidor Dacentec

    # du -s -B1 gapps
    58245558272 gapps

    # du -s -h gapps
    55G

    # find gapps -type f | wc -l
    40649

    tar+gpg

    30586597483 gapps.tar.gpg



    Dacentec -> HttpZoom usando borg com armazenamento criptogrado


    borg create ssh://..

    real 43m57.148s
    user 30m28.504s
    sys 2m31.916s

    borg info

    Código:
                           Original size      Compressed size    Deduplicated size
    All archives:               58.15 GB             43.12 GB             38.34 GB
    
                           Unique chunks         Total chunks
    Chunk index:                   57140                60854

    Server: VPS httpZoom

    # du -s -B1 gapps
    38347456512

    # du -s -h gapps
    36G

    # find gapps -type f | wc -l
    86

    borg create --compression zlib,9 ssh://..

    real 128m21.106s
    user 116m26.876s
    sys 2m7.608s

    borg info

    Código:
                           Original size      Compressed size    Deduplicated size
    This archive:               58.25 GB             30.52 GB             27.29 GB
    All archives:               58.25 GB             30.52 GB             27.29 GB
    
                           Unique chunks         Total chunks
    Chunk index:                   57649                61368

    Server: VPS httpZoom

    du -s -B1 gapps
    27298213888

    du -s -h gapps
    26G

    find gapps -type f | wc -l
    79

  7. #7
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239
    gapps atualizado no servidor cliente (Dacentec)

    Anterior
    # du -s -B1 gapps
    58245558272 gapps
    # du -s -h gapps
    55G
    # find gapps -type f | wc -l
    40649

    Atualizado
    du -s -B1 gapps
    58247561216
    du -s -h gapps
    55G
    find gapps -type f | wc -l
    40674


    backup incremental

    borg create --compression zlib,9 ssh://..

    real 0m27.658s
    user 0m14.160s
    sys 0m0.984s

    borg info

    Código:
                           Original size      Compressed size    Deduplicated size
    This archive:               58.15 GB             30.48 GB              2.68 MB
    All archives:              116.40 GB             61.00 GB             27.29 GB
    
                           Unique chunks         Total chunks
    Chunk index:                   57694               122716

  8. #8
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239
    Why was Borg forked from Attic?

    Borg was created in May 2015 in response to the difficulty of getting new code or larger changes incorporated into Attic and establishing a bigger developer community / more open development.

    More details can be found in ticket 217 that led to the fork.

    Borg intends to be:


    • simple:
      • as simple as possible, but no simpler
      • do the right thing by default, but offer options

    • open:
      • welcome feature requests
      • accept pull requests of good quality and coding style
      • give feedback on PRs that can’t be accepted “as is”
      • discuss openly, don’t work in the dark

    • changing:
      • Borg is not compatible with Attic
      • do not break compatibility accidentally, without a good reason or without warning. allow compatibility breaking for other cases.
      • if major version number changes, it may have incompatible changes





    What are the differences between Attic and Borg?

    Borg is a fork of Attic and maintained by “The Borg collective”.

    Here’s a (incomplete) list of some major changes:

    • more open, faster paced development (see issue #1)
    • lots of attic issues fixed (see issue #5)
    • less chunk management overhead (less memory and disk usage for chunks index)
    • faster remote cache resync (useful when backing up multiple machines into same repo)
    • compression: no, lz4, zlib or lzma compression, adjustable compression levels
    • repokey replaces problematic passphrase mode (you can’t change the passphrase nor the pbkdf2 iteration count in “passphrase” mode)
    • simple sparse file support, great for virtual machine disk files
    • can read special files (e.g. block devices) or from stdin, write to stdout
    • mkdir-based locking is more compatible than attic’s posix locking
    • uses fadvise to not spoil / blow up the fs cache
    • better error messages / exception handling
    • better logging, screen output, progress indication
    • tested on misc. Linux systems, 32 and 64bit, FreeBSD, OpenBSD, NetBSD, Mac OS X


    Borg is not compatible with original attic (but there is a one-way conversion).
    Última edição por 5ms; 12-06-2017 às 01:27.

  9. #9
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239

    Comparison of Attic vs Bup vs Obnam

    Dan Williams
    2015-03-31

    1. Backup/Restore home

    Number of entities: 26,785
    Number of files: 24,838
    Total data set size: 15GB

    2. Backup/Restore fileserver

    Number of entities: 5,120,641
    Number of files: 4,385,287
    Total data set size: 7.2TB

    http://librelist.com/browser/attic/2...-bup-vs-obnam/

    Obs: O nome borg parece ser uma homenagem a Jonas Borg, criador e único desenvolvedor do attic. Essencialmente, attic é um projeto pessoal colocado à disposição mas não aberto a colaborações. Em maio de 2015, um grupo de usuários decidiu pelo fork e o programa original foi significativamente modificado desde então. Os resultados do artigo não se aplicam ao borg.
    Última edição por 5ms; 12-06-2017 às 12:32.

  10. #10
    WHT-BR Top Member
    Data de Ingresso
    Dec 2010
    Posts
    17,239

    Para os truly paranoid

    Thomas Waldmann, "dono" do borgbackup:

    • Borg exists to be able to change things. So if you don't like or can't live with a changing software, don't use it.

    Usuários:

    • I think extracting files of an archive should possible every time independent of the used version of borgbackup. At least with latest bb version should extracting of older version of archive possible.


    • I really like the promise of attic to keep backwards compatibility forever for storage.


    • Who knows when i will need to restore this old backup?


    • Tar files have lasted decades, I don't see a reason not to strive to reach that kind of quality.


    Thomas Waldmann:

    • I am still proposing just breaking compat now and then. Someone who wants conservative long time archiving might better off using tar (or attic) maybe.


    • Keeping something compatible forever sounds great, but from development and maintenance standpoint, it is a pain. You have now not just the bugs in the latest code, but all remaining bugs in all the old versions, too.

    • BTW, I pushed some code to the repo. It is not compatible with attic due to changes of the magic strings.


    Usuário: This commit changes magic number without a good justification. this seem to be contrary to even the goals stated in the summary here (namely "Don't break it accidentally / without good reason / without warning")

    https://github.com/borgbackup/borg/issues/1

    Enquanto esse sujeito estiver no comando, borg é SaaP (Software as a Playground)
    Última edição por 5ms; 12-06-2017 às 22:40.

Permissões de Postagem

  • Você não pode iniciar novos tópicos
  • Você não pode enviar respostas
  • Você não pode enviar anexos
  • Você não pode editar suas mensagens
  •