Home
Reading
Searching
Subscribe
Sponsors
Statistics
Posting
Contact
Spam
Lists
Links
About
Hosting
Filtering
Features Download
Marketing
Archives
FAQ
Blog
 
Gmane

From: Steve Wilson <stevew-olO2ZdjDehc3uPMLIKxrzw <at> public.gmane.org>
Subject: Re: ZFS + glusterfs on two or three nodes
Newsgroups: gmane.linux.file-systems.zfs.user
Date: Monday 9th December 2013 15:05:01 UTC (over 5 years ago)
Another distributed filesystem worth consideration is MooseFS 
(moosefs.org) or a fork of MooseFS called LizardFS (lizardfs.org). We 
have about 150TB of redundant storage (300TB raw storage) in MooseFS 
filesystems.  These have been in production for a couple of years and 
we've been very impressed with their stability, especially after an 
initial attempt with GlusterFS that was less than satisfactory.

MooseFS doesn't have automatic failover of its master server but in 
practice that hasn't proven to be a major drawback for us.  As with 
other similar filesystems, performance suffers when file I/O involves 
many small files.  But I think it's at least on par with GlusterFS in 
that regard.  MooseFS is extremely easy to set up and administer; I 
would definitely recommend it for consideration to anyone looking at 
GlusterFS.

Steve

On 12/09/2013 08:48 AM, Gordan Bobic wrote:
> Does it handle failure of a single machine even remotely gracefully? 
> What about a splitbrain scenario?
>
> I understand they added some rudimentary quorum support  to GLFS since 
> I last used it - and back then splitbraining was a major issue that 
> was sufficient to prevent deployment in any production environment 
> where I would have otherwise wanted to use it (and where the 
> performance profile of it was such that it wouldn't be crippling).
>
>
>
>
> On Mon, Dec 9, 2013 at 1:26 PM, Kate Ward
 > wrote:
>
>     I'm using GlusterFS to host ~7T of data on two HP N54L MicroServer
>     machines (4x2T HDD + 128G SSD) running Ubuntu 12.04, each backed
>     by ZFS using lz4 compression. Performance is reasonable (45-85
>     MB/s using AFP over 1Gbit network), but less than what I got
>     talking directly to ZFS one machine (85-115 MB/s). /FYI, I'm
>     mounting via the glusterfs FUSE layer, and not via NFS./
>
>     I wanted some important data replicated (r=2) but most
>     non-important data (e.g. movies) unreplicated (r=1). To achieve
>     this, I had to setup two separate ZFS mounts (tankz/gfs/r1 and
>     tankz/gfs/r2) on each machine, and then setup two separate
>     GlusterFS pools (gfsr1 and gfsr2) with the appropriate replication
>     values. This was required due to GlusterFS not being able to mix
>     replications per-file or per-path in a single pool.
>
>     For backups, I am currently backing up the mounted GlusterFS pools
>     weekly to a 3rd machine with 4x3T on ZFS using rsync. I've had no
>     issues with this, and "restores" are simply a cd into the
>     appropriate directory and cp the file to where I need it.
>
>     I'm looking at adding a 3rd machine to the GlusterFS pool. To
>     properly achieve this for my r=2 files, I'm expecting to need a
>     convoluted (and probably not recommended) matrix setup that will
>     allow a single machine to die. Gluster doesn't support odd machine
>     count numbers for r=2 pools, but I don't want a 4th machine yet.
>     My solution for three machines A, B, and C will be to map three
>     r=2 pair as [(A,B), (B, C), (A, C)] as a single pool. Data won't
>     be balanced properly between machines, but I'm not personally
>     concerned at this time as my r=1 dataset dominates in space
>     anyway. The r=1 dataset thankfully won't need this mess.
>
>     GlusterFS has been incredibly stable for me after ~1y of daily
>     usage. I have ~40k raw photos, multiple T of video and audio data,
>     as well as many years worth of downloads hosted on it. Setup can
>     be accomplished in an afternoon, and once it is going, I've had
>     practically zero maintenance needs.
>
>     - kate
>
>
>     On Sat, Dec 7, 2013 at 8:05 PM, Arman Khalatyan
     > wrote:
>
>         István,
>
>         ZFS+Gluster sounds yet another challenge task for the servers.
>         If you require performance you should have powerful servers.
>         ZFS and Gluster are require CPU power.
>
>         I like idea of Gluster, it is brilliant, no metadataserver
>         like in other parallel file-systems, if you loose one brick
>         you can restore only data from the brick. Unfortunately
>         performance is degrading on many files.
>
>         Glusterfs was ok on SL6.4 with  XFS on 6jbods with 4 servers
>         until number of files became more than 1M per brick.
>         Something else to keep in mind: by the gluster design on brick
>         timeout current io goes to active bricks. Later gluster 
>         generates - - - - -T per file to point out where actual data
>         is lying. All this together causes that ls, find , rsync,
>         becoming  very slow. Even re-balance does not fix T files. One
>         can fix them using self made scrips on bricks. More
>         information you can find in:
>         http://joejulian.name/blog/dht-misses-are-expensive/
>
>         On XFS:
>         In some point we had a very bad power-cut XFS got some files
>         with 0 size, later we found we hit some bug on xfs-logbuf
>         which was fixed in the next kernel release.
>
>         So keep in mind always to make a backup for whatever fs you
>         are using.:)
>
>         Arman.
>
>
>
>
>
>
>
>
>
>         On Fri, Dec 6, 2013 at 4:00 PM, Branden Timm
>         > wrote:
>
>             I wonder if you'd be willing to share a bit more about
>             your experience?  We are running Gluster on XFS right now,
>             but I have some hardware on order to build some new
>             Gluster on ZFS boxes.  We've found that with XFS the total
>             system bandwidth is great, but small synchronous write
>             performance is terrible.  We're hoping that the use of a
>             PCI-e SLOG device in the new servers helps with small
>             synchronous write performance.
>
>             Anything you'd be willing to share would be greatly
>             appreciated.
>
>             Cheers!
>
>             -Branden
>
>
>             On Friday, November 29, 2013 7:35:44 AM UTC-6, Arman
>             Khalatyan wrote:
>
>                 We had a very bad performance degradation on glusterfs
>                 dur to the number of files and dirs in the whole
>                 filesystem.
>                 Replacing xfs to zfs back-end was not helping to
>                 improve performance.
>                 Because of missing stable RDMA connection on gluster
>                 we use tcp ip overib.
>                 On single streaming  io was not going over 150MB/s and
>                 it is using DDR infiniband network. with lustre we get
>                 almost full bandwidth
>                 Finally we move to zfs+lustre, with  files>10M it is
>                 rocking!
>                 But we are not require replication, maybe for your
>                 needs the  latest gluster will fit to your needs?
>
>                 A positive experience was that Virtual machines on
>                 striped glusterfs  are pretty stable.
>
>                 good luck with gluster:)
>                 Arman.
>
>
>                 On Fri, Nov 29, 2013 at 2:10 PM, Pongrácz István
>                 
wrote:
>
>                     Hi,
>
>                     I am wondering, what is your experience or
>                     recommendation for using glusterfs on top of zfs.
>
>                     Tha basic plan is replication:
>
>                       * hold kvm raw image files (some ntfs, one zfs)
>                       * share these files between compute nodes
>
>                     As I can see, I need three nodes to use with
>                     quorum to avoid split brain situation.
>
>                     I am interesting about real-world experiences with
>                     glusterfs in general.
>
>                     I assume, holding some big files not a big deal
>                     for glusterfs, but holding for example several
>                     openvz containers could cause perfomance drop due
>                     to the lot of files.
>
>                     The alternative is to use bzman (thanks to bassu,
>                     maybe I can replace my solution with bzman later)
>                     to get near realtime replication, where quorum is
>                     not required.
>
>                     Any hints are welcome, thank you!
>
>                     István
>
>                     To unsubscribe from this group and stop receiving
>                     emails from it, send an email to
>                    
zfs-discuss...-VKpPRiiRko7s4Z89Ie/[email protected]
>
>
>             To unsubscribe from this group and stop receiving emails
>             from it, send an email to
>            
zfs-discuss+unsubscribe-VKpPRiiRko4/[email protected]
>            
.
>
>
>         To unsubscribe from this group and stop receiving emails from
>         it, send an email to
zfs-discuss+unsubscribe-VKpPRiiRko4/[email protected]
>        
.
>
>
>
>
>     -- 
>     Kate Ward >
>     To unsubscribe from this group and stop receiving emails from it,
>     send an email to
zfs-discuss+unsubscribe-VKpPRiiRko4/[email protected]
>    
.
>
>
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to
zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/[email protected]

To unsubscribe from this group and stop receiving emails from it, send an
email to zfs-discuss+unsubscribe-VKpPRiiRko7s4Z89Ie/[email protected]
 
CD: 8ms