Monday, October 21, 2013

Large filesystem on CentOS-6

I had the pleasure of testing out a new server which came with 10 x 4Tb drives. Configured on and HP Smart Array P420 in RAID 50 that gave me a 32Tb drive to use.

I installed CentOS-6 but the installer would not allow me to use the full amount of space. It turns out the maximum ext4 filesystem on CentOS-6 is 16Tb. This is the limit of 32bit block addressing and 4k blocks. (232 * 4k = 232 * 212= 244=240 * 24 = 16T)

That did not make for a very exciting test. Only half the disk available was available in a single filesystem. Sure I could create two logical volumes and then create two filesystems but that seemed a bit like a DOS solution.

Some research turned up the options of using ext4 48bit block addressing. This enables a filesysem up to 1Eb in size and allows for future 64bit addressing for an even larger limit.

The catch was of course that 48bit addressing is not supported by the tools which come with CentOS-6.

Fedora 20 (rawhide) does come with the required updates to e2fsprogs (e2fsprogs-1.42.8-3) which enable 48bit addressing (somehow future-proofed by being called 64bit addressing).  I then set out to rebuild the fedora 20 rpm for CentOS-6. Amazingly the compile was clean though I did have to disable some tests which is a tad alarming. All in all it did not take long to produce a set of replacement rpms for e2fsprogs.

After installing the new tools I was able to create a new filesystem larger than 16Tb. I started at 17Tb to give me room to play. It seems that the CentOS-6 kernel does already have support for 48bit addressing. I ran a number of workloads and I could not find any problems. Still, I don't know what bugs may be lurking in there.

dumpe2fs shows the new filesystem feature '64bit'.

I also attempted to do an offline resize of the filesystem to the maximum size of my disk. This just worked as expected. Online resize is not available until a much later kernel version. I did not attempt this because there are know bugs.

The last limit I wanted to test was the file size limit. Even on my new filesystem the individual file size limit is 16Tb. It is not every day that I get to make one of them so I did
 dd if=/dev/zero of=big bs=1M count=$(expr 16 \* 1024 \* 1024)

At ~500Mb/s it still took 9 hours to complete.
And it turns out that the maximum file size is actually 17592186040320 and not 17592186044416. That is 4k (one block) short of 16Tb. (You can actually check that on any machine with the command truncate big --size 17592186040321 )

That also raised an issue which was a problem on ext3. How long does it take to delete a large file? Well this is the largest possible file and it took 13 minutes.

In conclusion, the 16Tb filesystem limit is easliy raised on CentOS-6 but it comes at the expense of using untested tools and kernel features. Although I did not find any problems in my testing, this could pose a substantial risk if you have over 16Tb of data which you do not want to loose.

If anyone is interested in my rpms you can get them from


  1. I totally agree with you.
    When it comes to working with a huge file system any working solution is preferable. In my case, it was a set of virtual data rooms found with help of the virtual data rooms reviews.

  2. That is really nice post and really agree with you. Now a days data file are increase day by day M&A data room provide you the best solutions for your business and m&a deals.

  3. I was working and suddenly I visits your site frequently and recommended it to me to read also. The writing style is superior and the content is relevant. Thanks for the insight you provide the readers!

  4. Pay attention to the time of day - between dawn and dusk the peak time for mosquitoes. Reducing external activity during those times to prevent mosquito bites.
    شركة مكافحة حشرات
    شركة مكافحة النمل الابيض بالخبر
    شركة مكافحة حشرات بالخبر
    شركة رش مبيدات بالخبر

  5. I have setup the RAID card to create one RAID 6 volume with 1 hot ... How do I partition the disk so I can create a big XFS file system on it?


  6. Norton is the product that is known for its effective and powerful attack on viruses and malware. It’s one of the top-rated and preferred antivirus programs used worldwide in various organizations, offices, homes, and corporate sectors. It offers several easily accessible and budget-friendly subscription plans. Norton Antivirus software is a part of the Norton family of computer security products that comes with attractive features and virus detection tools. The detailed information can be found on norton install with product key. Download, install and activate the product by taping the web-link of Norton that is norton install with product key.
    norton install with product key | download webroot with key code | download webroot with key code | download webroot with key code

  7. Quickbooks Install Diagnostic Tool is a usefull software that resolves any error that occurs in the installation process of QB desktop on your computer, it scans the system, recognizes error and resolves it.

  8. A Norton client account is needed for any action you do on the Norton site. From downloading the item to updating plans, a client account is the primary resource among clients and Norton. Making a client account is simple, and anybody with a functioning email address can make one out of a couple of moments. To make a Norton account, follow the means given beneath. if you have any query related to norton antivirus then you can contact now.

  9. Amazing and infromative post, checkout this as well:-
    online physiology tutor

  10. People in Australia speak English. Linguistic diversity in Australia is diverse on the fragmentation scale, which for Australia is 0.3349. Followers of Christianity make up the religious majority in the country. 89.2% of Australia's population lives in cities. This percentage is the urban population of Australia. The urbanization rate in Australia is 1.2. According to the data on tourists entering Australia, 6,382,000 tourists arrive in the country annually.

  11. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work. for More Information Click Here :- Verizon Email Login Issue

  12. Nice post. I learn something more challenging on different blogs everyday. It will always be stimulating to read content from other writers and practice a little something from their store. I’d prefer to use some with the content on my blog whether you don’t mind. Natually I’ll give you a link on your web blog. Thanks for sharing.
    Click Here
    Visit Web

  13. شركة تنظيف بجازان
    شركة كشف تسربات المياه بجازان
    تسرب المياه تعتبر مشكلة تسرّب المياه من المشاكل التي تؤرق الأفراد بشكل كبير؛ وذلك لما تُلحقه من ضرر بالمباني والمنازل، لذلك لا بد من السعي الدؤوب للكشف عما إذا كان هناك تسريب مائي أم لا. ويُعرف تسريب المياه بأنّه خلل يطرأ على توصيلات المياه بمختلف أشكالها، سواء كان ذلك في مواسير الصرف أو أكواعها أو مختلف أجزاء مواسير المياه، ومنها: الحنفيات، والخزانات، والسخانات، فيتسبب ذلك الخلل بخروج المياه من المسار المحدد لها إلى محيط المواسير فتتجمع المياه بالقرب من الأماكن المحيطة بالتسريب كالأرضيات والجدران فتلحق الضرر بها نتيجة تفاعلها مع مواد البناء، وتآكل الحديد وتهالك الخرسانة، وبالتالي تفاقم المشاكل في حال التغاضي عن علاجها.

  14. QuickBooks is often regarded as error-free software, but there are a few glitches that can cause problems for users. quickBooks database server manager network diagnostics error can appear on the computer screen when upgrading and running QuickBooks in multi-user mode. When accessing the company register, users can experience errors such as Error H202 and other network connection errors. If you’ve made a similar error, don’t worry; we’ve got you covered with this post. The user can find it difficult to deal with QBDBSM network diagnostics errors.

  15. Stunning site! Do you have any accommodating clues for trying essayists? I’m wanting to begin my own site soon yet I’m somewhat lost on everything. scandal Sienna Miller brown Poncho Would you prompt beginning with a free stage like or go for a paid alternative? There are such a large number of alternatives out there that I’m totally overpowered .. Any thoughts? Welcome it!

  16. This comment has been removed by the author.

  17. Creating a large filesystem on CentOS-6 can be accomplished by using the "dd" command. The "dd" command can create a filesystem up to 2TB in size.

    Anyway, I want to recommend you see this service:

  18. This comment has been removed by the author.

  19. Amazingly the compile was clean though I did have to disable some tests which is a tad alarming for https://www.friscoconcretepros I'll try it again later. Thank you!

  20. There are several Tattoo Shops Tyler TX that offer a wide range of tattoo services. These shops are known for their skilled artists who can create unique and personalized designs. Whether you are looking for a small and delicate tattoo or a large and intricate piece, these tattoo shops in Tyler, TX can cater to your needs. They prioritize cleanliness and safety, ensuring that all equipment is sterilized and that proper hygiene protocols are followed.

    With their professional and friendly staff, you can expect a comfortable and enjoyable tattooing experience. So, if you are in Tyler, TX and looking to get a tattoo, these tattoo shops are worth considering.