Thursday, July 18, 2013

Sparse files on Windows

Once again I am drawn away from Linux to solve a Windows problem. The source of the problem is Hyper-V which (as always) has a cryptic error message about 'cannot open attachment' and 'incorrect file version'.

The source of the error was tracked down to the file being flagged as Sparse.

What is a sparse file?
Under UNIX/Linux, a sparse file is a file where not all of the storage for the file has been allocated. Handling of sparse files is normally transparent but some tools like file copy and backup programs can handle sparse files more efficiently if they know where the sparse bits are. Getting that information can be tricky.

In contrast, under windows, a sparse file is a file which has the sparse flag set. Presumably the sparse flag is set because not all of the storage for the file has been allocated (much like under Linux). Interestingly, even if all the storage is allocated, the sparse flag may still be set. (It seems the flag indicates the potential to be sparse rather than actually being sparse. There is an API to find out the actual sparse parts).

The the problem started when I happened to download a Hyper-V virtual machine using BitTorrent. When the files are being created, not all of the content exists so it is indeed sparse. Once all the content has been supplied, the file is (to my mind anyway) no longer sparse. However, under windows it seems, once a sparse file, always a sparse file.

Microsoft provide a tool to check and set the sparse flag:
fsutil sparse queryflag <filename>
fsutil sparse setflag <filename>
Note 1: Have they not heard of get and set
Note 2: You can't use a wildcard for <filename>
The amazing thing to note here is that there is no clearflag option. This might lead you to believe that you can not do that. In fact you can. For users in a pickle, there is a program called Far Manager which can (among other things) clear the flag. Far Manager is open source and a quick peek at the code shows that it uses a standard IOCTL to do this named FSCTL_SET_SPARSE.

So with that knowledge, it is actually quite easy to make a file not be sparse any more. In fact, I wrote a program called unsparse.

Not only does the tool have the ability to clear the sparse flag, it can recursively process a directory and unsparse all the sparse files found, making it perfect to fix up a Hyper-V download.

Look for the program soon on my chrysocome website


  1. In engineering, a distributed file may be a variety of data file that makes an attempt to use system house additional with efficiency once the file itself is partly empty. This is often achieved by writing transient info (metadata) in place of the empty blocks to disk rather than the "empty" house that makes up the block, less space. The total block size is written to disk because the actual size only if the block contains "real" (non-empty) knowledge. Once reading files, the system plainly converts in turn in place of empty blocks into "real" blocks crammed with null bytes at run time. The use is of this change. The advantage of files is that storage is merely once truly needed: space is saved, and huge files may be created aside there's free house on the system. This as well reduces the time of the main write because the system ought not to allot blocks for the "skipped" house. If the sparse file with a program that doesn't clearly support them could copy the whole, size of the file, as well as the zero sections that don't seem to be chosen on disk losing the advantages of the goods within the file. Files are not hold by all backup software package. Sparse files area unit files that have giant amounts of area near them, while not the whole size from the file system. They’re helpful for sinking the number of your time and disc space concerned in making loop file systems or giant disk pictures for guests, among things. The term sparse file is employed to mean one contain holes; it's simple to grant one on a running system as a result of its disk usage is a smaller amount than its size. Will-owed are able to see this actions with Now the higher than method can the files within the system, it becomes to search out all the files in a very file system or list, mainly after they area unit some. Don’t worry, there's link in Nursing choice in realize command that helps of to search out all the files in one go. Let’s see link in nursing case below.thank you so much...

    Cheap Essay Writing Service

  2. I am also investigating this issue. Very great article. Thank you!
    torture game 2

  3. Very Informative Post, I would love to read more, keep writing. I want to share some links, useful for students who find it difficult to complete their academic assignment. If you are the one, please visit coursework writing service

  4. Thanks for the helpful information you shared with us.
    strike force kitty 2

  5. This was among the best posts from your team it let me learn many new things. mutilate a doll 2

  6. TheTexting’s SMS API allows you to exchange Short Message Service (SMS) communications to 180+ countries worldwide. Using TheTexting’s powerful SMS API, users send millions of messages around the globe every day.

  7. The holiday shopping season is upon us with Thanksgiving, Black Friday and Christmas just around the corner. Businesses need to constantly let people know of their deals and offers to compete in the market and Bulk SMS is the only channel they can use to reach customers in real time.

  8. Thanks a lot for the post. It has helped me get some nice ideas. I hope I will see some really good result soon. If you cannot understand how to write a memorandum, be sure that you can order memorandum notes on from our professional writers.

  9. This is really great work.Thank you for sharing such a useful information here in the blog.

  10. Support for sparse files is introduced in the NTFS file system as another way to make disk space usage more efficient. When sparse file functionality is enabled, the system does not allocate hard disk drive space to a file except in regions where it contains nonzero data.

    link here

  11. Online biology essay services have come up with Biology Essay Writing Services for biology research paper writing service students in order for them to score straight A’s in their custom biology research paper services.