Thursday, July 18, 2013

Sparse files on Windows

Once again I am drawn away from Linux to solve a Windows problem. The source of the problem is Hyper-V which (as always) has a cryptic error message about 'cannot open attachment' and 'incorrect file version'.

The source of the error was tracked down to the file being flagged as Sparse.

What is a sparse file?
Under UNIX/Linux, a sparse file is a file where not all of the storage for the file has been allocated. Handling of sparse files is normally transparent but some tools like file copy and backup programs can handle sparse files more efficiently if they know where the sparse bits are. Getting that information can be tricky.

In contrast, under windows, a sparse file is a file which has the sparse flag set. Presumably the sparse flag is set because not all of the storage for the file has been allocated (much like under Linux). Interestingly, even if all the storage is allocated, the sparse flag may still be set. (It seems the flag indicates the potential to be sparse rather than actually being sparse. There is an API to find out the actual sparse parts).

The the problem started when I happened to download a Hyper-V virtual machine using BitTorrent. When the files are being created, not all of the content exists so it is indeed sparse. Once all the content has been supplied, the file is (to my mind anyway) no longer sparse. However, under windows it seems, once a sparse file, always a sparse file.

Microsoft provide a tool to check and set the sparse flag:
fsutil sparse queryflag <filename>
fsutil sparse setflag <filename>
Note 1: Have they not heard of get and set
Note 2: You can't use a wildcard for <filename>
The amazing thing to note here is that there is no clearflag option. This might lead you to believe that you can not do that. In fact you can. For users in a pickle, there is a program called Far Manager which can (among other things) clear the flag. Far Manager is open source and a quick peek at the code shows that it uses a standard IOCTL to do this named FSCTL_SET_SPARSE.

So with that knowledge, it is actually quite easy to make a file not be sparse any more. In fact, I wrote a program called unsparse.

Not only does the tool have the ability to clear the sparse flag, it can recursively process a directory and unsparse all the sparse files found, making it perfect to fix up a Hyper-V download.

Look for the program soon on my chrysocome website

1 comment:

  1. In engineering, a distributed file may be a variety of data file that makes an attempt to use system house additional with efficiency once the file itself is partly empty. This is often achieved by writing transient info (metadata) in place of the empty blocks to disk rather than the "empty" house that makes up the block, less space. The total block size is written to disk because the actual size only if the block contains "real" (non-empty) knowledge. Once reading files, the system plainly converts in turn in place of empty blocks into "real" blocks crammed with null bytes at run time. The use is of this change. The advantage of files is that storage is merely once truly needed: space is saved, and huge files may be created aside there's free house on the system. This as well reduces the time of the main write because the system ought not to allot blocks for the "skipped" house. If the sparse file with a program that doesn't clearly support them could copy the whole, size of the file, as well as the zero sections that don't seem to be chosen on disk losing the advantages of the goods within the file. Files are not hold by all backup software package. Sparse files area unit files that have giant amounts of area near them, while not the whole size from the file system. They’re helpful for sinking the number of your time and disc space concerned in making loop file systems or giant disk pictures for guests, among things. The term sparse file is employed to mean one contain holes; it's simple to grant one on a running system as a result of its disk usage is a smaller amount than its size. Will-owed are able to see this actions with Now the higher than method can the files within the system, it becomes to search out all the files in a very file system or list, mainly after they area unit some. Don’t worry, there's link in Nursing choice in realize command that helps of to search out all the files in one go. Let’s see link in nursing case below.thank you so much...

    Cheap Essay Writing Service