Games, Hard Drive and Allocation Units

[Drezha]

So my Steam Library was getting out of hand and I decided that I needed to get a new hard drive – my 6TB drive was left with 900GB of space and I’ve recently purchased Halo Infinite, which is another 55GB, so I’d soon run out of head room.

After ordering a 14TB from Amazon, to give me plenty of headroom, I set about copying the data from one drive to the other using trusty Teracopy (I’d use Robocopy, but Teracopy at least gives an indication of when it’ll finish). This is where I came upon issues – the new drive was using far less space for the same data. At least, Teracopy was telling me that all the files had been copied across fine.

I ended up digging down and this is where I found the issue – the old hard drive was formatted in exFAT in the past (I assume so I could access from the Mac if needed) and the new drive was NTFS formatted. Wiztree (rather than WinDirStat, as that kept crashing) was able to scan both drives and show me my issue (and if you’ve read the title, you know). It appears that the allocation units on the exFAT drive were set to 2MB!

This has meant a huge waste in space – 37% of the space on the drive is wasted as it’s allocated to files that aren’t using it! The image below shows that the Steam folder is using 4.1TB but is only 2.6TB in size. The worst offender was Star Wars Jedi Fallen Order, where the game size was about 30GB, but was taking up 138GB due to a large number of small files!

Means a reformat and redownload of all the games would have solved my issues if I’d investigated prior. I’m feeling a bit of a chump now to be honest.

"Everything looks interesting until you do it. Then you find it’s just another job" - Terry Pratchett

[Ed P]

Interesting point. It is a pity that there isn’t a Windows utility whereby *.cfg or *.txt  *.gif etc files could be automatically put into and accessed from a single large archive file. Then nice large allocation units could give the optimum read/write speed without wasting half the capacity of the drive.

[keith with the teef]

For sure thats  pretty cool error. Thanks for sharing :), needles to say I jumped to see if my drives are all ntfs.

If I get board, I may check some of my old drives, I have not disposed of, just for curiosity.

BTW the new halo game can suffer from stuttering. Word is more ram.

Cheer your self up go see the new Matrix at the Imax tomorrow.

 

 

[Drezha]

keith with the teef wrote:

For sure thats pretty cool error. Thanks for sharing :), needles to say I jumped to see if my drives are all ntfs.

NTFS won’t necessarily save you, if the same error has been made when formatting! You can select a different allocation unit size on format – I believe the default on NTFS is 4096 B, whereas the smallest possible on exFAT on Windows is 512 KB by looks. Can’t recall if the drive came formatted or whether I formatted it myself but certainly a 2048 KB size seems like overkill! Perhaps fine if I was using it for storing large files like movies or virtual machines.

I’ll see how the new drive performs soon. In fairness, I try and avoid running the more modern games off the hard drive and instead copy them to an SSD and run from there – which is why Steam is superior to other game stores in that regard as it’s dead simple to move games (where as Epic is a right pain in the ass and doesn’t always work). The hard drive is more for storage, as my internet isn’t the fastest and if I want to play games, I often want to play them there and then!

"Everything looks interesting until you do it. Then you find it’s just another job" - Terry Pratchett

[Dave Rice]

The exFat default is 128kb, which is bad enough.

[Drezha]

Unsure if it’s related, but the new hard drive appears to perform better when opening and downloading Steam updates.

To be honest, the additional space is nice, as I won’t have to worry about updating it now for a considerable period (I would hope). However, when games like COD Warzone are 180GB, it’s no wonder large hard drives are needed!

"Everything looks interesting until you do it. Then you find it’s just another job" - Terry Pratchett

[Author]

Posts