Tuesday 27 October 2020

Get Azure Data Lake Gen2 Container, Folder Size


Problem


How to get Azure data lake gen 2 container or folder size 

Solution


PowerShell AZ Module has to be installed to run below commands

Run below code to connect to your Azure account

Connect-AzAccount

To get the container total size, specify the container (file system) name in the variable at first line and run the code

$myfilesystem = "adventureworks"

$ctx = New-AzStorageContext -StorageAccountName "vcadlsgen2" -UseConnectedAccount

$Files = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $myfilesystem -Recurse `

            | Where-Object IsDirectory -eq $false

$Total = $Files | Measure-Object -Property Length -Sum

$Total | Select-Object @{Name = "SizeInBytes"; Expression={$_.Sum}},

                       @{Name = "SizeInKB"; Expression={$_.Sum/1KB}}  


Output - 



To get the folder size, specify the container (file system) name as well as folder path in the variable

$myfilesystem = "adventureworks"

$mypath = "Address"

$ctx = New-AzStorageContext -StorageAccountName "vcadlsgen2" -UseConnectedAccount

$Files = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $myfilesystem -Path $mypath -Recurse | Where-Object IsDirectory -eq $false

$Total = $Files | Measure-Object -Property Length -Sum

$Total | Select-Object @{Name = "SizeInBytes"; Expression={$_.Sum}},

                       @{Name = "SizeInKB"; Expression={$_.Sum/1KB}} 


Output -






5 comments:

  1. You can easily run anAzure Data Lake Gen2 instance in a Linux virtual machine on your computer by using Virtual Box, Docker, or any other container technology. To run an instance of the Data Lake Gen2 in a container, you need to install Azure CLI 2.0 and then create a JSON file that describes parameters for the container and the Data Lake Gen2 configuration.

    ReplyDelete
  2. Hi Vaibhav,
    Approximately how long would it take to get a response from this script for a 0.5 PB folder?

    ReplyDelete
  3. Big Data is a term used to describe the large amounts of data that are created by the internet and by companies. Thebig data technologies are used to handle big data and increase its value by making sense of the data and extracting meaningful insights that can be used by companies.

    ReplyDelete
  4. Thanks for a very interesting blog. What else may I get that kind of info written in such a perfect approach? I’ve a undertaking that I am simply now operating on, and I have been at the look out for such info. abrollcontainer in Germany

    ReplyDelete