Monday, May 4, 2015

Hadoop free space and file sizes

It is useful to understand what would be the size of data and free space if you want to write something to HDFS. Default block size in HDFS is 64MB, so one file will take at least 64MB. Also, default replication ratio is 3x. The size will be:
3 * Sum(i)(size[i] / 64 + 1)
Check the block size and replication ratio:
$HADOOP_HOME/bin/hadoop fsck / 
Check the free space (plain free space, not taking into account replication or block size):
$HADOOP_HOME/bin/hadoop dfsadmin -report
How big is the folder (it is actually replication ratio times bigger):
$HADOOP_HOME/bin/hadoop dfs -dus [/some/folder]

16 comments:

  1. There are lots of information about latest technology and how to get trained in them, like Big Data Hadoop Training in Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Course in Chennai). By the way you are running a great blog. Thanks for sharing this.

    Best Hadoop Training in Chennai
    | Best hadoop training institute in chennai

    ReplyDelete
    Replies
    1. Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. big data projects for students But it’s not the amount of data that’s important.Project Center in Chennai

      Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Corporate TRaining Spring Framework the authors explore the idea of using Java in Big Data platforms.

      Spring Training in Chennai

      The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. Cloud is one of the tremendous technology that any company in this world would rely on(Salesforce Training institutes in Chennai). Using this technology many tough tasks can be accomplished easily in no time. Your content are also explaining the same(Salesforce developer training in chennai). Thanks for sharing this in here. You are running a great blog, keep up this good work.

    ReplyDelete
  3. Hi Admin, I went through your article and it’s totally awesome. You can consider including RSS feed for easy content sharing, So that you can drive huge traffic to your blog. Hadoop Training in Chennai | Big Data Training in Chennai

    ReplyDelete
  4. This comment has been removed by the author.

    ReplyDelete
  5. This is my first visit to your site.It is a stunning post. Exceptionally valuable to me. I preferred it.Keep update your blog.
    Regards, Hadoop Training Chennai | Hadoop Training in Chennai

    ReplyDelete
  6. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Hadoop Training in Chennai

    ReplyDelete
  7. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  8. it’s really nice and meanful. it’s really cool blog. Linking is very useful thing.you have really helped lots of people who visit blog and provide them usefull information.


    Hadoop Training in Hyderabad

    ReplyDelete