BSBRES801 Hadoop Distributed File System Assignment Sample

• Subject Code : BSBRES801
• University : Victoria University My Assignment Services is not sponsored or endorsed by this college or university.
• Subject Name : IT Computer Science

Initiate and Lead Applied Research - Section A

1. Hadoop Distributed File System can be effectively used in order to manage the namespace of the file system along with regulating the access of the client for the files. Yes, I will recommend to use the system on an online video sharing platform because of the presence of high number of DataNodes that help in implementing efficient implementation of parallel processing. Therefore, it can be interpreted that the system is highly fault intolerant.

2. RDD refers to the collection of data elements in the form of clusters across many machines. In this data collection, mostly the elements of Scala objects are collected. Whereas, the Data Frame relates to the category of data collection in which the data is collected and then organized into named columns. It can be referred to as similar to like a table in the relational database. The other kind of data collection is Paired RDD wherein the data collection process is done with the key value pair. The data collection in this category has all the new features like RDD but also has new features due to the presence of Key value pair.

3. The two types of operation include transformations wherein which a new data set is created from the existing one and it results into an action in which the value to the driver program is achieved. In the transformation process the old data set is used for computation and the new value from the data asset is achieved and this whole process is termed as transformation. Therefore, it can be interpreted that transformation process is only applicable when the user wants the results on the driver program.

One type of transformation is map under which the data set passes through the function and returns could be achieved in the forms of new RDD results. There is a type of action referred to reduce under which all the data elements are aggregated and the final result of the function is achieved on the driver program. One transformation in Spark is called Lazy, in this transformation the transformation function is not applied to the data elements directly but it is only applied when the action is required to return on the driver program.

4. Parallelization formula:

a. S (n ) is he theoretical speed which is equivalent to 10.

P is 80%

The value of I is not given and therefore can be assumed to 1

1((1−0.8)+0.8)) = 1

b.Maximum Speed up

With the application of the unlimited resources and 80% of the program being parallel,

Maximum Speed up: 11−(0.8)) = 5 times

5.

a. Reduce by key feature of the Spark can be utilized in order to reduce the transmission cost.

b. The job detail view elaborates upon all the stages of the job done.

Initiate and Lead Applied Research - Section B

b. In line 10, filter is applied in order to examine the each input or output request in the system.

c. map is a higher order function that is used in the statement in order to apply the function on each data element so that the result could be obtained in the similar manner.

d. reduceby key function is used in the statement in order to transform the data and it also means that the data in the process is easily evaluated.

2.

a. Number of columns: len (df.index) this formula can be used .

b. df = pd. DataFrame({"Letters": ["a", "b", "c"], "Numbers": [1, 2, 3]})

print(df)

index = df. index.

number_of_rows = len(index) find length of index.

print(number_of_rows)

c. words = words. reduceByKey (lambda, MaxEnrolment, <50)

Remember, at the center of any academic work, lies clarity and evidence. Should you need further assistance, do look up to our Computer Science Assignment Help