What is “Slowly Changing Dimension”? The “Slowly Changing Dimension” is “One” of the “Very Important Concept” in “Data Warehousing Solution”. “Slowly Changing Dimension” is the “Change” of “Attribute”, or, the “Value” of the “Entities” over a “Period” of “Time”. Example: “Consider” that there is a “Delta Table”, called “Person”, which has an “Attribute”, called as…

What is “Join Selection Strategy”? When “Any Type” of “Join”, like the “Left Join”, or, the “Inner Join” is “Performed” between “Two DataFrames”, “Apache Spark” “Internally” decides which “Algorithm” will be used to “Perform” the “Join” Operations between the “Two DataFrames”. That particular “Algorithm” that is “Responsible” for “Planning” the “Join” Operation between the “Two…

Why “Broadcast Variable” is Used? The “Broadcast Variable” is used mainly for “Performance Tuning”. What is a “Broadcast Variable”? The “Broadcast Variable” is a “Programming Mechanism” in “Apache Spark” through which it is possible to “Keep” the “Read-Only Copy” of the “Data” of a “Table”, or, a “DataFrame” in “Each Node” of the “Cluster”, instead of “Sending” the “Data” of…

What is the “Importance” of “Partition”? “Apache Spark” is known for its “Speed”. The “Fast Speed” of “Computing” comes from the “Parallel Processing”. “Partition” is the “Key” for “Parallel Processing”. If the “Data”, to work with, is “Partitioned” in a “Proper Way” then the “Query Performance” on that “Data” would be “Improved” as the “Parallel Processing”…

How “Delta Tables” Become “Slow” to “Query”? “Delta Tables” can “Grow” “Very Large Over Time”, leading to “Slow Query Performance” and “Increased Storage Costs”. How to “Speed Up” the “Query Performance” of the “Delta Tables”? Keeping the “Delta Tables” “Fast” and “Clean” is important for “Maintaining” the “Efficiency” of “Data Pipelines”. There are “Several Operations” and “Trade-Offs” that can “Positively Influence” the “Speed” of the “Delta Tables”.

“Disadvantages” of “Storing Data” in “Data Lake” In a “Data Lake”, like — “Azure Data Lake Storage Gen 2” in “Azure”, where the “Underlying Data” of a “Table” is “Kept” in the “Form” of “Multiple Files” of “Any Format”, it becomes “Very Difficult” to “Perform” the following “Operations” on those “Files” - Merge Operation “Updating” a “Record”…

What is “Unity Catalog”? “Unity Catalog” is a “Fine-Grained Data Governance Solution” for “Data” present in a “Data Lake”. Why “Unity Catalog” is Used “Primarily”? Suppose, in a “Lake House Project”, there is a “Database” in the “Hive Metastore” of a “Databricks Workspace”, and, in that “Database”, there are “Twenty Delta Tables” present. If the “Requirement” is to “Provide” a…

What is a “Tuple”? “Python Tuples” are the “Immutable Sequences of Arbitrary Objects”. Once “Created”, the “Elements” within the “Python Tuples” “Cannot” be “Replaced” or “Removed”, and, “New Elements” “Cannot” be “Added” to the “Python Tuples”. How “Python Tuples” are “Created”? “Python Tuples” are “Delimited” by “Parentheses”, i.e., “()”. The “Items” within the “Tuples” are “Separated” by “Commas”, i.e…

“Usage” of “Multiplication Operator” on “List” As with “Strings” and “Tuples”, the “Lists” support “Repetition” using the “Multiplication Operator”. The “Multiplication Operator” is often used for “Initializing” a “List” of a “Size, Known in Advance” with a “Constant Value” for the “Repetition”.