Apache Spark

@ApacheSpark

Lightning-fast unified analytics engine

spark.apache.org

Joined June 2013

1 Following

44.8K Followers

Replies Reposts Media

Apache Spark
@ApacheSpark
Oct 28
We are happy to announce the availability of Apache Spark 4.1.4! 🎉 Spark 4.1.4 is the fourth maintenance release containing security and correctness fixes. This release is based on the branch-4.1 maintenance branch of Spark. We strongly recommend all 4.1 users to upgrade to
1
Apache Spark
@ApacheSpark
Oct 27
Agents are writing a ton of Spark code these days. Most of it is error-ridden, and the errors are the boring kind: a column that doesn’t exist, a table nothing upstream declares, a session config invented from thin air. At OSA Con 2026, @lisancao (@databricks) will cover
Lisa N. Cao
1
Apache Spark
@ApacheSpark
Oct 26
The core issue is that their source system is both relational and a document database. All of the documents are stored as XML. Figuring out how to handle that in production was a headache, because it is very challenging for end users to parse XML or JSON. That is where Variant
2
Apache Spark
@ApacheSpark
Oct 22
Agents don’t ask one question a day. They ask 10 or 100, against the same data stack and the same SLA. At Open Lakehouse + AI in Bellevue, Felix Cheung (@nvidia) walks through GPU-accelerated Apache Spark with no code change, plus @DeltaLakeOSS, @ApacheIceberg, and Spark
Apache Spark
@ApacheSpark
Oct 21
Apache Spark can analyze your query plans and derive the dependencies for you. You don’t need to declare them anymore. @lisancao sits down with Andreas Neumann on Spark Declarative Pipelines. Spark owns the graph, manages session lifecycle and writes, while @ApacheAirflow