Apache Spark

Raised as a usability problem by 11 people on Hacker News over 13 years (2013–2025), in 11 threads.

People raised many different problems with Apache Spark, but no single problem has yet been raised by 3 people in 3 different months and threads.

Unreviewed issues were grouped by machine: Claude Haiku labelled each comment as a usability complaint, named what it's about and summed up the problem in a line, and a small model grouped similar problems. Nobody has checked them by hand. Expect mistakes.

The newest 10 complaints

PySpark code is annoying to write, and it runs much slower than alternatives2025-11-14
Default settings fail on large single machines, causing OOM errors and slow saves needing manual config tuning2021-11-18
Demo video doesn't show what the product is or how to use it2020-08-05
Nested RDD mappings are a nightmare; developers must use non-obvious tricks2019-08-17
Jobs sometimes pause for minutes when they should finish in under a minute2018-11-13
Large shuffles fail or need many bug fixes and workarounds, very painful for users2016-09-01
Most setup tutorials require many console commands that are obsolete2016-03-30
Website is hard to read2015-06-12
Logs not copied across cluster, so failed tasks needed manual SSH and vague error messages2015-03-24
Docs are on one giant list and scattered across several sites, hard to navigate2013-09-12