Skip to content

[AURON #1924] [BUILD] Add -Xlint:_ to scala-maven-plugin and resolve related warnings#2216

Open
1fanwang wants to merge 3 commits intoapache:masterfrom
1fanwang:build/scala-xlint
Open

[AURON #1924] [BUILD] Add -Xlint:_ to scala-maven-plugin and resolve related warnings#2216
1fanwang wants to merge 3 commits intoapache:masterfrom
1fanwang:build/scala-xlint

Conversation

@1fanwang
Copy link
Copy Markdown

Which issue does this PR close?

Closes #1924. Sets up the next sibling under the [BUILD] epic #1869 (#1925) without changing its scope.

Rationale for this change

-Xlint:_ enables Scala's bundled lint checks, several of which catch real bug classes that the existing -Ywarn-unused + -Xfatal-warnings do not. This turns the flag on in both the base scala-maven-plugin config and the scala-2.13 profile, then resolves every warning it surfaces so the matrix build stays green.

What changes are included in this PR?

pom.xml -- add -Xlint:_ and -Wconf:cat=lint-inaccessible:s to both scala-maven-plugin arg lists. The lint-inaccessible silencer is commented inline: Auron implements Spark plugin SPIs (AppHistoryServerPlugin, the Shims trait, AuronRssShuffleWriterBase) whose contracts take private[spark] types (ElementTrackingStore, SparkUI, IndexShuffleBlockResolver, ShuffleWriteMetricsReporter). Exposing them in our overrides is required by Spark, not a defect, so silencing the category project-wide is cleaner than annotating every override site.

Source fixes for the warnings that are real bug classes (8 sites across 8 files):

  • lint:adapted-args -- :+ ("k", v) on a Seq[(String, V)] was being silently auto-tupled. Made the Tuple2 explicit via ->. Files: ConvertToNativeBase, NativeBroadcastExchangeBase, NativeParquetInsertIntoHiveTableBase.

  • lint:nonlocal-return -- return from inside a for { ... } / foreach { ... } body desugars to throw/catch via NonLocalReturnControl. Replaced with Iterable.exists in sparkver.matchVersion, and an early-exit while-loop over an iterator in SparkOnHeapSpillManager.spill.

  • pattern shadowing (lint:type-parameter-shadow / pattern-shadow) -- pattern bindings reusing names from the enclosing scope. Renamed: case in: ... => -> case limited: ... => / case fileIn: ... => in AuronBlockStoreShuffleReaderBase; case expr => expr -> case other => other in SparkUDAFWrapperContext; case nativeXScanExec -> case scan in AuronEmptyNativeRddSuite.

Are there any user-facing changes?

No -- build-only.

How was this patch tested?

Local ./auron-build.sh --pre --sparkver <v> --scalaver <s> -DskipBuildNative -Dspotless.skip=true was green for:

  • Spark 3.0 / Scala 2.12
  • Spark 3.1 / Scala 2.12
  • Spark 3.2 / Scala 2.12
  • Spark 3.4 / Scala 2.12
  • Spark 3.5 / Scala 2.12
  • Spark 3.5 / Scala 2.13

Spark 3.3 / Scala 2.12 left for the matrix CI to cover (middle of the range, unlikely to surface different lint categories). Happy to iterate on any warning CI catches that I missed locally.

…esolve related warnings

Enable `-Xlint:_` in both the base scala-maven-plugin config and the
scala-2.13 profile, then resolve the warnings it surfaces:

- lint:adapted-args (3 sites in metric-map construction): replace
  `:+ ("k", v)` with `:+ ("k" -> v)` so the Tuple2 creation is explicit
  instead of silent argument-list adaptation.
- lint:nonlocal-return (2 sites): drop `return` from inside closures by
  using `Iterable.exists` for `matchVersion`, and an early-exit `while`
  over an iterator in `SparkOnHeapSpillManager.spill`.
- lint:type-parameter-shadow / pattern-shadow (3 sites): rename pattern
  bindings that shadowed enclosing-scope vals (`case in: ... =>`,
  `case expr => expr`, `case nativeXScanExec`).
- lint:inaccessible: silenced project-wide via
  `-Wconf:cat=lint-inaccessible:s` in both profiles, with a comment
  explaining why -- Auron implements Spark plugin SPIs whose contracts
  use `private[spark]` types (ElementTrackingStore, SparkUI,
  IndexShuffleBlockResolver, ShuffleWriteMetricsReporter); exposing
  them is required by Spark, not a defect.

Verified with `./auron-build.sh --pre --sparkver <v> --scalaver <s>
-DskipBuildNative` for spark-3.0/3.1/3.5 + scala-2.12 and spark-3.5 +
scala-2.13.

Closes apache#1924
@cxzl25 cxzl25 requested a review from Copilot April 27, 2026 12:05
@1fanwang 1fanwang changed the title [AURON #1924][BUILD] Add -Xlint:_ to scala-maven-plugin and resolve related warnings [AURON #1924] [BUILD] Add -Xlint:_ to scala-maven-plugin and resolve related warnings Apr 27, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR tightens Scala compilation by enabling full Scala linting (-Xlint:_) in the Maven Scala compiler configuration, and updates a handful of Scala sources to eliminate the newly surfaced lint warnings while preserving existing runtime behavior.

Changes:

  • Enable -Xlint:_ for Scala compilation (base + Scala 2.13 profile) and silence lint-inaccessible warnings project-wide via -Wconf.
  • Replace non-local returns in Scala collection loops with exists / while early-exit patterns to avoid NonLocalReturnControl.
  • Fix/avoid lint issues such as adapted-args tuple auto-tupled append and pattern binding shadowing.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file
File Description
spark-version-annotation-macros/src/main/scala/org/apache/auron/sparkver.scala Replaces loop+return with exists to avoid non-local return lint.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/shuffle/AuronBlockStoreShuffleReaderBase.scala Renames pattern binders to avoid shadowing warnings in nested matches.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeParquetInsertIntoHiveTableBase.scala Makes appended metric tuples explicit using -> to avoid adapted-args lint.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/NativeBroadcastExchangeBase.scala Makes appended metric tuples explicit using -> to avoid adapted-args lint.
spark-extension/src/main/scala/org/apache/spark/sql/execution/auron/plan/ConvertToNativeBase.scala Makes appended metric tuple explicit using -> to avoid adapted-args lint.
spark-extension/src/main/scala/org/apache/spark/sql/auron/memory/SparkOnHeapSpillManager.scala Removes non-local return from foreach by switching to an iterator while loop.
spark-extension/src/main/scala/org/apache/spark/sql/auron/SparkUDAFWrapperContext.scala Renames pattern binder to avoid shadowing (case other => other).
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronEmptyNativeRddSuite.scala Renames pattern binders in tests to avoid shadowing warnings.
pom.xml Adds -Xlint:_ and suppresses lint-inaccessible warnings (base + Scala 2.13 profile).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@yew1eb
Copy link
Copy Markdown
Contributor

yew1eb commented Apr 28, 2026

@1fanwang Thanks for the contribution! LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUILD] Add -Xlint:_ and resolve related linting warnings

3 participants