Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ This can be created through the Webserver UI by enabling the "in cluster configu

image::airflow_connection_ui.png[A screenshot of the 'Edit connection' window with the 'in cluster configuration' tick box ticked]

Alternatively, the connection can be {airflow-managing-connections}[defined] using an environment variable in URI format:
Alternatively, the connection can be {airflow-managing-connections}[defined{external-link-icon}^] using an environment variable in URI format:

[source]
AIRFLOW_CONN_KUBERNETES_IN_CLUSTER: "kubernetes://?__extra__=%7B%22extra__kubernetes__in_cluster%22%3A+true%2C+%22extra__kubernetes__kube_config%22%3A+%22%22%2C+%22extra__kubernetes__kube_config_path%22%3A+%22%22%2C+%22extra__kubernetes__namespace%22%3A+%22%22%7D"
Expand Down Expand Up @@ -64,7 +64,7 @@ include::example$example-pyspark-pi.yaml[]
----

This is called from within a DAG by using the connection that was defined earlier.
It is wrapped by the `KubernetesHook` that the Airflow Kubernetes provider makes available https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py[here].
It is wrapped by the `KubernetesHook` that the Airflow Kubernetes provider makes available https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py[here{external-link-icon}^].
There are two classes that are used to:

* start the job
Expand Down Expand Up @@ -105,7 +105,7 @@ image::airflow_dag_log.png[Airflow Connections]

NOTE: If the `KubernetesExecutor` is employed the logs are only accessible via the SDP logging mechanism, described https://docs.stackable.tech/home/stable/concepts/logging[here].

TIP: A full example of the above is used as an integration test https://github.com/stackabletech/airflow-operator/tree/main/tests/templates/kuttl/mount-dags-gitsync[here].
TIP: A full example of the above is used as an integration test https://github.com/stackabletech/airflow-operator/tree/main/tests/templates/kuttl/mount-dags-gitsync[here{external-link-icon}^].

== Logging

Expand Down
14 changes: 7 additions & 7 deletions docs/modules/airflow/pages/usage-guide/security.adoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
= Security
:description: Secure Apache Airflow by configuring user authentication and authorization.
:airflow-access-control-docs: https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html
:airflow-access-control-docs: https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/access-control.html
:keycloak: https://www.keycloak.org/

Secure Apache Airflow by configuring user authentication and authorization.
Expand Down Expand Up @@ -47,7 +47,7 @@ spec:
<1> The reference to an AuthenticationClass called `ldap`
<2> The default role that all users are assigned to

Users that log in with LDAP are assigned to a default {airflow-access-control-docs}[role] which is specified with the `userRegistrationRole` property.
Users that log in with LDAP are assigned to a default {airflow-access-control-docs}[role{external-link-icon}^] which is specified with the `userRegistrationRole` property.

You can follow the xref:tutorials:authentication_with_openldap.adoc[] tutorial to learn how to set up an AuthenticationClass for an LDAP server, as well as consulting the {crd-docs}/authentication.stackable.tech/authenticationclass/v1alpha1/[AuthenticationClass reference {external-link-icon}^].

Expand All @@ -61,7 +61,7 @@ An OpenID Connect provider can be used for authentication.
Unfortunately, there is no generic support for OpenID Connect built into Airflow.
This means that only specific OpenID Connect providers can be configured.

IMPORTANT: Airflow deployments on the Stackable Data Platform only support {keycloak}[Keycloak].
IMPORTANT: Airflow deployments on the Stackable Data Platform only support {keycloak}[Keycloak{external-link-icon}^].

[source,yaml]
----
Expand All @@ -84,7 +84,7 @@ spec:
<2> The reference to the Secret containing the Airflow client credentials
<3> The default role to which all users are assigned

Users that log in with OpenID Connect are assigned to a default {airflow-access-control-docs}[role] which is specified with the `userRegistrationRole` property.
Users that log in with OpenID Connect are assigned to a default {airflow-access-control-docs}[role{external-link-icon}^] which is specified with the `userRegistrationRole` property.

The Secret containing the Airflow client credentials:

Expand Down Expand Up @@ -130,7 +130,7 @@ Further information for specifying an AuthenticationClass for an OIDC provider c

== Authorization

The Airflow Webserver delegates the {airflow-access-control-docs}[handling of user access control] to the https://flask-appbuilder.readthedocs.io/en/latest/security.html[Flask AppBuilder].
The Airflow Webserver delegates the {airflow-access-control-docs}[handling of user access control{external-link-icon}^] to the https://flask-appbuilder.readthedocs.io/en/latest/security.html[Flask AppBuilder{external-link-icon}^].
The AuthManager in the Flask AppBuilder can be configured to fetch the user roles from the authentication backend, e.g. LDAP.
Instead of using the integrated authorization, the Stackable Data Platform also provides an AuthManager which delegates the authorization requests to an xref:opa:index.adoc[Open Policy Agent (OPA)].

Expand All @@ -142,7 +142,7 @@ You can view, add to, and assign the roles displayed in the Airflow Webserver UI

==== LDAP

Airflow supports assigning {airflow-access-control-docs}[Roles] to users based on their LDAP group membership, though this is not yet supported by the Stackable operator.
Airflow supports assigning {airflow-access-control-docs}[Roles{external-link-icon}^] to users based on their LDAP group membership, though this is not yet supported by the Stackable operator.
All the users logging in via LDAP get assigned to the same role which you can configure via the attribute `authenticationConfig.userRegistrationRole` on the AirflowCluster object:

[source,yaml]
Expand All @@ -164,7 +164,7 @@ spec:
==== OpenID Connect

The mechanism for assigning roles to users described in the LDAP section also applies to OpenID Connect.
Airflow supports assigning {airflow-access-control-docs}[Roles] to users based on their OpenID Connect scopes, though this is not yet supported by the Stackable operator.
Airflow supports assigning {airflow-access-control-docs}[Roles{external-link-icon}^] to users based on their OpenID Connect scopes, though this is not yet supported by the Stackable operator.
All the users logging in via OpenID Connect get assigned to the same role which you can configure via the attribute `authentication[*].userRegistrationRole` on the `AirflowCluster` object:

[source,yaml]
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/airflow/partials/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,6 @@
* xref:airflow:troubleshooting/index.adoc[]
* xref:airflow:reference/index.adoc[]
** xref:airflow:reference/crds.adoc[]
*** {crd-docs}/airflow.stackable.tech/airflowcluster/v1alpha1/[AirflowCluster {external-link-icon}^]
*** {crd-docs}/airflow.stackable.tech/airflowcluster/v1alpha2/[AirflowCluster {external-link-icon}^]
** xref:airflow:reference/commandline-parameters.adoc[]
** xref:airflow:reference/environment-variables.adoc[]
Loading