DAG not recognized. Airflow 3.1 with Docker.

[maithtruong]

Hi everyone, can someone please help me check this problem?

I used Airflow with Docker. I did proper mappings so within the container bash, airflow-scheduler and airflow-webserver can recognize the Python file within the dags folder. But it doesn't recognize the DAG itself and wouldn't show on the UI!

So it sees that dags/flight_operations.py exists but can't see the DAG in the file, trying to list the DAG just shows "No data".

docker-compose.yml
dags script

Things I've tried doing:

• using sys.path so the container recognize the scripts in the scripts folder
• creating a Dockerfile to install the necessary dependencies
• add PYTHONPATH: /opt/airflow in environment for each of the airflow services

Thank you in advance :D.

[potiuk]

No access to your private repo. There is https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-discovery-safe-mode that you might need to enable if your Dag fole does not contain DAG or airflow , also there might .airflowignore. It's highly recommended that all your scripts are in Dag Bundle folder when you are importing them (then no sys.path manipulation is needed. You can follow https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html

[maithtruong]

Sorry, I made it public now. Thank you, I'll try those solutions :D.

[maithtruong]

Ok, so I tried the solution @potiuk gave me:

• add AIRFLOW__CORE__DAG_DISCOVERY_SAFE_MODE: "False" to YAML file

I also found this link:
Standalone DAG Processor Required
Airflow 3.0 now requires the standalone DAG processor to parse DAGs. This dedicated process improves scheduler performance, isolation, and observability. It also simplifies architecture by clearly separating DAG parsing from scheduling logic. This change may affect custom deployments that previously used embedded DAG parsing.

So I modified the YAML file as such:

• add a separate airflow-init service and initialize the db using it (previously i used the api-server to initialize it)
• add an airflow-dag-processor service
• modify the rights so the service can access the scripts folder

I still put the scripts outside of the dags/ folder because the dag processor seem to be reading from there.
The DAGs are recognized now!