I have Airflow and Airbyte installed locally with Docker. I want to set a connection in Airflow to connect Airbyte. I read the Airbyte docs and did exactly what it says but I am getting error. I have configured Airflow’s docker compose yaml to install necessary packages.
ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:- apache-airflow-providers-http apache-airflow-providers-airbyte apache-airflow-providers-airbyte[http]}
My Airflow executor is CeleryExecutor
In Airflow I configured the connection how excatly the Airbyte’s docs says. I also tried with Conn Type: Airbyte but still getting the error.
The error says:
HTTPConnectionPool(host='localhost', port=8001): Max retries exceeded with url: /api/v1/health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f30e9e4fb10>: Failed to establish a new connection: [Errno 111] Connection refused'))
2
Answers
Finally got around to testing this. For me using the Airbyte connection type that comes with the Airbyte provider plus including the username and password (default is "airbyte"/"password") worked with Airflow 2.5.1 and Airbyte provider 3.2.0.
On the Airbyte side I followed their getting started docs.
Airbyte’s blog covers this scenario and how to get it working: https://airbyte.com/tutorials/how-to-use-airflow-and-airbyte-together