Local#

This page shows you how to set up PyFlink development environment in your local machine. This is usually used for local execution or development in an IDE.

Set up Python environment#

It requires Python 3.6 or above with PyFlink pre-installed to be available in your local environment. It’s suggested to use Python virtual environments to set up your local Python environment. See Create a Python virtual environment for more details on how to prepare Python virtual environments with PyFlink installed.

Execute PyFlink jobs in terminal#

You could execute PyFlink jobs locally as following:

curl -L https://raw.githubusercontent.com/apache/flink/master/flink-python/pyflink/examples/table/word_count.py -o word_count.py
python3 word_count.py

If there any any problems, you could check the logging messages in the log file as following:

# Get the installation directory of PyFlink
python3 -c "import pyflink;import os;print(os.path.dirname(os.path.abspath(pyflink.__file__)))"
# It will output a path like the following:
# /path/to/python/site-packages/pyflink

# Check the logging under the log directory
ls -lh /path/to/python/site-packages/pyflink/log
# You will see the log file as following:
#  -rw-r--r--  1 dianfu  staff    45K 10 18 20:54 flink-dianfu-python-B-7174MD6R-1908.local.log

Execute PyFlink jobs in IDE#

You need firstly configure the Python virtual environment for your IDE. See Configure a virtual environment for more details on how to configure the Python virtual environment in IntelliJ IDEA.

Right click on the job file and execute it. If there are any problems, you could check the logging messages in the log file which resides under the log directory of PyFlink installation directory as following:

# Check the logging under the log directory of PyFlink
ls -lh /path/to/python/site-packages/pyflink/log