Introduction

What Is FlowCV?

FlowCV at its core is a free open source dataflow framework for computer vision processing workflows.

Key Features:

• Free and Open Source
• Intuitive User Interface
• OpenCV Integration
• Highly Performant C++ Code
• Plugin SDK to Easily Extend Functionality
• Cross Platform Compatability

The advantage of the dataflow framework is that it allows you to quickly create complex directed graph processing flows that can branch and run in parallel to maximize processing resources and throughput.

To make the creation of these workflows easy and intuitive there is a node graph editor user interface. The editor allows you to easily create new nodes, connect the various inputs and outputs, adjust settings and send the resulting data over the network, for example to an embedded control system for automation, robotics or for data collection.

Once you're done creating your flow graph you can deploy it for production on your platform of choice using the headless processing application.

Why Was FlowCV Created?

The short answer is that nothing like it existed and I've been wanting something like this for several years now so I decided to just write it myself and offer it as open source in the hopes that others will find it useful and help contribute to its development so it can become even better and more useful, much like OpenCV itself which FlowCV is dependent on.

The longer answer is that I think a dataflow framework approach to computer vision is a great way to quickly prototype OpenCV solutions. You can quickly build out workflows and experiment with different nodes and go through various settings to see the results, all in real-time without writing a single line of code. With other software out there taking a similar approach such as Unreal Blueprints, Houdini, Nuke, Touch Designer and Blender Geometry Nodes, my hope is that FlowCV will one day reach a similar level of adoption and popularity and be used for everything from education to deployment in production quality applications.

FlowCV is still in the very very early stages of development, there is still a lot of work to be done from testing to implementing many more nodes and continuing to make improvements to the editor application. Please consider helping by using the application and providing feedback.

If you are a developer and want to help contribute to its development please see the section on becoming a contributor

Supported Platforms

Currently supported platforms:

• Windows 10+ (64-bit)
• Ubuntu 20+
• MacOS 12+

Installation

For easy installation use one of the pre-compiled binary install packages available in the github release section

Windows

• Download setup file
• Run the setup exe and follow the prompts

Ubuntu

• Download the debian package
• The run command in folder with deb package: sudo apt install ./flowcv-app_x.x.x_ubuntu-amd64.deb
  • If you're missing dependencies you can run: sudo apt --fix-broken install

To run after install just run:

FlowCV.sh

macOS

• Download MacOS DMG Package
• Double Click on DMG file
• Open Mounted DMG
• Drag FlowCV_Editor to Applications

If you get an error when launching you may need to modify the Privacy settings for Input Monitoring

Open Privacy Settings in System Settings:

Unlock Input Monitoring and Enable FlowCV_Editor:

Build From Source

System Requirements:

• C++ 17 Compiler:
  • Windows 10+ (64-bit): Visual Studio 2019 or 2022
  • Ubuntu 18+: gcc 8+ (technically you could use gcc 7 but see note below)
  • macOS 12+: XCode 13+
• CMake 3.20+
• On Ubuntu and macOS you will need OpenCV 4.1+ library package installed

** in gcc 7 std::filesystem is implemented in std::experimental::filesystem so the code won't work without some changes to the includes and namespaces involving std::filesystem

Clone the Repo

The documentation is added as a submodule, it is not required for any of the build steps so if you don't want the documentation source you don't need to clone recursively.

Make sure you have LFS installed and enabled, some of the various binary files (icons, images, etc.) are stored using LFS.

git clone https://github.com/FlowCV-org/FlowCV.git

if you do want the documentation source:

git clone --recursive https://github.com/FlowCV-org/FlowCV.git

For platform specific build instructions use the links below

Building for specific platforms

• Windows
• Linux
• MacOS

Building On Windows

• Building with CLion
• Building with Visual Studio and CMake GUI
• Building with Visual Studio and CMake cmd line

Building With CLion IDE

1. Open Project/Folder
2. In Settings Panel in CMake section create a Release Build
3. For CMake options set the following to ON or OFF based on what you want to build:
   • -DBUILD_EXAMPLES=ON
   • -DBUILD_EDITOR=ON
   • -DBUILD_ENGINE=ON
   • -DBUILD_PLUGINS=ON
4. Then build all in release

Building With CMake and Visual Studio (GUI Method)

1. Download CMake for Windows
2. Run CMake GUI

Now set the build path and source directory:

Now Add Entry for the build type:

After adding the entry press the Configure button, set the checkboxes for which executable you want to build

Then press Configure again and then Generate

3. Once complete it will generate a Visual Studio solution file in the Build dir that you can load: OpenCV_Dataflow_Framework.sln
4. After opening the solution file in Visual Studio then build solution

Building With CMake and Visual Studio (Command Line Method)

1. cd to the source path
2. mkdir Build
3. cd Build
4. run the following command (Change the build options as desired):

cmake .. -DBUILD_EXAMPLES=OFF -DBUILD_EDITOR=ON -DBUILD_ENGINE=ON -DBUILD_PLUGINS=ON -DCMAKE_BUILD_TYPE=Release

5. open Visual Studio project: OpenCV_Dataflow_Framework.sln
6. Build Project

OpenCV Download

The CMake build script should automatically download and extract the pre-compiled binaries during the build initialization but if for some reason you run into an issue you can download and extract the file manually.

OpenCV 4.5.5

add the following build setting to CMake (change path to your extracted location):

-DOpenCV_DIR=c:\path\to\extracted\opencv\files

Building On Linux

First install system dependencies (skip any of the ones you already have installed):

sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install libglfw3
sudo apt-get install libglfw3-dev

Download OpenVino Toolkit version 2022.1

Follow the installation instructions

After install run the OpenVino OpenCV Installer script:

/opt/intel/openvino_2022/extras/scripts/download_opencv.sh

Building With CLion IDE

1. Open Project/Folder
2. In Settings Panel in CMake section create a Release Build
3. For CMake options set the following to ON or OFF based on what you want to build:
   • -DBUILD_EXAMPLES=ON
   • -DBUILD_EDITOR=ON
   • -DBUILD_ENGINE=ON
   • -DBUILD_PLUGINS=ON
4. Then build all in release

Building From Command Line

if you already have CMake installed but it's < version 3.20, then first remove the old one:

sudo apt remove --purge cmake

then run:

sudo snap install cmake --classic

Now you're ready to build

cd to the source dir then (Change build options as needed):

mkdir Build
cd Build
source /opt/intel/openvino_2022/setupvars.sh
cmake .. -DBUILD_EXAMPLES=ON \
-DBUILD_EDITOR=ON \
-DBUILD_ENGINE=ON \
-DBUILD_PLUGINS=ON \
-DOpenCV_DIR=/opt/intel/openvino_2022/extras/opencv/cmake
cmake --build . --config Release -j 8

** replace -j 8 with your number of CPUs you want to utilize for compiling

Build On MacOS

First install system dependencies

You'll need home-brew to install the packages

Once you have home-brew installed then run the following commands to install the package dependencies

brew install glfw3
brew install glew
brew install cmake

Download OpenVino Toolkit version 2022.1

Follow the installation instructions

After install run the OpenVino OpenCV Installer script:

/opt/intel/openvino_2022/extras/scripts/download_opencv.sh

Building With CLion IDE

1. Open Project/Folder
2. In Settings Panel in CMake section create a Release Build
3. For CMake options set the following to ON or OFF based on what you want to build:
   • -DBUILD_EXAMPLES=ON
   • -DBUILD_EDITOR=ON
   • -DBUILD_ENGINE=ON
   • -DBUILD_PLUGINS=ON
4. Then build all in release

Building With XCode

Open a terminal and cd to the repo directory

Then Run (Change build options as needed):

mkdir Build
cd Build
cmake .. -DBUILD_EXAMPLES=ON \
-DBUILD_EDITOR=ON \
-DBUILD_ENGINE=ON \
-DBUILD_PLUGINS=ON \
-DOpenCV_DIR=/opt/intel/openvino_2022/extras/opencv/cmake
cmake --build . --config Release j 8

** replace -j 8 with your number of CPUs you want to utilize for compiling

Node Editor

The Node Editor User Interface is the front end interface for creating flow graphs, this is where you will create new nodes, connect the various inputs and outputs and configure parameters.

The Node Editor is built on top of the dataflow framework as an easy way to visually connect and configure nodes but the UI is not a dependency of the dataflow framework so if anyone wanted to they could build their own UI on top of FlowCV if for example they wanted to make a web based front end.

There are various elements to the UI and viewer nodes that allow you to interactively see the results of your flow graph in real-time.

Node Editor UI Elements

The Editor consists of 3 main Elements:

1. The Node Editor, this is where you create, connect and edit the nodes.
   • Right mouse button drag to pan the grid view around
   • Right mouse button click for context menu for creating new nodes
   • Mouse wheel to zoom in and out
   • Left mouse button click to select nodes

2. Properties Panel, this is where node properties will be displayed for editing various node parameters.
   • Left mouse button double click to open node properties in the properties panel
   • When there is multiple node properties in the list you can scroll to see them all
   • If you double click on a node again it will highlight its properties title in the list

3. Viewer or other docked control, initially your UI might come up different by default, but you can drag windows and dock them to various areas of the UI

Node Editor Shortcut Keys

In the Editor Section:

In the Viewer:

Editor Settings

Editor Settings can be found under the Edit menu, Settings:

The settings panel allows you to change various settings for the editor

• How many entries to save in the recent file history
• Flow Buffer Count (How many threads to allocate for node branching)
• Show UI FPS
• Extra plugin search paths

Nodes

Currently Implemented Nodes (Many more to come)

Internal Nodes:

Included Plugins:

Extra Plugins (separate repos):

Working With Node Properties

The properties panel will display the parameters for various nodes.

The property panel can be docked or undocked and moved where ever you prefer.

To activate / show a node property control in the panel simply double click on a node.

To highlight a node property section when multiple node properties are active at once simply double click on the node you want to highlight and the title section will turn green.

Click on the x in the title bar to remove the node property from the list.

You can also collapse the node property section if you want to keep it in the list but just want it out of the way temporarily.

Copy / Paste Nodes

Copying and Pasting nodes can be a quick and convenient way to duplicate nodes within your current flow graph, copy selected nodes between different node editor flows or to share node configuration with others online without having to send an entire flow file.

When cutting/copying selected nodes the node editor is actually querying the node state for each selected node and building a JSON object with all the relevant node information and connections as well as relative node placement to each other. Similar to the .flow file format used by the node editor to load and save flows.

You can paste the clipboard from a copied node graph into a text editor and it will look like this:

{
    "connections": [
        {
            "from_id": 2001,
            "from_idx": 0,
            "to_id": 3001,
            "to_idx": 0
        }
    ],
    "editor": [
        {
            "id": 2001,
            "location": {
                "x": 742.0,
                "y": 8.0
            }
        },
        {
            "id": 3001,
            "location": {
                "x": 940.0,
                "y": 33.0
            }
        }
    ],
    "nodes": [
        {
            "id": 2001,
            "name": "Video_Capture",
            "params": {
                "camera_settings": {
                    "Auto Exposure": -1.0,
                    "Backlight": -1.0,
                    "Brightness": -1.0,
                    "Contrast": -1.0,
                    "Exposure": -1.0,
                    "Focus": -1.0,
                    "Gain": -1.0,
                    "Gamma": -1.0,
                    "Hue": -1.0,
                    "ISO Speed": -1.0,
                    "Iris": -1.0,
                    "Saturation": -1.0,
                    "Sharpness": -1.0,
                    "Zoom": -1.0
                },
                "force_index": false,
                "fps": 30,
                "fps_index": 7,
                "list_index": 0,
                "set_load_man": false,
                "src_index": 0,
                "src_mode": 4,
                "src_name": "System Default"
            }
        },
        {
            "id": 3001,
            "name": "Transform",
            "params": {
                "aspect_mode": 0,
                "flip_mode": 0,
                "interp_mode": 0,
                "rotate_amt": 0.0,
                "rotate_mode": 0,
                "scale": {
                    "x": 100.0,
                    "y": 100.0
                },
                "scale_mode": 0,
                "translate": {
                    "x": 0,
                    "y": 0
                }
            }
        }
    ]
}

You can edit the information to make changes if needed or paste the buffer to someone on Slack or Discord and they can then paste that buffer into their node editor and get your same nodes, settings and relative layout to start working with.

Viewer Node

The Viewer Node is a simple 2D image viewer and is the primary way to preview your flow output.

There are a few configuration settings across the top of the viewer window.

You can configure the aspect ratio to be locked to width or height or be free.

Aspect Ratio

When locked to Height the viewer can only be resized by changing the height of the window, the width will be automatically adjusted as the window is resized.

Same thing when locking to Width by adjusting the width of the window the height will be automatically adjusted.

If using the corner resize then the window will snap to the new width or height once the mouse is released.

Free mode will allow you to resize the window to any arbitrary size with no regard to the original input image aspect ratio.

Channel Select

You can select which channel(s) of the input image to preview

If the image type is Grayscale then the channel selections will have no effect.

Inspect

You can inspect image details by holding the CTRL key down while hovering the mouse over the viewer window:

Color Map

If the input image type is 16-bit or floating point then it will automatically have a colormap applied which you can adjust from the colormap options that will appear:

You can scale the map range across the image data by adjusting the scale:

Depth_Viewer_3D Node

Depth_Viewer_3D is a 3D Pointcloud Viewer for inspecting depth data from 3D Cameras or other sources.

Window settings across the top from left to right:

• Camera FOV
• Toggle Between Orbit and Fly Mode
  • When in Fly Mode Adjust Fly Speed
• Toggle Show Grid On/Off
  • Adjust Grid Scale
• Turn MSAA On/Off
• Set Background Color

Mouse/Keyboard Controls:

• In Orbit Mode:
  • Right Mouse Button - Orbit Around Center
  • Middle Mouse Button - Shift Center Up/Down
  • Mouse Wheel - Zoom In/Out (Moves camera in Fwd/Back Vector)
• In Fly Mode:
  • Right Mouse Button - Change camera Pitch and Yaw
  • Middle Mouse Button - Pan camera Up/Down/Left/Right
    • You can hold the CTRL key and right mouse button for the same effect
  • Mouse Wheel - Zoom In/Out (Moves camera in Fwd/Back Vector)
  • Fly Keys:
    • W - Forward
    • S - Back
    • A - Left
    • D - Right
    • Q - Up
    • E - Down

If you double click on the node it will open up additional settings in the settings panel that will allow you to adjust 3D Point size, Shading Mode and more.

DNN - Deep Neural Networks

FlowCV is compiled using OpenCV 4.5.5 with OpenVino support so along with all the normal OpenCV DNN functionality you will also get extra OpenVino functionality using Intel's optimized inference engine.

For more info on OpenCV Deep Learning details see the following resources:

• Deep Learning in OpenCV
• Deep Neural Networks - OpenCV Documentation / Tutorials
• OpenCV DNN - A Definitive Guide

DNN Models and other related files:

• OpenCV Zoo
• OpenCV Extras
• OpenCV Samples

For specific details about FlowCV DNN Nodes see below:

Currently implemented DNN Nodes in FlowCV

• Image Classification
• Object Detection
• Human Pose
• Segmentation
• Text Detection & Recognition

DNN - Image Classification

Image classification will recognize and classify the contents of an image along with the confidence value.

See Deep Learning in OpenCV for information about supported frameworks and supported layers.

Some tested models with parameters:

DNN - Object Detection

Object detection will recognize various classes within an image and provide a bounding box, class label and confidence value.

See Deep Learning in OpenCV for information about supported frameworks and supported layers.

Some tested models with parameters:

DNN - Human Pose

Human pose estimation will estimate the skeletal pose of the human body and/or hand(s) depending on the model and mode. Some models also support multi-person or multi-hand estimation.

See Deep Learning in OpenCV for information about supported frameworks and supported layers.

Some tested models with parameters:

DNN - Segmentation

Semantic segmentation will create a color coded mask for each recognized class, the classes and colors are defined in separate files with correlating indices.

Colors are defined as RGB with values separated by spaces.

Example classes file:

Unlabeled
Road
Sidewalk
Building

Example color file:

0 0 0
128 64 128
244 35 232
70 70 70

See Deep Learning in OpenCV for information about supported frameworks and supported layers.

Some tested models with parameters:

DNN - Text Detection and Recognition

Text detection and recognition nodes can detect where text is in an image and then provide the output with bounding box position and text string.

Pass the output of text detection into the input of text recognition to enable the full process.

See Deep Learning in OpenCV for information about supported frameworks and supported layers.

Some tested models with parameters:

Text Detection:

Text Recognition:

Processing Engine

The Processing Engine is a headless processing application used to deploy flow graphs for production or testing on target platforms where you either don't have an active display/monitor or you don't want the extra overhead of the GUI engine running for final deployment.

Usage:

FlowCV_Engine.exe  -f <string> [--] [--version] [-h]


Where:

   -c <string>,  --cfg <string>
     Custom Config File
     
   -f <string>,  --flow <string>
     (required)  Flow File

   --,  --ignore_rest
     Ignores the rest of the labeled arguments following this flag.

   --version
     Displays version information and exits.

   -h,  --help
     Displays usage information and exits.

See Headless Example for more info.

Running A Flow In Headless Mode

Example Run:

FlowCV_Engine.exe -f face_detection.flow

Example Output:

Data Flow Processing Engine - v0.1.2

Looking for Plugins in: C:\Devl\FlowCV\FlowCV\cmake-build-release\Plugins
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\CsvFile.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\ImageLoader.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\ImageWriter.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\Openvino_Plugins\\Head_Face_Emotion.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\Openvino_Plugins\\Pose_Estimation_2D.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\OscSend.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\RealSense\\Realsense_Camera.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\SerialSend.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\ShapeColorizer.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\ShapeCounter.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\SimpleBlobTracker.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\TcpSend.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\UdpSend.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\VideoCapture.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\VideoLoader.fp"
"C:\\Devl\\FlowCV\\FlowCV\\cmake-build-release\\Plugins\\VideoWriter.fp"
16 Plugin(s) Loaded
Loading Flow File: ./simple_test.flow
Flow State Loaded, 5 Nodes Loaded and Configured
Flow Processing Started

Code Overview

I'm not going to cover every directory and file in this overview but I will go over a few of the key areas.

The directory structure is organized as follows:

project
│   README.md  
│
└───CMake
│   │   [various cmake config files]
│   │   ...
│   
└───Editor_UI
│   │
│   └───Common
│   │   │   entry.cpp               - Location of Editor Application main
│   │   │   ...
│   │
│   └───Node_Editor
│       │   node_editor.cpp         - Editor UI code
│       │   ...
│   
└───Examples
│   │   OpenCV_Dataflow_Test        - Simple example showing how to use the dataflow framework to build processing applications
│   
└───FlowCV_SDK
│   │   [SDK Files and Folders]     - The core FlowCV SDK for building applications or plugins
│   │   ...
│   
└───Internal_Nodes
│   │   [Various Internal Nodes]    - The core OpenCV functionality nodes
│   │   ...
│   
└───Managers
│   │   FlowCV_Manager.cpp          - A simple management class to keep track of node instances, connections and processing
│   │   Internal_Node_Manager.cpp   - A class for registering and handling internal node instances
│   │   Plugin_Manager.cpp          - A class for registering and handling external node plugins
│   │   ...
│   
└───Plugins
│   │   [Various Plugin Nodes]      - External node plugins to extend FlowCV functionality
│   │   ...
│   
└───Processing_Engine
│   │   Headless_Process_Engine     - Application for running flows in headless processing mode
│   
└───Templates
│   │   Internal_Node               - A template for creating new internal nodes
│   │   Plugin                      - A template for creating new plugin nodes
│   
└───third-party
│   │   [various third-party dependecies]
│   │   ...
│   
└───Tools
    │   PluginMaker.py              - Python script to help automate the creation of new nodes

Example Projects

Aside from the Node Editor and the various internal and external nodes the repo also comes with a few example projects to further illustrate how the FlowCV framework can be used to process computer vision workflows or create your own application using the included UI libraries and wrappers.

1. Headless Processing Engine
2. OpenCV DataFlow Test

1. Headless Processing Engine

This example is used as a way to run flows without the UI in cases where you want to deploy your computer vision processing solution to a target platform for production or testing that may not have a display interface configured.

Check out the Processing Engine Section for specific details on how to use it.

Code Overview

Create a command line parser using TCLAP library:

CmdLine cmd("FlowCV Processing Engine", ' ', APP_VERSION);
ValueArg<std::string> flow_file_arg("f", "flow", "Flow File", true, "", "string");
ValueArg<std::string> cfg_file_arg("c", "cfg", "Custom Config File", false, "", "string");
cmd.add(flow_file_arg);
cmd.add(cfg_file_arg);
cmd.parse(argc, argv);

Instantiate the Flow Manager class:

FlowCV::FlowCV_Manager flowMan;

Get Application Path and Load Config File (Platform Specific Methods)

#ifdef __linux__ // [Linux Method for App Path Here]
    ...
#endif
#ifdef _WINDOWS // [Windows Method for App Path Here]
    ...
#endif
#if __APPLE__ // [MacOS Method for App Path Here]
    ...
#endif
    std::string configFile;

    if (cfg_file_arg.getValue().empty()) {
        configFile = cfgDir;
        configFile += std::filesystem::path::preferred_separator;
        configFile += "flowcv_editor.cfg";
    }
    else {
        configFile = cfg_file_arg.getValue();
    }

    appSettings.configPath = configFile;
    ApplicationLoadSettings(appSettings);

Recursively load plugins from the plugin folder if it exists:

std::string pluginDir = appDir;
pluginDir += std::filesystem::path::preferred_separator;
pluginDir += "Plugins";
if (std::filesystem::exists(pluginDir))
    flowMan.plugin_manager_->LoadPlugins(pluginDir.c_str());

Get Extra Plugins From Config Plugin Paths

if (!appSettings.extPluginDir.empty()) {
    for (const auto &path : appSettings.extPluginDir) {
        if (std::filesystem::exists(path))
            flowMan.plugin_manager_->LoadPlugins(path.c_str(), false);
        }
    }
}

Load a flow graph file, exit if there is an error:

if (!flowMan.LoadState(flow_file_arg.getValue().c_str())) {
    cout << "Error Loading Flow File" << endl;
    return EXIT_FAILURE;
}

Initiate signal handling to exit gracefully in event of a ctrl+c or ctrl+x

Init_Signal();

Start the flow processing thread:

flowMan.StartAutoTick();

Wait in loop while flow processing is happening

while(!g_bTerminate) {
    this_thread::sleep_for(chrono::seconds(1));
}

If terminated and loop exits, then stop background processing thread and exit the application

flowMan.StopAutoTick();

return EXIT_SUCCESS;

2. OpenCV Dataflow Test

This example shows the follow things:

• how to initialize the manager
• find/load plugins
• iterate all loaded plugins
  • print out node info
• iterate all internal nodes
  • print out node info
• create some nodes and wire them together
• iterate over the active flow graph (get all node instances and connections)
• create a dockspace UI with some parameter controls and viewers

Code Overview

Use FlowCV namespace and instantiate FlowCV Manager:

using namespace FlowCV;
FlowCV_Manager flowMan;

Recursively load plugins from the plugin folder if it exists:

if (std::filesystem::exists(pluginDir))
    flowMan.plugin_manager_->LoadPlugins(pluginDir.c_str());

Iterate over all of the plugins and print out information about the plugin node:

uint32_t pCount = flowMan.plugin_manager_->PluginCount();
std::cout << "External Plugin Count: " << pCount << std::endl;
std::cout << "Plugin List: " << std::endl;
auto categories = getCategories();
for (int i = 0; i < pCount; i++) {
    NodeDescription plgNodeDesc;
    if (flowMan.plugin_manager_->GetPluginDescription(i, plgNodeDesc)) {
        std::cout << "--------------------------------" << std::endl;
        std::cout << "    Name: " << plgNodeDesc.name << std::endl;
        std::cout << "    Category: " << categories[plgNodeDesc.category] << std::endl;
        std::cout << "    Author: " << plgNodeDesc.author << std::endl;
        std::cout << "    Version: " << plgNodeDesc.version << std::endl;
        std::cout << "    Inputs: " << plgNodeDesc.input_count << std::endl;
        std::cout << "    Outputs: " << plgNodeDesc.output_count << std::endl;
        std::cout << "--------------------------------" << std::endl;
    }
}
std::cout << std::endl;

Iterate over all of the internal nodes and print out information about the node:

uint32_t iCount = flowMan.internal_node_manager_->NodeCount();
std::cout << "Internal Node Count: " << iCount << std::endl;
std::cout << "Node List: " << std::endl;
for (int i = 0; i < iCount; i++) {
    NodeDescription intNodeDesc;
    if (flowMan.internal_node_manager_->GetNodeDescription(i, intNodeDesc)) {
        std::cout << "--------------------------------" << std::endl;
        std::cout << "    Name: " << intNodeDesc.name << std::endl;
        std::cout << "    Category: " << categories[intNodeDesc.category] << std::endl;
        std::cout << "    Author: " << intNodeDesc.author << std::endl;
        std::cout << "    Version: " << intNodeDesc.version << std::endl;
        std::cout << "    Inputs: " << intNodeDesc.input_count << std::endl;
        std::cout << "    Outputs: " << intNodeDesc.output_count << std::endl;
        std::cout << "--------------------------------" << std::endl;
    }
}
std::cout << std::endl;

Create a few nodes:

uint64_t capId = flowMan.CreateNewNodeInstance("Video_Capture");
uint64_t blurId = flowMan.CreateNewNodeInstance("Blur");
uint64_t viewId = flowMan.CreateNewNodeInstance("Viewer");
uint64_t viewId2 = flowMan.CreateNewNodeInstance("Viewer");
uint64_t writeId = flowMan.CreateNewNodeInstance("Video_Writer");

Set initial state of the Blur node by creating a JSON object and setting some parameter key value pairs:

nlohmann::json blurState;
blurState["lock_h_v"] = true;
blurState["blur_mode"] = 1;
blurState["blur_amt_h"] = 5.0f;
NodeInfo ni;
if (flowMan.GetNodeInfoById(blurId, ni))
    ni.node_ptr->SetState(blurState.dump());

Set initial state of the video capture node using a JSON object:

nlohmann::json camState;
camState["src_index"] = 1;
camState["src_mode"] = 2;
if (flowMan.GetNodeInfoById(capId, ni)) {
    ni.node_ptr->SetState(camState.dump());
}

Wire up the connections between the various nodes:

flowMan.ConnectNodes(capId, 0, blurId, 0);
flowMan.ConnectNodes(blurId, 0, viewId, 0);
flowMan.ConnectNodes(capId, 0, viewId2, 0);
flowMan.ConnectNodes(blurId, 0, writeId, 0);

Print information about the current active nodes and their connections to each other:

PrintNodeConnectionInfo(flowMan);

Initialize the ImGui Wrapper class and create a Test Gui interface:

ImGuiWrapper imgui;
imgui.Init(1280, 720, "Test GUI");
ImGuiIO &io = ImGui::GetIO();

Start the flow processing thread:

flowMan.StartAutoTick();

Main loop for UI Handling:

while(!imgui.ShouldClose()) {
    ...
}

Poll events and start new frame:

  ImGuiWrapper::PollEvents();
  ImGuiWrapper::NewFrame();

Start ImGui dockspace:

  ImGuiWrapper::StartDockSpace(true);
  ImGuiID dockspace_id = ImGui::GetID("InvisibleWindowDockSpace");

Create an example main menu bar with some basic load/save functionality:

  if (ImGui::BeginMainMenuBar())
  {
      if (ImGui::BeginMenu("File"))
      {
          if (ImGui::MenuItem("Load", "CTRL+L")) {
              flowMan.StopAutoTick();
              flowMan.LoadState("./simple_test.flow");
              flowMan.StartAutoTick();
          }
          if (ImGui::MenuItem("Save", "CTRL+S")) {flowMan.SaveState("./simple_test.flow");}
          ImGui::Separator();
          if (ImGui::MenuItem("Exit", "CTRL+Q")) { break; }
          ImGui::EndMenu();
      }
      if (ImGui::BeginMenu("Edit"))
      {
          if (ImGui::MenuItem("Undo", "CTRL+Z")) {}
          if (ImGui::MenuItem("Redo", "CTRL+Y", false, false)) {}  // Disabled item
          ImGui::Separator();
          if (ImGui::MenuItem("Cut", "CTRL+X")) {}
          if (ImGui::MenuItem("Copy", "CTRL+C")) {}
          if (ImGui::MenuItem("Paste", "CTRL+V")) {}
          ImGui::EndMenu();
      }
      ImGui::EndMainMenuBar();
  }

Process properties UI for each node to display their properties in the property panel:

  ImGui::SetNextWindowDockID(dock_id_prop, ImGuiCond_FirstUseEver);
  ImGui::Begin("Properties");
  for (int i = 0; i < flowMan.GetNodeCount(); i++) {
      if (flowMan.NodeHasUI(i, GuiInterfaceType_Controls)) {
          NodeInfo nInfo;
          if (flowMan.GetNodeInfoByIndex(i, nInfo)) {
              std::string param_name = nInfo.desc.name;
              param_name += "_";
              param_name += std::to_string(nInfo.node_ptr->GetInstanceCount());
              if (ImGui::CollapsingHeader(param_name.c_str(), ImGuiTreeNodeFlags_DefaultOpen)) {
                  flowMan.ProcessNodeUI(i, imgui.GetImGuiCurrentContext(), GuiInterfaceType_Controls);
              }
              ImGui::Separator();
          }
      }
  }
  ImGui::End();

Process Main UI for each node that has a Main User Interface:

  for (int i = 0; i < flowMan.GetNodeCount(); i++) {
      if (flowMan.NodeHasUI(i, GuiInterfaceType_Main)) {
          NodeInfo nInfo;
          if (flowMan.GetNodeInfoByIndex(i, nInfo)) {
              if (nInfo.desc.name == "Viewer")
                  ImGui::SetNextWindowDockID(dock_main_id, ImGuiCond_FirstUseEver);
              flowMan.ProcessNodeUI(i, imgui.GetImGuiCurrentContext(), GuiInterfaceType_Main);
          }
      }
  }

End frame and update UI Draw events:

  ImGuiWrapper::FrameEnd();
  imgui.Update();

If closing, stop background processing thread before exiting:

flowMan.StopAutoTick();

Template Tool

The template tool helps automate the process of creating new internal or external (plugin) nodes.

You run it using python with the follow options:

Usage: PluginMaker.py [options]

Options:
  -h, --help            show this help message and exit
  -n NAME, --name=NAME  Specify New Plugin Name
  -a AUTHOR, --author=AUTHOR
                        Specify The Plugin Author
  -v VERSION, --ver=VERSION
                        Specify Plugin Version: 0.1.0
  -d ADD, --add=ADD     Add to Main CMake Project File
  -c CATEGORY, --category=CATEGORY
                        Specify Plugin Category: Source, Filter, etc.
  -i INPUTS, --inputs=INPUTS
                        Specify Number of Inputs with CSV Names: in,in,in
  -o OUTPUTS, --outputs=OUTPUTS
                        Specify Number of Outputs with CSV Names:
                        out,color,depth
  -p ITYPE, --intype=ITYPE
                        Specify Input Type, must match number of inputs (use
                        -l 1 for list): 1,2,4
  -t OTYPE, --outtype=OTYPE
                        Specify Output Type, must match number of outputs (use
                        -l 1 for list): 1,2,4
  -l LIST, --list=LIST  Set to 1 to get List of Categories and I/O Types (use
                        -l 1 for list)
  -e EXT, --ext=EXT     Create Internal or External Component, default 1, set
                        to 0 for Internal

For external plugin nodes you can use the -d option to have it automatically add the new plugin to the CMake build list.

For internal nodes it will automatically add the #include to the internal_nodes.hpp file, if you make a mistake or want to delete the node source files you will have to edit internal_nodes.hpp manually to remove the inlude statement.

SDK Overview

The FlowCV SDK is located in the FlowCV_SDK folder within the repo and can be referenced from outside projects when creating your own custom plugins for FlowCV.

The directory structure is as follows:

FlowCV_SDK
│   FlowCVConfig.cmake              - Main FlowCV CMake config file  
│
└───CMake
│   │   [various cmake config files]
│   │   ...
│   
└───include                         - FlowCV include files
│   │   ...
│   
└───src                             - FlowCV source files
│   │   ...
│   
└───third-party                     - Various third-party libraries
│   │
│   └───dspatch                     - DSPatch Dataflow Framework
│   │
│   └───FileBrowser                 - Simple ImGui File Browser library
│   │
│   └───imgui                       - ImGui - Immediate Mode Graphical User Interface library
│   │
│   └───imgui_wrapper               - Wrapper class for common ImGui functions
│   │
│   └───implot                      - Simple ImGui Graphical Plot library
│   │
│   └───json                        - Great JSON library
│   │
│   └───node-editor                 - ImGui based node editor library
│   │
│   └───tclap                       - Command line argument parser library

I tried to keep the amount of dependencies as small as possible to keep the overall project small and to make it quicker and easier to build.

The included third-party libraries are either relatively small or have been modified from their original code/repo so it was decided to just add them directly to this repo instead of as sub-modules and for the convenience of keeping everything self-contained.

OpenCV on the other hand is fairly large so on Windows the CMake build script will automatically download the pre-built binaries and development source needed for building this project from source.

On other operating system you will need to pre-install the necessary dependencies first before building, see the building from source section for more details.

Check out the Plugins section for more details on using the SDK to build your own plugin.

Creating A Plugin

To use the FlowCV SDK in your external project for making your own plugin simply add the following line to your CMake initialization options:

-DFlowCV_DIR=\path\to\your\flowcv\sdk\location

Add this to your CMakeLists.txt file:

find_package( FlowCV REQUIRED )

Add these to your add_library section:

    ${FlowCV_SRC}
    ${IMGUI_SRC}
    ${DSPatch_SRC}

Add this to the target_link_libraries section:

    ${OpenCV_LIBS}

You can either use the plugin template file as a starting point or use the PluginMaker.py python script to generate a more ready to go starting point.

A few key things to understand when creating your own plugin:

In the plugin initialization function you need to set some info for your plugin:

SetComponentName_("Plugin_Name");
SetComponentCategory_(Category::Category_Other);
SetComponentAuthor_("Author");
SetComponentVersion_("0.0.0");

Initialize the global instance counter for your plugin and increment it:

SetInstanceCount(global_inst_counter);
global_inst_counter++;

Then set the number, name and type of inputs and outputs:

// 1 inputs
SetInputCount_( 1, {"in"}, {IoType::Io_Type_CvMat} );

// 1 outputs
SetOutputCount_( 1, {"out"}, {IoType::Io_Type_CvMat} );

All of the image/data processing that happens on the Flow processing thread will happen in the Process_ method, this is where you will create all of your OpenCV or other processing functionality and output any of the results:

void PluginName::Process_( SignalBus const& inputs, SignalBus& outputs )

You can grab the pointer to the input like this:

auto in1 = inputs.GetValue<cv::Mat>(0);

Check that the pointer is valid before using it.

It is also recommended in the case of cv::Mat to also check to make sure the Mat object is not empty before attempting to process any image data.

Then if you are planning to do any operations that will modify the input image you should first copy it to a local buffer or use an OpenCV operation where the resulting output will be put into the local buffer. Do not attempt to modify the input image pointer buffer directly as other nodes may also be using it in parallel and you will get undesirable effects if you modify it while it's in use.

Next is GUI Section, this is where you first tell FlowCV if your plugin has any kind of interface associated with it:

bool PluginName::HasGui(int interface)

The current types are:

GuiInterfaceType_Controls,
GuiInterfaceType_Main,
GuiInterfaceType_Other

• Controls is for any of the standard properties that will show up in the properties panel for your node when selected.
• Main is for any standalone/separate window interface you want to create for your node for example like the Viewer node
• Other is anything doesn't fit the first two options.

Next is the Update GUI method for actually drawing your interface:

void PluginName::UpdateGui(void *context, int interface)

FlowCV will pass in the current ImGui context and what type of interface it is requesting to be drawn

First you need to set the context:

auto *imCurContext = (ImGuiContext *)context;
ImGui::SetCurrentContext(imCurContext);

Then you can test for which interface it is and draw your conrols/UI:

    if (interface == (int)FlowCV::GuiInterfaceType_Controls) {

    }

Inside the if statement you can use all of the standard ImGui functionality to implement your interface.

When creating controls it is recommended to use the CreateControlString helper function for naming controls to avoid naming conflicts.

ImGui keeps track of all UI controls by their control name so if you have different nodes with the same control name or even the same name within the same node it will have issues updating and working if there is a name conflict.

So for example to create an ImGui:Checkbox control do something like this:

ImGui::Checkbox(CreateControlString("Enable Setting", GetInstanceName()).c_str(), &setting_boolean);

That will create the control with the string you specify and then also get the instance name (node name + instance count) and append that to your control using a hidden string option so your control has a unique name. Keep in mind it could still be possible to have the same name within your node so be careful about how you name the actual control portion.

Next you have the section for Getting and Setting the state, this is used by the node editor when loading and saving flow graphs as well as for copying and pasting nodes.

std::string AprilTagPlugin::GetState()

and

void AprilTagPlugin::SetState(std::string &&json_serialized)

in these methods you will use standard JSON creation and parse to get and set the required state key value pairs.

for example if I wanted to save a parameter in the GetState method I would do something like this:

using namespace nlohmann;

json state;

state["border_size"] = border_size_;

std::string stateSerialized = state.dump(4);

return stateSerialized;

and to load/set that parameter in the SetState method:

using namespace nlohmann;

json state = json::parse(json_serialized);

if (state.contains("border_size"))
    border_size_ = state["border_size"].get<int>();

Check out some of the included Plugins to see more examples of how everything works

Help Needed

While there has been a lot of ground worked laid down already there is still a lot of work left to do.

Here is a list of things that are still needed (but no where near a complete list)

• More OpenCV functionality implemented as nodes... lots more!! (this is probably where I need the most help because there is so much to do)
• More camera hardware plugin support (MYNT EYE, Azure Kinect DK, Flir machine vision... just to name a few)
• Other computer vision or machine learning algorithms that aren't part of OpenCV
• ROS 2 integration / support
• Beta testing
• Making example / tutorial / how-to videos
• And much much more

If you would like to help with any of these things or have other ideas that you think would be good to implement please check out the contributor section and become a contributor to help out!

Becoming A Contributor

A few guidelines to be aware if you would like to become a contributor.

In general we are trying to follow the Google C++ Style Guide except in the case of third-party libraries that may not be using the same coding style in which case you should adopt their code style if changes are needed.

The rule for what should be an "Internal" node versus a "Plugin" is based on a few factors:

Internal

• Functionality based purely on OpenCV built-in functions
• No third party libraries (other than OpenCV or the few included within the FlowCV SDK)

External

• Uses third-party libraries or other external code
• Requires platform independent functionality
• Has functionality that would benefit greatly from easy updatability (in other words easy to swap out plugin for updates instead of re-building entire application)

Another note on plugins is that if the plugin requires additional external dependencies to build then consider whether it will be better to make the plugin its own separate plugin project like for example when supporting specific hardware like the RealSense 3D camera plugin.

Once you're ready to start contributing go ahead and fork the repo.

check the currently open issues for bugs or feature request you are interested in working on.

create a branch for your work.

when ready make a pull request to the target dev branch.

make sure your pull request pass any required tests, fix any issues as needed.

check for reviewer comments and provide feedback or changes based on reviewers comments

and that's pretty much it.

I appreciate any help you can provide and look forward to your contributions!

Feel free to email if you have any questions: richard@flowcv.org

Acknowledgements

A big thanks to the following people and their open source projects without which FlowCV would not be possible!