TULIPP
H2020-ICT-O4-2015
Grant Agreement n° 688403

Reminder of the main goals of the project

TULIPP (http://www.tulipp.eu) – Towards Ubiquitous Low-power Image Processing Platforms, is an EU initiative, funded by the Horizon 2020 programme, targeting the development of high-performance, energy-efficient embedded systems for the growing range of increasingly complex image processing applications.

The objectives of the project are defined as follows:

  • Objective 1 – Define a reference platform for low-power image processing applications
  • Objective 2 – Instantiate the reference platform through use cases applications
  • Objective 3 – Demonstrate and plan improvements of defined key performance indicators
  • Objective 4 – Start-up and manage an ecosystem of stakeholders to extend image processing norms

Scientific and technological objectives

TULIPP developed a reference platform that defines implementation rules and interfaces to tackle power consumption issues while delivering high, efficient and guaranteed computing performance for image processing applications (contribution to objective 1). Using this reference platform enables designers to develop an elementary board at a reduced cost to meet typical embedded systems requirements: Size, Weight and Power (SWaP) requirements. Moreover, for less constrained systems which performance requirements cannot be fulfilled by one instance of the platform, the reference platform is also scalable so that the resulting boards be chained for higher processing power (contribution to objective 2).

The project has delivered three Use Cases covering Medical X-Ray Imaging, automotive ADAS (Advanced Driver Assistance Systems) and UAVs (Unmanned Aircraft Vehicles). The new Use Cases, coupled with the TULIPP embedded computing reference platform, deliver outstanding results for embedded vision applications.

The Medical X-Ray Imaging Use Case combines an embedded computing board with a medical X-ray imaging sensor to eliminate the noise on images when radiation doses are reduced. The ADAS Use Case enables the implementation of pedestrian detection algorithms running in real-time on a small, energy efficient, embedded platform. The UAVs Use Case equips a UAV with real-time obstacle detection and avoidance capabilities based on a lightweight and low-cost stereo camera setup.

At first glance, Medical X-Ray Imaging, ADAS and UAVs would appear to have very little in common, but that’s only true when viewed from the perspective of the final application as they all have a requirement for high-performance image processing and they also suffer from the so-called SWaPC (Size, Weight and Power and Costs) computing constraints typical of embedded systems. TULIPP has addressed these challenges by taking a diverse range of application domains as the basis for defining a common reference processing platform comprising the hardware, the operating system and its programming environment that captures the commonality of real-time, high-performance image processing and vision applications.

Assessment of the main scientific and technology results

TULIPP’s Medical X-Ray Imaging Use Case demonstrates advanced image enhancement algorithms for X-Ray images running at high frame rates. It focuses on improving the performance of X-Ray imaging Mobile C-Arms, which provide an internal view of a patient’s body in real-time during the course of an operation to deliver, increases surgeon efficiency and accuracy with minimal incision sizes, aids faster patient recovery and lowers nosocomial disease risks. Using TULIPPs platform, which is the size of a smartphone, the Use Case demonstrates how radiation doses to which patients and staff are exposed, which are typically 30 times ambient radiation levels, can be reduced by 75% at the same time as maintaining the clarity of the real-time X-Ray images which would otherwise be rendered useless by the increases in the noise level on the images that a reduced radiation dose can cause.

ADAS adoption is dependent on the implementation of vision systems or on combinations of vision and radar and the algorithms must be capable of integration into a small, energy-efficient Electronic Control Unit (ECU). An ADAS algorithm should be able to process a video image stream with a frame size of 640×480 at a full 30Hz or at least at the half rate. The TULIPP ADAS Use Case demonstrates pedestrian recognition in real-time based on Viola & Jones algorithm. Using the TULIPP platform, the ADAS Use Case achieves a processing time per frame of 66ms, which means that the algorithm reaches the target of running on every second image when the camera runs at 30Hz.

TULIPP’s UAV Use Case demonstrates a real-time obstacle avoidance system for UAVs based on a stereo camera setup with cameras orientated in the direction of flight. Even though we talk about autonomous drones, most current systems are still remotely piloted by humans. The Use Case uses disparity maps, which are computed from the camera images, to locate obstacles in the flight path and to automatically steer the UAV around them. This is the necessary key towards totally autonomous drones.

The TULIPP Use Cases, coupled with TULIPP the development kit, comprising hardware platform, multi-core real-time operating system, development tool chain and guidelines, have demonstrated that the computational demands of complex image processing can be delivered in a diverse range of embedded applications within the context of challenging size, weight, power consumption and costs constraints.

The project began work in 2016 and finished on January 2019 with a hands-on-tutorial to the ecosystem. TULIPP provided vision-based system designers with a reference platform that defines implementation rules and interfaces designed to tackle power consumption issues while delivering guaranteed, high-performance computing power. TULIPP started an ecosystem of stakeholders that will last after the end of the project to provide its starter-kit and continuous improvements of the reference platform.

Potential impact and exploitation of results

TULIPP fulfilled the impacts expected by the call in the following way:

Impacts expected on the ICT 4 call objective (1)
Reinforce and broaden Europe’s strong position in low-power computing in traditional and new market segments by strengthening the technical competencies of European suppliers and the academic community

TULIPP impact
The technical improvements done during the project – with the study on the standards, the interfaces and the hardware components, the guidelines, the starter kit made available with a full platform comprising a power-efficient Hardware, a power-aware RTOS and tool chain – contributes to impact positively the position of Europe in low-power computing, not limited to traditional embedded computer vision systems but also on new technologies, like artificial neural network, that makes reachable in shorter-term new market segments, like agriculture robotic and other markets, like medical assistance at home, in longer term.
In addition, the implication of Sundance’s CEO on the PC/104 standard and the adoption of the Microsemi SoM as a replacement of the Zynq SoM will also have a positive impact on European companies and markets.
The TULIPP reference platform could initially be used across three key sectors (Medical, Automotive, UAV). The number of sectors will be enlarged after the project (e.g robotic for agriculture).
Thanks to the ecosystem, and the final tutorial, the project encourages new application domains to benefit from the solutions developed in the project.

Impacts expected on the ICT 4 call objective (2)
Reduction of energy consumption of servers by 2 orders of magnitude as compared to state of the art in 2013.
Double the productivity in efficiently programming and maintaining advanced computing systems powering cyber-physical systems as compared to state of the art in programming embedded systems in 2013.

TULIPP impact
All the efforts done during the TULIPP project was aimed at reaching the best “performance/energy consumption” ratio for embedded systems.
Between the initial implementation of the use cases to the implementation on the TULIPP platform, the performance/watt was improved by a factor 15 on some applications. The aim was- a factor 10 was expected at the beginning of the project. For instance, the medical use case now (in 2019) runs on a 10 watts Xilinx Zynq 7030 SoC while it was running on a 150 watts platform based on a PC with 2 intel core i7 processors before the project in start in 2015. Higher performance per watt is expected with the Zynq MPSoC ZU4 and will be reached this year. Even higher performances are expected in the next years with next-generation targets (e.g. Xilinx Versal) but also with adding more heterogeneity to the hardware (e.g. an Intel/Movidius Myriad SoC together with a Xilinx Zynq MPSoC). This is strength of the TULIPP reference platform to be able to benefit from technology improvement and to integrate new technology as it becomes
available.
This efficiency is achievable because the chosen hardware components are almost as easy to program as a general purpose processor.
The productivity is also drastically improved with the TULIPP tool chain that analysis the source code, provide the application programmer with hints to modify his code and to map functions on the accelerators of the hardware architecture. Moreover, the Lynsyn board together with the monitoring tool helps the developers to analyse the performance improvement of the mapping. On top of this tool, a design space exploration tool tries several mappings to determine the best possible mapping automatically.
The TULIPP platform provides the application developers with a gain in productivity to achieve the best possible performance-per-watt for systems where this criterion is crucial, such as Cyber-Physical-Systems.

Impacts expected on the ICT 4 call objective (3)
Increase the adoption of form-factor data-centres and heterogeneous highly parallel computing systems.

TULIPP impact
The TULIPP hardware choice was to adopt the PC/104 standard to develop the instance of the platform. This solution allows to stack several carrier boards to extend the hardware computing resources. Each carrier board can be equipped with a different module. Thus one board can be implemented with a Zynq SoC while the other one will mount a Myriad2 SoC to build a heterogeneous platform.
This solution will be used for a sequel project to build a vision system for agribots.
This standard and its possibilities are described in the D1.3 deliverable and discussed along with other standards and will be a part of the book issues in 2019.
Through its ecosystem, TULIPP spread knowledge on heterogeneous highly parallel computing systems and showed its members the benefit they can expect from such a system and the ease of programming such a, system have reached now.
While several results of the project have been made open source (i.e. the hardware, the tool chain, the guidelines, the reference platform), the other parts are available under license (the RTOS and its libraries).
Impacts expected on the ICT 4 call objective (4)
Higher involvement of SMEs, both on the supply and the demand-side. TULIPP impact
The TULIPP consortium is at 50% composed of SMEs.
The project was, from its inception oriented to involve a larger part of SMEs, to network with them to get feedback from them, and eventually, make business with them either through a partnership or through new businesses providing them with solutions fine-tuned for their applications.
The ecosystem includes a large part of SMEs (22 SMEs, 5 bigger Industrials). Some of them are on the supply-side providing hardware IPs, chips or boards, while some others are on the demand-side providing products to customers and requiring an energy-efficient embedded computer.
Inside the project, the supply-side SMEs now propose their products as a bundle (hardware + RTOS).

Impacts expected on the ICT 4 call objective (5)
Increase adoption of concurrency in applications across all sectors; higher degree of parallelism in applications; increased public trust in embedded applications due to secure and reliable architectures.

TULIPP impact
The concurrency is made easier thanks to the heterogeneity which will increase the adoption of such parallel and heterogeneous computing targets.
While the project did not address security, it increases the trust in embedded applications with safety thanks to the RTOS implemented in the project and available as a product at the end of the project.

Partners in TULIPP and how they will exploit the results

Thales – www.thalesgroup.com
Thales was the coordinator of the TULIPP project, leader of the reference platform and owner of the medical use case. The work carried out within the project provides Thales with an embedded solution for embedded intensive image processing that allows image denoising with a limited thermal dissipation. The ecosystem built during the project will support further improvement of the technology and new solutions to Thales needs.

Efficient Innovation SAS – www.efficient-innovation.fr
Efficient Innovation was in charge of the management of the project. The experience in research project management gained thanks to the project will be used for future consortia management.

Fraunhofer IOSB – www.iosb.fraunhofer.de
The Fraunhofer was in charge of the hardware development of the project and owner of the drone use case. The work carried out in the project provides the Fraunhofer with an embedded solution for image processing from video camera sensors and will allow further development of autonomous drones.

Hipperos – www.hipperos.com
Hipperos was in charge of the operating system developed in the project. The operating system was greatly improved during the project and is now a product on the shelf that the company can sell. The work carried out by RUB and TUD allowed Hipperos to get closer contact with Xilinx and to be considered as a partner for future developments.

Norges Teknisk-Naturvitenskapelige Universitet (NTNU) – www.ntnu.no
NTNU was in charge of the development of the tool chain in the project. The tool chain has been made available open source and will be the root for new development and research project in the university.

Ruhr-Universität Bochum (RUB) – www.ruhr-uni-bochum.de

Technische Universität Dresden (TUD) – tu-dresden.de
RUB and TUD worked together to provide support in the operating system and the tool chain of the project with reconfigurable devices. The work carried out by the two universities allows the operating system to implement

Sundance Multiprocessor Technology – www.sundance.com
Sundance was in charge of the production of the hardware and the communication with the ecosystem of TULIPP. The close work between the partners allowed Hipperos and Sundance to provide a ready-to-use solution for image intensive computing at low-power to their customers with a bundle containing hardware boards, operating system and tool chain.
The two companies will actively continue to work with the ecosystem being a group of potential customers.

Synective Labs – www.synective.se
Synective labs was the owner of the ADAS use case and implemented a pedestrian detection algorithm. The significant improvement gained during the project allowed the company to get new contracts in the ADAS domain.

Major issues

During the first year of the project, it appeared strongly difficult to define a new standard on image processing as there were a strong legacy on the application domains with a wide variety of interfaces and components. Therefore the consortium changed the orientation of the work package 1 with a set of guidelines and a methodology to capture them. This new orientation was presented in D1.1 and at the mid-project review.

This orientation was, however, not significant enough, alone, to be useful for the development of power-efficient applications and platforms. The consortium decided to come back to the initial idea to analyse the current standards and give information on their relevance, advantages and drawback in regards to their utilisation in embedded image processing platforms.

Thanks to the first integration week from WP5, the consortium understood that the guidelines bring useful information to the developers. Therefore, getting back to the initial plan did not mean to drop the work on guidelines. On the contrary, a methodology was added to improve their quality and make them generic enough to be relevant for any embedded image processing application – meaning not only the project use cases.

A subset of standards was thus selected as the best suitable to achieve the goal and some clues are given on how they could evolve to improve their energy efficiency.

The modification of the orientation of WP1 delayed the deliverable D1.1 but did not impact the discussions in the instantiation work packages (WP2, WP3 and WP4) which continued to analyse the use cases given by WP5. As an example, WP2 could analyse the best suitable chip and deliver earlier than expected a first instantiation of the TULIPP hardware platform. As the work continued on the instantiation work packages, knowledge on development processes, best practices, rules to follow and constraints was gained by the team and provided material to WP1 to write the first version of the reference platform D1.2 integrating the initial plan together with the guidelines. Because of the initial delay from D1.1 and because the work carried out in WP1 to collect all the material and pack it into a consistent document was a tremendous effort, D1.2 was issued with several months of delay. However, the consortium continued to work at a very strong pace on WP1 and D1.3 was delivered 3 months before its due date to the advisory board and the ecosystem to give them time to provide feedback and let the TULIPP consortium integrate the feedbacks in a final version of D1.3 before releasing it to the European Commission.

The initial deviation had a positive impact on the project as it allowed the consortium to come with the guidelines concept that is well received by the ecosystem.