HS Analysis is the worldwide leading company for digitalization in pathology and is working closely together with different partners, as automation in pathology is going to be a part of daily routine for pathologists. With the help of explainable AI and modern Deep Learning supported methods, HS Analysis team believes to seriously inspire diagnostic and research areas such as the early detection and therapy of cancer or other diseases.

Figure 1 – Use explainable AI to understand what lead to the decission of the AI

Figure 2 – Visualization and analysis of fluorescence data

To reach this goal we provide a variety of hardware and software tools ranging from Do-It-Yourself Artificial Intelligence (DIY AI) over Machine learning to deep and active learning to detect your Region of Interest on brightfield and fluorescence data in an automated manner and provide you with the information you need.

Figure 3 – Support of drawing tablets to generate ground truth data if necessary. This can be done locally or online

Explainable AI using Class Activation Maps

AI is making significant progress in almost all industries. However, the explainability of techniques like Neural Networks is still not given in most cases. This is the reason why Neural Networks are often called “black boxes”. To make predictions of AI more interpretable, Explainable AI is gaining more and more importance. Explainable AI (XAI) tries to make AI solutions understandable to humans. Methods like CAM (class activation maps) try to improve the interpretability of such complex models. In particular, CAM tries to explain which parts of the image contribute most to the output of the neural network. Especially in the medical domain, this can be helpful to understand why the model predicts an anomaly in a structure.

Multi model imaging approach + AI to address difficult cancer types

The tumour core (dark-brown area in the upper left insert) can easily be identified by histology or MRI in vivo. However, explicit details about invasive tumour cells, the presence of embedded islands of cancer stem cells (CSC) and how these permit the aggressive nature of Glioblastoma, can presently not be observed in vivo, as the penetration depth and spatial resolution of the available methods are inadequate.

A Novel concept of deep tissue molecular imaging, Photoacoustic CARS: By coupling Raman-based molecular optical excitation with ultrasonic waves high molecular specificity and high spatial resolution, real-time imaging, deep down into the tissue, can be realized. In combination with other microscopic methods like FLIM, 2+3 Photon or OCT a combined Picture of the Tumour area can be composed, stretching over several depth levels.

AI concept: 1) Increase of image resolution by corrective feedback loop with hardware and pattern enhancement. 2) Annotation of
regions of interest (ROI) for diagnostic purposes by using a novel data management system and AI network. After training the AI for one
modality, transfer learning will consecutively reduce training times of other modalities for similar ROIs. 3) Annotated images are used to
create ground truth data, for the training of AI, by classification of annotated objects. After training the first AI model, active learning is applied
to calculate uncertainty of classification and increase the detection rate of classes, using less data.

Treatment of lung carcinomas

Lung cancer such as non-small cell lung carcinoma (NSCLC) is the leading cause of death among cancers. A big hurdle in the treatment is the correct choice of drugs as tumors can be resistant upon activation of certain metabolic pathways

Trough combination of several pathway specific cellular indicators, like the nuclear localization of a pathway specific protein or interaction of two proteins, the activity of pathways can be assessed before the drug is used. In this fashion a personalized combination of anti cancer drugs can be administered.

With our Lung Cancer Analysis projects, you get the possibility to prescreen and automatically run a non-small cell lung cancer (NSCLS) analysis of your lung biopsies and detect cancerous tissues and further analyze them. For the detection an unsupervised deep learning network is used, so you don’t need to create ground truth data at all. This neuronal network prescreens and segments your biopsy into normal and cancerous areas, which can be inspected in the viewer and used for further analysis.

In the first step mark rectangles of the Region of Interests that should be analyzed. Afterwards the analysis can be started without any further actions needed. While analyzing the tissue, it will be prescreened and segmented into normal and cancerous areas, which can be inspected in the viewer after the Analysis has finished.

CTC based Tumor assessment

Initial studies in the field of basic research show that the analysis of so-called Circulating Tumor Cells (CTC), individual tumorous cells that enter the bloodstream and can lead to metastases, are a promising way of creating an individual profile of the tumor in a non-invasive way create. If one analyzes to what extent the CTCs are in a vital form or are apoptotic, the metastatic potential of the cells can be determined.

To achieve this, CTCs are isolated from a blood sample and analyzed by an EpiSpot procedure, identifying an apoptotic marker protein. The protein is identified by a fluorochrome labeled antibody which is recorded and analyzed automatically by an AI powered reader system. The system determines whether the circulating tumor cells are apoptotic or not and assesses the metastatic likelihood.

HSA Kit – Autonomous Tumor Examination and Cell Level Analysis

HSA Kit – Efficient and Precise Tumor Examination using Autonomous AI Systems

Laboratories in the pathology sector are facing a variety of challenges connected to image quantification and digital data handling. Particularly recent automatization and digitalization of clinical microscopy as well
as personalized medicine increase the workload for pathological laboratories dramatically.
Our customers from clinical laboratories are specialized in interpreting laboratory tests and evaluating cells, tissues, and organs to diagnose disease. A complete and correct pathology report is crucial to getting a
precise diagnosis and deciding on the best individual treatment plan for the patient. To achieve the best medical care, our customers use state-of-the-art imaging equipment and the most advanced image analysis techniques to examine thousands of issue samples on an everyday base.

Why Employing an Autonomous AI System for Clinical Pathology?

HSA Kit saves time and supports doctors with numerical and visualized data organized in an integrated database.
The intensities and the color shadowing in histological samples are variable, leading to inconsistent differences between the features of interest. − Pixel based quantification or thresholding will often not succeed.
Here, the deep learning network of HSA Kit takes additional factors to distinguish different elements of the image using autonomous AI systems. After annotating some features in the training module, our algorithm creates a solid classification matrix. The resulting segmentation indicates the cross-section area of the features of interest and provides at the same time all relevant statistical data describing size, relative position amount and shape.

With HSA Kit accurate counting and documented quantification of various elements in microscopy images are now achievable within minutes!

Annotation Tools

Use our annotation tools to create your own annotations. These can be used to make custom analyses based on your annotations or for our Deep Learning model creation workflow. This enables you to have all tools needed to create your own Deep Learning models for segmentation tasks and image analysis channeled in one single Software. (Also see AI Cockpit)

To be able to reflect your specific use-case, you have the possibility to create and share your own structure tree. Here you can add your main parent structures as well as substructures of the different structures. Additionally, classes can be added to the different structures to also prepare your annotations to train classification models.

Automatic slide alignment

HSA Kit provides the possibility to align multiple whole slide images automatically with minimal human interaction. The alignment allows cell level comparison of different stains.

Automatic multiplex analysis

Whether its confocal or more complex multiplexing methods like Wavelength Multiplexed Fourier Ptychographic Microscopy, multiplexing has become a common methodology to get more and more information out of images. With our solution we can analyze over 4 channels simultaneously and create annotations in concert with image co registration. This allows secondary staining where one or several initial staining’s are lost but can be overlayed to mark the position of unstained organelles. In this example secondary staining with FISH has removed the initial DAPI staining but using image co-registration the DAPI annotations can be overlayed and thereby mark the position of the nuclei.

Large whole slide images(70000 x 70000 pixel), with thousands of annotations and multiple color channels, are displayed fluently due to our annotation-focus-function. High hardware demands? This clip was recorded on a gaming-laptop.

Zebrafish video annotation

Our focus for the Zebrafish model organism is on video analysis, which is particularly important for drug screenings to ameliorate or prevent diseases.

Our deep learning algorithm automatically segments videos of the zebrafish heart detecting atrium and ventricle parts. In addition, it calculates heart parameters such as heart rhythm, rate and fractional shortening. Our model is an efficient combination of the deeply improved classic trustworthy DL methods and unique enhancements in accuracy for segmentation and calculations, which enables a comprehensive analysis of the cardiac microscopy videos.

Moreover, we provide a remarkably user-friendly software set up in temporary art for generating ground-truth data especially for analysis of video sequences, where a user employs innovative Time-Line Tool for a graphical and temporal overview at the same time.

Along with the already implemented features such as interpolation (automated generation) of annotations, in the near future we will release a software upgrade allowing users to see graphically the heart rate and heart rhythm directly on the Time-Line tool.

Such cardiac metrics as heart rhythm, rate and fractional shortening play a key role in determining the pathological causes of heart attacks. For this reason, it is of great importance to digitalize and automate the work of pathologists when studying and evaluating video sequences of healthy and mutated zebrafishes.

Our software allows us to annotate video files quickly through interpolation . By annotating your structure of choice (here heart) in two scenes the scenes in between will be annotated automatically by AI (dotts in the lower red and blue line indicate annotated scenes).

The annotated atrium and ventricle of the zebrafish heart. (dotts in the lower red and blue line indicate annotated scenes)

Brain analysis

The demyelination of axons causes MS and other neural disorders. In this project the role of demyelination on the formation of schizophrenia is analyzed. For this myelin rich brain structures are annotated automatically by AI.

Immunofluorescence or brightfield brain slices are recorded by us or the customer. Consecutively the software automatically annotates the respective brain regions, corrections are possible afterwards. Directly or in a second step substructures are annotated.

Our software allows us to instantly analyse and annotate regions of the brain based on their structural composition and location.

In a second step substructures like individual cell types, nuclei or aggregates are annotated.

Single cell tracking and transcriptome analysis

Cells are recorded by a confocal microscope with a laser module that allows laser ablation or bleaching of cells. With a specialized AI model all cell-borders, in the area of interest, are annotated. Their features, like circumference, area change or specific length of cell borders are also displayed in real time and statistically analyzed.

Once a cellular behavior of interest is found, in this example contraction whilst surrounding cells expand, the cell is tracked. The sample is shortly chemically fixed and the cell cut out of the tissue with a laser beam. The cell is then analyzed with single cell RNA-seq and the behavioral pattern potentially linked to transcriptome changes.

A novel proteomics tool: AI powered protein identification and structural prediction

The holy grail of protein prediction is to calculate the correct protein structure with its epitopes and functionalities. The first step on this road is the identification of the amino acid sequence. A novel approach allows the prediction of the sequence by peptide array, using sequence prediction by bound overlapping peptides. On the array peptide spots harbor bound peptides that represent the human peptidome. Peptides in the test sample bind to the peptides on the peptide spots and are subsequently identified by antibodies.

The location of each antibody bound peptide spot is referenced with the structure of the peptides bound there. With the help of a novel AI powered system the amino acid sequence is determined. Subsequently the system is used to make epitope and structural predictions for the bound proteins.

Diagnosis of vital data

In the emergency room every second counts. Therefore it is of vital importance that the right diagnosis is found quickly and reliably. Under stress and extreme time pressure humans are prone to errors and keeping an overview over large amounts of vital data can be difficult.

With the help of AI vital data input from diagnostic instruments (EEG,ECG. blood pressure etc.) and doctors is processed and compared with a reference database for incidences. A preliminary diagnosis as well as potential measures that can be taken to ameliorate the situation are proposed together with an estimation of survivability.

Non invasive brain computer interface to treat psychological conditions with the help of virtual reality.

Exposure to the right visual and auditory cues has been found to ameliorate several psychological disorders such as PTSD or anxiety disorder. As such, virtually exposing the right cues at the right time promises healing potential for a plethora of psychological disorders. Ideally, a personalized therapy would directly address specific aspects of the disorder that are patient specific. With present technologies this process however takes a lot of time and does not allow to react flexibly to the underlying processes.

To ensure quick and reliable reaction the the brains impulses the virtual reality screen is combined with a brain computer interface (BCI) headset. The headset records EEG signals non invasive and feeds them to a specifically designed hardware system which uses a spiked neuronal network(SNN) to analyse the data upon arrival. A SNN is, compared to a normal neuronal network more energy efficient as it processes data as it arrives in “spikes” and not continuously.

A innovative approach to food testing

Testing food for bacteria in a fast and reliable manner is one of the challenges of our time. In particular when multi resistant germs are on the rise due to widespread use of antibiotics in the meat industry. But also vegetables can house a variety of harmful bacteria or contaminants.

To address these issues effectively food samples are analysed by Raman spectroscopy or general microscopy (brightfield, fluorescence etc.) or both to make the required assessment. The data is then compared to a reference database containing pre-established information about relevant bacterial strains or other contaminants and a final assessment about the safety and quality of the food is made.

CVUA (Chemie veterineres Untersuchungsamt) uses AI assistants for improved assessment.

In the Histo-point-counting and Histo-classification procedures, meat samples are examined for quality by AI in accordance with all statutory regulations. The AI assistant informs the staff about the severity of the deviation of the sample from the desired standard. The AI assistant informs you when a critical state of the examination is reached. AI Assistant also creates reports (in paper form and digital) to achieve traceability according to regulatory requirements. Due to the perfect integration into the entire system, all further processes are initiated in order to do the work of the inspectors quickly and easily. This means that the CVUA has a completely digital process in accordance with all standards. By and large, the software is a perfect software for daily work in a CVUA because it was created together with employees of a CVUA.

The software is smoothly integrated into your network infrastructure and runs in a docker software container that offers high system stability, operating system-flexibility and scalability.

The well structured classification tool allows the systematic classification of meat samples according to general standards. First a grid is chosen that divides respective sample into tiles. Then the composition of each tile is classified in food assessment categories like fat, tissue, bone etc. For each of the categories you simply klick on the percentage range. In addition one can make easy comments in the file or measure areas of interest. Finally the results are exported as an Exel file that allows statistical evaluation.

In-Situ-Überwachung phototropher Prozesse zur maßgeschneiderten Produktion von Algeninhaltsstoffen

Microalgae have a very interesting product spectrum, with possible uses in food and feed, personal care, agrochemicals and basic mixtures for new materials. To upscale production quickly and without losses it is necessary to tightly control the growth and health of the algae. This however has proved difficult with conventional sensors and methods. In particular then a system is needed that can adapt to multiple production types and industries.
Algae production types: 1. Flat plate airlift reactors 2. Raceway Pond 3.Tube reactors 4. Open Pond

To solve one part of the problem, novel AI powered systems are used in concert with newly developed sensors to control and continuously improve production. This allows seamless upscale of production and significantly reduced costs due to faster reaction times for contamination and technical errors. But to optimize the entire production line a more holistic approach is needed.
Standard Sensors: 1) Pressure, 2) pH, Temperature, 3) Optical density
Multiport (new sensors): 4) Carotenoids 6) Phycobilins

The AI supported bio-factory-system allows supervision of the whole production line. Each step of the production cycle is affected to varying degrees by AI supervision.
In particular the Quality Control module benefits the greatest. Here all steps in the production cycle are mapped digitally in form of a virtual model of the actual factory. The effect of delays or problems in each module are predicted to the whole production line, effects on product numbers and additional costs are extrapolated. This system further allows predictions on how changes in the setup will affect the stability of production line in addition to product quantity and costs.

HSA-Workshop

The HSA workshop is a cooperative project with the Karlsruhe Institute of Technology and allows the development of novel analytic systems:
1) Custom-build multimodal microscopes 2) Wavelength modulated Raman spectrometers 3) Frederik: A autonomous deep learning system to detect objects in real time 4) Annotated images or statistical data is automatically returned to your system of choice

Combined hardware-software development and IoT

A prototype on an electrical breadboard. Many colorful wires are to be seen as well as individual modules on perfboards mounted to the main breadboard. — Some of the most frequently used applications of for artificial intelligence and deep learning include data analysis and interpretation of sensor data. We can not only provide the software for these applications but will also develop sophisticated hardware in parallel to software solutions for optimized data acquisition.
This form of coordinated simultaneous development allows a direct analysis of a wide range of received signals, providing high-quality data to our analysis software, and thus ensuring optimal results.
Furthermore, the hardware can be designed with full communication capabilities including internet access, providing Internet-of-Things (IoT) capabilities to any device that requires them.
The combination of hardware fitted to the application with according AI-software allows for flexible integration into any system. Throughout the development process, the location and level of data processing and autonomy can be adjusted: Using integrated software on the device or sending all measurements straight to a dedicated analysis server impact the cost and speed of data transfer as well as cloud computing and will be fitted to the intended use-case accordingly.
Similarily, the same principle is applicable to remote control of devices through the internet – either using self-hosted solutions or full-on cloud services.

ESR Viewer

ESR Data can be shown and analyzed in the HSA Kit Software.

Multiple spectrums can be viewed, zoomed in, hidden.

Areas of the chart can be selected and with parameters analysed.

Quantum dot aggregate analysis

Composite films with Quantum Dot sputterings are used in modern display technology for products ranging from televisions to smartphones. One example is QLED, which is distinct from OLED, in that it isn’t self-emissive, and still makes use of a backlight. It uses a quantum dot colour filter in front of its LCD backlight, which improves contrast and color vibrance.

To provide good results and be able to reduce production costs, the HSA Kit Web can be used to run 2D and 3D computer vision analysis on the QD films and use machine learning to predict the production output joined to production parameters.

3D view of quantum dot aggregates, including classification by volume

Detection of skin diseases

To alleviate the diagnostic burden on doctors and waiting times on patients we strive to develop an app based solution for the automatic detection of skin diseases. This modulewill use viso-tactile diagnostic to analyse the area and create a 3d model and output a probability of the respective disease. Medical personnel can be contacted via the app to address the diagnosis and give mental support. The medical data can be uploaded to a secure database for exchange between doctors.

Detection of Glaucoma

We strife to create an App that allows the patient to analyze their eye with the cellphone camera by themselves for a first analysis. The App will render a 3D image of the eye from a sequence of images, taken from different angles. Medical personnel can be contacted via the app to address the diagnosis and give mental support. The medical data can be uploaded to a secure database for exchange between doctors.

After taking the image the app allows direct communication between the doctor and the patient and access for both to a encrypted database containing the patients medical data. In addition, as part of a national project to improve the diagnostic process, Multi-omic data such as genetic/proteomic tests or feces analysis can be used to complement the diagnosis. As such it allows a new level of personalized treatment.

Event based Cameras

Conventional cameras have several limitations that make them inefficient: They cant follow moving objects and thereby often have blind spots. For the same reason they cannot focus on specific objects, like faces, to give additional information. On top of that they are energy inefficient because they are constantly active.

An event-based Camera can solve these issues. By using a Spiked Neural Network (SNN) AI it can follow a specific target and focus onto its features. It can compare those features with a database and identify the object, giving a notification as response or causing a reaction like sounding a physical alarm or locking doors. In addition it uses much less power to operate because it is activated event based.

Additive Manufacturing

The AM process to ‘print’ objects with different materials is often poorly understood. The exact processing parameters and conditions often vary substantially. It is not seldom that insufficient data available for e.g., laser deposition consisting of only a few sets of processing parameters. These datasets may not always be enough for standard machine learning techniques and property prediction of processing parameters may fail. However, our deep-learning based optimisation techniques can reuse the data from earlier projects and studies to substantially improve the situation, potentially providing information that would cost years to obtain by more conventional means.

Key advantages:

Thorough understanding of all relevant conditions for reproducible and scalable AM processes
Higher AM product and process output
Systematic use of all project data: store and reuse when needed

Batteries

Electric vehicles and energy storage from renewable sources are the major drivers of the modern battery industry. The importance of improved performance of battery systems cannot be overstated. The key parameters: charging capability, energy density, and costs have all to be improved substantially in as short time span as possible.

AI-based methods are powerful tools to exploit the existing data to propel the discovery of novel battery materials, improve battery packs, and optimise battery management systems. As the field is still rapidly developing, the available parameter sets are often very limited and incomplete. In such situations our deep learning networks can prove to be invaluable assets.

Key Advantages:

Aid the design of new, improved, battery materials with minimal toxicity
Multiscale battery optimisation including the battery packs and management systems
Keep R&D costs down: replace experimental processing parameter optimisation by deep learning networks and efficient scale-up

Smart Industry: Control, Optimisation and Automation of Industrial Processes

Every business has a number of core processes, which must be fully understood, under control and optimised. The core process could vary quite substantially: it could be a production process, such as materials production, or it could be a business processes, such as management of the supply chain, complex logistics, etc. To maximise and maintain the quality of the final product while keeping the costs down is universally valid for basically any manufacturer. Historically, most of manufacturers concentrate on a subset of available data as overarching, ubiquitous and robust solutions were simply not present to tackle this level of complexity.

The deep learning networks implemented within HSA KIT can be applied to a wide variety of numerical datasets. Then the selected network can identify most relevant process parameters, their specification for the required accuracy in the final output and predict the most favourable parameter combination for optimal accuracy vs. cost ratio. Thereby, the customer can identify outliers, maintain required quality level and manage risk in the processes.

Key Advantages:

Industrial digitalisation: establish full monitoring of the process parameters
AI-optimisation: use all the available digital data to remove the bottlenecks
Deep-learning automation: allow neural networks to find the optimum of the multi-parameter process optimisation problem

Data Validation and Analysis

Regardless of the customer objective (scientific, engineering, financial, or business applications), it is always preferred to have the input data as clean and complete as possible as this is a scientific prerequisite to extract maximum information.

HSA KIT is designed to tackle precisely this kind of problems: extract maximum information from the incomplete data sets! It will automatically generate models that can help you visualise and understand the data, asses the quality and identify the outliers. Further, these models can assist in expanding the data sets (new acquisitions), optimise and make predictions – all necessary steps for effective decision-making.

Key Advantages:

Assisting customers in possession of complex, multi-dimensional, but incomplete datasets
Understand, trim, and prepare the data for seamless usage of AI algorithms
Asses the quality of the presented data, quantify uncertainty

Design of Experiments

Systematic experimental approaches are common and essential in pharmaceutical and chemical industry, to name just a few. They are often used to support rational design of new, improved materials. But it goes way beyond novel materials development. Market research, business or finance projects can be easily put in the same category as one has to test and improve the effectiveness of a model. Regardless of the target, the data acquisition is expensive and time intensive. Efficient information extraction can make a significant difference.

AI methods are designed for this. However, if the data is of limited both quantity and quality – as it is in many experiments, deep learning software and experienced operators are the answer. HSA KIT can generate models that can assist the understanding of the data. It can predict an optimal next step the experimentalists should take. Further, it can be applied to a wide range of numerical datasets.

Key Advantages:

Get more from fewer experiments!
AI-based decision-making process to guide the evolution of the experimental approach (target / candidate selection, etc.)
State of the art, beyond conventional design, deep-learning approach to systematic experimentation

Optoelectronic Devices

An everyday example of a modern optoelectronic device is a mobile phone display. Display technology is very important for the future of the electronic devices including the upcoming handheld medical diagnostics devices. As nanotechnology and miniaturisation are major driving forces behind the development of smart materials, understanding the microscopic details of surfaces and thin films used in these devices is of paramount importance.

Together with a number of industrial and academic partners we have developed an AI software module within HSA KIT to analyse surfaces of thin films used in quantum dot light emitting diode devices (QLED). HSA KIT can provide automatic deep learning analysis of microscopic images and automatic optimisation of production parameters, which is used in production of a wide plethora of smart nanomaterials (OLED, batteries, solar cells, super capacitors, etc.)

Key Advantages:

Aid the design of new, improved, optoelectronic materials
Multiscale device optimisation including the whole thin film stacks
Keep R&D costs down: replace experimental processing parameter optimisation by deep learning networks and efficient scale-up

Diagnostics Techniques Integration

A possibility to perform additive manufacturing on nanoscale opens new high-tech possibilities. The nano-patterned surfaces are highly functional and have been successfully used as scanning tools. This is critically important in smart device development, as many diagnostics devices could be transformed in miniature, handheld devices providing an unprecedented mobility to the operators.

A true breakthrough is in the fact that we can integrate several diagnostics (e.g. spectroscopic: electromagnetic, Raman, ESR) tools into a single device. Using our expertise in nanotech sector, we are currently developing an AI-based software analysis tool, which can store different diagnostics signals, cross reference them and extract maximum information about the measured sample.

Key Advantages:

Extract the maximum available information in a systematic manner
Integrate and analyse all the available data on equal footing
Use automation together with the latest miniaturisation techniques and materials to design an easy-to-use portable diagnostic device with automated streamlined output, which can aid the decision-making process in intuitive manner

Medical diagnostics miniaturisation

The latest trend in medical diagnostics are the smart devices, often combined with latest nanotech materials development. By combining our knowledge in both medical and nanotech sectors, we are developing integrated hardware/software solutions for our partners in medical diagnostics.

Our focus is on integration of different diagnostics techniques e.g., blood analysis is combined with tissue analysis, etc. where correlations between results stemming from different techniques are analysed by means of deep learning and other advanced AI techniques. The HAS KIT works fully autonomously on the imbedded hardware of miniaturised devices.

Key Advantages:

Integrate and analyse all the available data on equal footing
Provide mobility and automated quick response analysis for first responders
Identify and provide necessary information for safe and systematic decision making